This is a great initiative. The last time I remember setting up a mail server using Postfix or Dovecot was around 2015.
It was a traumatizing experience. It rarely worked, all my email went to spam and I absolutely hated the amount of hackery and duct tape required to keep it working. Configuration files, missed emails...
It was so unpleasant, to this day I avoid messing with mail on servers at all costs. I'd much rather pay for Mailgun or some provider.
Lately I've been seeing an uptick of these apps making self-hosted email easier. Mail in a box, Maddy, etc. I will unconditionally support anything that makes doing this easier.
I'm not switching from GSuite, but I'm happy to see progress in the space.
Maddy comes with a lot of the 'good stuff' like DKIM and DMARC ready to be used, whereas setting that up yourself with Postfix was a pain (from my experience a few years ago) and probably meant you didn't bother.
I've had better deliverability, especially to Gmail and Microsoft, since using Maddy, probably for this reason.
Now, of course, I must admit that self-hosted e-mail seems like it will always inevitably run into a blocklist problem, usually because of IP neighbours. But personally, I've managed to avoid that in the last year that I've been running Maddy -- main exception is I had to fill in Microsoft's silly form to get unblocked at the start (but I've had to do this for every deployment I've ever done).
I dunno, I'm pretty shit when it comes to Unix systems and programming in general (spent 6 hours fighting a postcss and tailwind config file and lost) but I got dkim and dmarc set up on my own server for my own mail in maybe 30 minutes?
So you agree is not about the software but you did not implement best practices with Postfix.
Also, whenever you give an address to someone else, family or friend, risk of going to spam increases.
Links or src in emails without https, higher risk,
Does Maudy prevent you from doing that?
And your sales people will call this a misconfiguration serious enough to fire the sysadmin who did this. The problem is that, typically, nobody trains or supervises the filter applied to the outgoing mail, and nobody is warned that it did, in fact, reject an email, and there is no manual override if the filter is, in fact, wrong. And spammers (from the viewpoint of a particular organization) don't exist and can't exist inside the organization, so any rejected outgoing email is, by definition, a false positive.
One "weird trick" I discovered with deliverability was that plaintext email from my server was rejected, but HTML email from it was accepted.
For anyone with deliverability issues and mail going to spam, it might be worth trying this out (after you have sorted all the usual points like DKIM, SPF, PTR record etc.) - this obviously only helps if your IP is clean. I was surprised though, and only discovered it by accident.
I had good deliverability to Google hosted mail, but not to Microsoft 365. I even had issues with deliverability doing 365 to 365, until I sent HTML mail.
What a world we live in, I always would thought the opposite. Probably it was like that before, but now maybe spammers send more plain text e-mails to appear legit. Never ending game of cat and mouse.
My guess (pure speculation) is that more spam is auto-generated via email libraries and similar, and that is producing plaintext mail to try to simplify what it's getting through the filters.
On the other hand, most human-generated mail is HTML-based, as it's written in a web-client or desktop/mobile "app client", that is using HTML to let people do bold and put pictures into their email.
I was as surprised as you when I figured this out!
As someone who has run his own mail servers since 2000, and still gets his email delivered to people's spam folders (even when I've emailed them in the past, or simply replied to their original message), this is very interesting, as I always enable plaintext outbound email.
Your symptoms sound exactly like the issues I was having - everything was "right" (in theory) - emails were leaving on the right IPv4, that had the correct PTR, etc etc. I just couldn't figure out what was wrong.
It was the same issue - even replies to existing threads had the issue. What got me looking into it more closely was that I had no issues delivering to G Suite tenants; it was only issues with 365 tenants.
I wonder if there's a non-trivial correlation between people choosing to genuinely self-host their own mail, and those who use plaintext outbound mail by default (and know what that means).
It's very odd. I don't know how Microsoft implements their own server-side spam filtering, but the issue was coming from their filters. No issues with any other host. G Suite accounts received plaintext mail fine.
I never found much by way of technical detail on 365 deliverability, beyond the usual DKIM/SPF/DMARC/PTR record guidance, and forms for getting yourself removed from the whitelist.
What's especially strange is that this issue applied even when sending mail as a 365 subscriber. Once there was HTML content present in the email, deliverability (to other separate 365 tenants) improved!
I also hated the headache of Postfix so I switched to OpenSMTPD on OpenBSD and it is so so much nicer. It feels much more set-and-forget than any other time I've tried running mail servers on Linux.
> Lately I've been seeing an uptick of these apps making self-hosted email easier. Mail in a box, Maddy, etc. I will unconditionally support anything that makes doing this easier.
I had great fun setting up Postfix and Dovecot recently.
However, I was very specific with always going for the minimum most simple solution. I do not have a dedicated database for instance, but use just mbox and POP3.
And I did not even bother much with anti-spam, I chose a super easy filter in Thunderbird instead, sending everything with 'unknown' in Received: header to the Thrash folder.
Getting accepted by others was also not much of an issue (but lots of details of course, PTR, SPF, DKIM, DMARC).
> And I did not even bother much with anti-spam, I chose a super easy filter in Thunderbird instead, sending everything with 'unknown' in Received: header to the Thrash folder.
I realise that everyone hates spammers, but having a "thrash" folder is a bit much, isn't it? ;)
I've been using Maddy now for about a month for my email, and it's incredible. The dev says it's not caddy for email – but in my experience, it's certainly been that easy.
I've got to write a longer article about this, but self-hosting email is incredibly liberating. I now own my entire digital identity. I can work on open source projects with developers around the world completely while relying only on my own capabilities.
From someone else with similar experience: I've been following all best practices to the best of my abilities (rDNS/DMARC/DKIM/SPF/etc). After all that was properly in place, the only problematic receiver was Microsoft (outlook.com/hotmail.com/live.com), which bounced and provided an appeal process.
I filled the form, was requested documentation from my ISP on the IP address.
I asked my VPS hosting company (since they provide the public IP and therefore act as ISP) and they proactively reached out to Microsoft, who lifted the restriction after that.
So since then no delivery issues. YMMV.
If you intend to host this from your residential address, it can be a good idea to tunnel external traffic over VPN through a VPS or similar. Not only for privacy reasons, but also to get around ISP blocks and IP banlists. I don't know how flexible maddy is, but in postfix in case the above scenario wouldn't be resolved, I could have set to use mailgun/mailroute for MS domains only and relay like normally for others.
What do you use as a mail client (on desktop and on phones), and do you have a solution for server side signatures? If so, how well have you found that it supports Reply and Forwarding insertion? Does it support embedding signature logos as hidden attachments to workaround blocked images?
Do you also have any issue with icloud? I somehow get bad IP range and never been able to have them whitelist me, re-attempting seems just put put me into a blackhole queue support on their team :(
i have been using mailinabox on a vps server and all the major email tests say the mailbox is fine.
only gmail treats my emails as spam. others just work fine. i don't know what to do other than having to call receipients and ask them to unspam the mail. That doesnt seem to "train" their spam filters so don't know
I have literally never (to my knowledge) been rejected by one of the big boys (i.e. gmail or outlook). Was quite nervous about it in the beginning, but I haven't run into any issues.
I have had a number of bumps:
1. In exchanging emails with someone with a custom ___domain, I found there SPF record was broken and thus my server was rejecting their emails. I've weakened my policy and now their mail goes to my Junk, which I then manually move to my inbox because I'm lazy and don't want to set up a custom rule.
2. I wanted to subscribe to the Tarnsap mailing list, and had to decrease the minimum TLS level for outgoing mail to "none." Dr. Percival believes TLS on SMTP is "silly" (which, in the sense that all email is insecure, is true, but in the sense that email with modern security measures is better than nothing, is in itself a "silly" opinion).
3. I had some server downtime recently (https://figbert.com/posts/wrong-way-to-switch-server-os/) and couldn't receive emails, which sucked. But that was on me.
Just to defend Dr. Percival a little here(since I have the same stance, though we do also support TLS), the RFC's require you to support non-encrypted SMTP. Since you HAVE to support it(not only per RFC but because in the real world so many SMTP communicators are stupid, lazy, or ignorant), there is little point in trying to make email secure. Until such time as everyone decides TLS1.3+ is required for SMTP, there is no hope, so why bother.
Our external auditors get all upset about it every single year, and every single year, I show them the RFC's and they then shutup about it for a year. If you feel strongly enough about it, try to get a RFC passed where SMTP requires TLS now.
I don't run Maddy, but I do run email for an organization of a few thousand people. It happens on occasion from various providers, The larger organizations(MS,Google,etc) will spam your logs SMTP errors with a URL. You visit the URL and do whatever actions they want that particular day and life goes on. It's not hard, but it is a bit annoying sometimes.
Generally if you have a static IP and you don't go being all stupid with spam, it's not THAT difficult, but you do have to jump a few hoops and then occasionally play wackamole with their spam prevention junk for the month.
It seems to come in waves, like email will be fine for a few months and then 1 provider after another will be all upset about gosh knows what that day and you have to visit URL's and push a few buttons.
I've been to lazy to track it, and the various reasons for that particular day, but this has been my experience. A few times a year you have to go babysit SMTP so email can be delivered again.
Not just MS, they pretty much all dump SMTP errors with URL's telling you about the SMTP error they gave you. Some are really awesome when you visit the URL, they say oh, do X and then you are good. Others say we just don't like you at the moment, with basically no detail... and then you get the full burden of figuring out why they didn't like you and trying again.
We host and manage the SMTP server(s) ourselves(We currently run Postfix). If you outsource your email to Google, etc, then they have to babysit the email logs for URL's, not you.
To share one trick I found (after you've done all the DKIM/SPF/DMARC/PTR things others point out), if you still have deliverability issues to Microsoft 365 tenants, it is worth sending mail as HTML rather than plain text.
I have no idea why that helps (well, I could guess that some spam heuristic thinks plaintext email without an accompanying HTML envelope is more likely to be spam), but changing this took me from near-constant "your email went to spam" to no issues sending even things that actually look spammy (i.e. an email just containing a link that might be of interest to the recipient)
Is there some sort of web interface for self-hosted e-mail that is reasonably good and sleek/modern-looking like Gmail, and also works well on mobile with swipe gestures and everything?
I imagine this comes down to personal preference a lot, but I strongly prefer Roundcube over Gmail. They have improved a lot over the past years, if you have some preconceptions. After 10y+ on Gmail I finally feel like I am getting a grasp over my own email.
But then I also very much prefer a hierarchical/directory tree approach to organizing e-mail than labels&search as Gmail does it.
More similar UX-philosophy can be found in Mailpile[1] and Cypht[2]. Both still have decent amount of moving parts but are continuously progressing.
Maybe it helps to host the email server in a less known and maybe a bit more expensive ISP. I have been self hosting my company's emails for 10+ years and even changed IPs halfway (VPS migration). I vaguely remember having to setup DKIM and such when that became a things and never got any deliverability problems since (fingers crossed).
Have you actually hosted an email server on all five of those? As you mentioned in your other comment, it can be very helpful if your VPS provider is able to respond to support requests asking Microsoft for IP addresses to be whitelisted. I can't imagine all VPS providers would be so helpful but you didn't mention which one you were using in that comment.
Late to the game. Vultr does block port 25 by default, but if you request it to be unblocked for your account and agree not to be abusive with the protocol, they'll open it up for you.
Don't know about others, but Linode and Hetzner have parts of their IP space on spamming blacklists. As does Digital Ocean. So it's a lottery, basically.
Docker is pretty portable by design. If you use bind mounts you can just scp the docker-conpose.yml and your bind mount volumes to your new server and you're done. If using Docker-managed volumes you'll need to docker cp the data out of them first before scp, then docker cp them into your volumes once you docker-compose up your containers.
Mail is one of the easier things to backup IMO. It's just a directory if you use Maildir. Just tar it up with other important stuff on your filesystem you probably also want to save. No need for fancy "streaming SQLite replication."
Replication and backups are usually different things. If you need replication, yes, backup tools won't cut it latency wise.
But proper backup tools like rsnapshot, borg, etc. pp. have no trouble correctly backing up email servers even at 10 or 100 times your mail store size.
Litestream author here. It has a 1s replication lag by default. It will snapshot the whole database periodically but otherwise it only incrementally uploads the new WAL pages (which are LZ4 compressed).
Depends on how much latency/what big a time window for data loss between runs you can afford. We went from streaming cyrus-imapd replication to minutely send/receive for availability, but that introduces a 30-90 time window where data can be lost. Fine for us (esp. considering the reduction in ops overhead), but not necessarily for everyone.
(Backups are done by rsnapshot pulling from a snapshot, so we have another copy outside ZFS just in case.)
Maddy is fantastic -- for greater flexibility/ease of deployment, you can pre-provision your DKIM keys as well. Maddy also works wonderfully when combined with other technologies, it's nice, simple and light.
It works great with Haraka[0] on the outside proxying email to maddy instances on the inside. I've also combined it with SES provisioned with Pulumi + k8s[1] (for those on AWS), which is a little bit more involved but is what I use on projects now.
Another entry in this space is chasquid[2] but I've only used/can recommend maddy. Maddy is so good it makes me think anyone could run a hard-multitenant email provider with ease (I'm essentially doing that for each of my projects).
I've been running my own selfhosted mail server setup for about six years [1] without much trouble. It's a lot of work in the beginning, and there is a learning curve, but I've never experienced any of the issues lots of other people seem to have with their setups.
Same experience here, used the workaround.org tutorial as strong inspiration (not blindly copy pasting, but to serve as guided path to follow and learn).
Steep learning curve indeed, but after initial setup it's just working and necessitate no tending except when I do major upgrades (of Debian so every 2.5 years on average), in case some config needs updating.
But this is really a private email server with no users except me, thus probably not encountering the issues related to mail servers with many users that might or might not have a good email hygiene.
Personally i use a mail server that's preconfigured and lives inside of a Docker container with all of the dependencies that it needs: https://hub.docker.com/r/mailserver/docker-mailserver ( GitHub: https://github.com/docker-mailserver/docker-mailserver ). With it, adding a new e-mail account is as simple as running a script with some parameters and it interoperates nicely with most clients such as Thunderbird or even the one built into Nextcloud (for a lightweight hosted solution without a separate app to manage).
It is pretty awesome to see new projects that try to improve the user experience for the system admins, for example, the Caddy web server and now the Maddy mail server. I'm not sure whether i'd have been able to set up a mail server the "old fashioned way", seemingly in line with the concerns of the other people here who have tried that more extensively. Somehow, while e-mail is super widespread and works decently for getting text information from place A to place B, it also feels a tad overcomplicated (the different components for sending/receiving mail, SMTP, IMAP, POP3, the sad need for anti virus scanning, SPF, DKIM, DMARC, the similarly sad need for anti spam protection).
That said, the containerized solution above has also worked nicely for me - i'll still probably use GMail/other hosted solutions for personal e-mail stuff and communication, but right now i've already switched over to using my own mail server for all of the automated e-mails that i need, such as from SonarQube and Zabbix, as well as GitLab.
I tried it recently and was irritated that it doesn't play nicely with a existing LetsEncrypt instance on the host. Docker doesn't like mounting symlinks, and LetsEncrypt uses symlinks to point to the latest live certificate and key. I could configure Certbot (LetsEncrypt) on deployment of a new certificate to copy it to the Docker instance and tell it to load it, but at this point I just fished out my old Postfix and Dovecot configurations and notes and did it myself; at least this way I'm in complete control.
I do approve of such a project existing though, and if I wasn't running other things on the host I would probably use it.
I've been using maddy as a personal and business email server for a while.
Resource consumption is basically zero, no problems with sending or receiving emails at all.
It did take me quite a few hours to properly set up a Kubernetes + Terraform config though. Getting all the bits and pieces for DKIM, SPF, DMARC right, including issuing certs via cert-manager, is quite finicky.
Unrelated but: what VPS hosts are reliable for email? I've had trouble with Linode being blocked in the past. I'm aware Google Cloud blocks port 25. I haven't tried EC2 but I assume the worst.
I've run mail servers on EC2 before. You'll need to raise a request to remove the SMTP 25 restriction (both inbound and outbound), and also apply for a DNS reverse ptr update for that IP for it to work.
VPS providers generally (don't know about GCP) lift port restrictions if you write them and request it explicitly.
At least Vultr and Linode will also communicate with other big providers (e.g. Gmail or Outlook.com) in case you have delivery issues due to history of the IP addressor subnet before you took it over.
This doesn't answer your question, but I gave up on hosting my own email. I host my own other stuff where I can but email is just too much of a pain; constant delivery problems (especially to MS), spam is a pain to keep up with etc.
Currently using Time4VPS with no issues so far: https://www.time4vps.com/?affid=5294 (affiliate link, personally i find it to be an affordable VPS host that i've used for years and host most of my stuff on).
I think the actual question would be: "Which VPS hosts have the least blacklisted IP addresses for hosting mail servers?" But i'm afraid that i cannot answer that one, because after a brief search i could not find any articles, which would check the IP blocks owned by different hosts and what percentage of them are in blacklists, which would probably be a large undertaking.
Until then, most of the answers will probably be subjective, along the lines of: "I use $HOST without many problems." (like my answer above, though even then someone could get a blacklisted IP address and their answer would be the exact opposite to mine).
I got good results with Hetzner. But it takes some time to set up a server that complies with all of Google requirements. Also, I once inherited a spammy IP address and took me some time to remove it from spam databases.
I think the advantage with Hetzner cloud is that you rent the IP address separately. If you get an IP with a bad reputation you can stop renting it and get a new one.
Edit: I have not actually tried changing the IP, its just an idea. Also, I hope the spammers don't start doing this and spoil it for everyone else.
Hetzner does not open port 25 outbound (SMTP) until you have paid your first invoice. I therefore believe them to handle this problem somewhat seriously.
Yes, basically that. I set up mail servers for a company in 2001-2008 and then it was way easier. I tried to do it again five years ago, for my personal server and quickly found the landscape changed a lot. But it is doable.
Fair dos. I do run my own mailserver and although I have had problems I haven't found it impossible. The complexity in running a mailserver has for sure increased since 2008 but it hasn't seen the same level of complexity expansion that, for example, making a web site has.
I do use SPF but I don't use DKIM or DMARC. On the inbound side, I do use greylisting and anti-spam. I don't tend to change my config between releases of debian stable.
Spam, phishing, virus blocking - no builtin measures. It is easy to integrate rspamd (literally adding one line into config). ClamAV can be used via clamav-milter though some configuration is needed - no direct & easy integration for now.
maddy implements de-facto standard milter protocol so it is possible to use any third-party filtering software supporting it.
Mailcow is a great system. I use it and I love it.
But it's a total mess. So many different components glued together, so much that can go wrong. The configuration is scattered into hundreds of files. There is no clear split between the product and user-configuration. To change some settings, you need to make adjustments in the core product (the config files that hold everything together).
Replacing a lot of components with Maddy would probably make projects like mailcow way slimmer.
I’ve been using YunoHost (https://yunohost.org/) for my VPS setup for years, including emails, and the setup is such a breeze. It takes care of everything from certs to spam detection, includes IMAP, and you can even choose you webmail interface.
Maddy sure looks like a great alternative if you just want/need emails.
I'm running a Postfix, Dovecot, OpenDKIM, OpenSPF stack, and it's not entirely trivial to setup and get working..
Next time, I'll try this, if I remember..
Next big issue is how difficult it is to actually get to send email from your email server when you're really self hosting it (as in, on a box in your actual home, like it's supposed to be)
Ditto for me (postfix, dovecot, opendkim, spf etc), took me about 3 months to get it working, and for the last 6+ years I haven't dared to touch my postfix configuration!
I've been using simple mail forwarder (https://github.com/huan/docker-simple-mail-forwarder) for a while now to catch-all and forward any email to my ___domain to my personal email on protonmail.
Advantages are that I can move to another email provider without much hassle, and that I can have a different email address for any service I sign up for.
The main disadvantage is that I can't make it look like I am replying from the same address.
I wonder if it's something that's doable?
AnonAddy can do this and it has a nice web GUI. It also allows you to individually block any of the email addresses from forwarding to your inbox, which helps make spam control much easier.
It’s every couple of months that someone at my job comes along and says hey give me access to this data because my team is building their own marketing email tool. I always ask them who they know at google so that they can call and get unbanned from gmail. So far no one ever has an answer…email is tough and people need to be very careful when they try to do it at scale. That’s not to say that projects like this aren’t great. They are. But enterprise scale email is very complicated especially in regulated industries like finance and healthcare.
I used to run CommuniGate from Stalker on my private mail server.
It is more or less one binary and one data folder, available for all operating systems, providing a lot of functionality (SMTP, IMAP, ActiveSync, Webmail, Groupware, ...). Everything can be configured via one uniform Web UI. Multi ___domain, multi IP, and you could even cluster together multiple servers.
It's closed source, but offers a free license for up to 5 mailboxes.
Sadly this product died during the last 10 years and is now in an unusable state.
I'd love to move my crufty mail setup to something like this but after 15 years of self-hosting, there are many wrinkles and dragons that people rely on (e.g. rspamd whitelist from iOS notes, IMAP authentication and proxying via Perl+sqlite) which these pre-packaged solutions don't (and probably rightly!) support.
But if I was setting up something new, I'd definitely use one.
Yes running your own mail is still hard, and our approach is to manage and automate existing MTAs via an Open Source app: Lightmeter.io.
Maddy looks great, and necessary. For people wanting to stick with Postfix, Lightmeter is worth a look. FYI we recently did the YC S21 interview but flunked the last hurdle this time around.
My one question: Caddy is known for effortless TLS, has it on by default and makes it "just work". Why can't Maddy also automatically set up TLS, instead of having to use a cert acquired by some other means?
I remember building my own mail server on a PIII with 500mhz and 128mb of ram. I used qmailrocks.org and learned so much about Linux and how things work. This was in like 2003 or 2004? My how times have changed!
I use nearlyfreespeech to host my personal website, but they don't allow for email hosting. What other similar host (cheap but reliable) would you recommend for hosting an email server like Maddy?
fwiw, I'm using postfix, dovecot and rspamd for a year now and it is working like charm ever since. However, gmx.de rejected emails due to reverse dns issues.
IMAP4rev1 is fully and almost [1] correctly implemented.
There are some useful protocol extensions supported but some other important ones are missing (e.g. CONDSTORE).
There are no known issues with popular clients but performance may be a bit rough. Both due to missing extensions and storage implementation quality e.g. SEARCH may get slow for large inboxes.
It was a traumatizing experience. It rarely worked, all my email went to spam and I absolutely hated the amount of hackery and duct tape required to keep it working. Configuration files, missed emails...
It was so unpleasant, to this day I avoid messing with mail on servers at all costs. I'd much rather pay for Mailgun or some provider.
Lately I've been seeing an uptick of these apps making self-hosted email easier. Mail in a box, Maddy, etc. I will unconditionally support anything that makes doing this easier.
I'm not switching from GSuite, but I'm happy to see progress in the space.