It's not the initial setup. It's the maintenance over the years that really makes you question the universe, life and the decision to host your own mail. When you can't send that important mail because $big_provider is blocking you. When someone decides to run a persistent brute force attack from a botnet, eating up 100% of your CPU and you have no meaningful ways to block it. When you need to explain to people why they can't send you that 100 MB video attachment which they sent to other people just fine but only your address is bouncing and why don't you fix your email already. When you need to research, understand and implement standard X pushed by $big_provider because otherwise things will stop working and you have a ton of better things to do. When you get a random alert that email volume is too large and you panic because someone hacked your server and is probably sending spam but realize it was just triggered by a huge kernel patch series sent by someone on a mailing list. When a zero-day CVE for your mail software just hit the top of the HN and the fix is not in your distro yet and you scramble to find a workaround. When a bunch of weird log lines appear in your mail logs and you don't understand where they are coming from and they seem benign but can't lose the feeling that someone is trying something malicious. When you constantly fear that you'll lose that good IP and ___domain reputation and one day wake up with half of the internet blocking you.
As mentioned last week here in a previous “run your own email” story, your setup can also get rejected for not sending enough email, meaning your volume can’t establish good reputation.
Yeah, my solution for low-volume self-hosted email is to relay my outgoing SMTP traffic through Amazon SES. I get good delivery to all the big players, bit still control all the parts of the email stack that I care about. (Plus at low volume, SES is basically free...)
I have had very few troubles sending outbound email directly, however there was one email provider that always rejected me because they were blocking all of DigitalOcean's IP space. This provider was quite niche, but it still bothered me.
My solution was to set up SMTP relaying based on the recipient ___domain. So nearly all my email can still be sent direct, but I have a list of domains that get routed through mailgun.com (or you could use SES or whatever).
I've also blocked tens of thousands of ips from DO, and AWS on several web sites.
Mostly for attempts at logging into wordpress admin accounts, some were spammers I'm guessing were using vpns that sometimes go through them.
Those and tens of thousands from brasil, and several other countries.
There was a time when I looked at sending in reports -
and a time when I asked someone in the wp plugin directory who had a detector-like plugin to have it spit out a chunk of fields that would be ready to fill in the amazon complaint form and to do a cidr lookup to port over to iptables.. but that never got made.
This was all made worse when maxmind went registration needed and ruined the most effective security plugin for wordpress I'd been depending on for years.
I've noticed an increase in the microsoft ips I'm blocking these days to.
for now I don't mind doing an ip lookup when I can block 64,000 ips or more at a time I find it's a solid win.
Is there anything I can do to get whitelisted? How can I contact you?
Not a huge deal if not, I've implemented the workaround already. But to be whitelisted after a chance meeting on HN would be a nice way to finish this story.
I always wondered why there isn’t a config option in MTAs that tries to route an email over different other MTA‘s submission port and tries until delivered. I mean a automated setup not static rules you mention. This would make self hosted setups so much easier. The chance that someone blocked your cloud server‘s IP address 4 years ago and never bothered removing it is high.
When you send an e-mail, as soon as it leaves your mail server it's out of your control and you have no say in where it goes before it ends up at the recipient. There's no such thing as self hosted sent e-mail.
Chances are you’re already compromised eitherways , most mailservers interact with you over STARTTLS, which means any middleman/isp can strip your tls encryption on your emails, midway while you’re reading them over imap. (ISPs worldwide have been caught doing this before)
Most providers do not mandate a strong SSL only imap system.
Along with other similar caveats, do not depend on your mail system to keep your emails private.
If you want privacy, encrypt your emails yourself.
Also most people who selfhost their email usually do not encrypt data at rest. As its not the default norm. As such anyone having access to your disk can compromise you at anytime too (especially likely if you’re using a vps).
So, unless youre encrypting your emails yourself E2E. Assume you have no privacy.
If you’re doing E2E, aws cannot decrypt your mails anyways.
The odds of the emails going through AWS, GCE, or Azure infrastructure in one form or another is probably pretty damn high even if you host your own SMTP service.
Sure, then you can use whatever SMTP option that is best for you. Some people are fine with using American cloud providers and some are not. If they are technical enough to host their own email server, they are more than likely to understand what choices they are making.
If you run your own incoming mail, nobody but you has a copy of your entire mail archive and you are self-hosting that data. It can't be handed over to the government, it can't be sold, it can't be analyzed by gmail, etc. That's where the big win is. If you send e-mail using SES, that's one untrusted hop out in front of an unknown number of untrusted hops that you can't opt into or out of - there's little practical difference. The privacy win come from hosting your own data, not your own smtp server with a carefully curated reputation that you spend hundreds of hours a year working on.
That is _your_ definition of self-hosting. Do you rely on an entity for your internet connection? At some point, you _will_ need to rely on someone else's infrastructure.
Do you run into any issues with your outgoing mail being marked as spam by the recipient? Particularly with GMail’s aggressive filtering?
I imagine 99% of outgoing Amazon SES traffic is transactional or marketing email for various online services. I worry this could make my personal emails look more like spam to the big providers. Or maybe it works out fine.
FWIW running a service who's job is to send email, I have found that GMail is one of the best providers to send to. They strongly value ___domain reputation and are popular enough that it was quick to build up a basic reputation. Now my messages are rarely flagged as spam even coming from a Digital Ocean IP address. I just rely on DKIM.
This is in contrast to Microsoft which seems to rely mostly on IP reputation and trying to send from a new server is incredibly difficult as they (reasonably) treat unknown Digital Ocean IPs as likely spam but (unreasonably) don't allow a good ___domain reputation to override that.
I have set up the same thing for myself and have been using it for several years now, so i’ll join in.
Aws ses has this offer where for a few thousand emails per month, email sending is free.
The steps are this:
1- Signup for aws ses, once you do that they’ll put you in a sandbox environment
2- After that they’ll ask you a few questions on why you need it, just tell them its because you’re a growing startup who expects to send thousands of emails per month, (make sure to say this, they don’t crosscheck later, if you dont say something along the lines of this, they usually reject your application to avoid having to serve small customers who might not scale their business later. )
3- After you’re approved, they provide you with a mail relay api key, just take that api key and attach it to your postfix or other smtpd installation
I use docker-mailserver[0] which packages everything I need for my mailserver into a small container and was good to go, it consumes minimal resources too.
For me, i just had to add the ses relay api key to the config file of my docker-mailserver install and it was all setup.
However you can do the same with any provider that gives you an option to act as your email relay, I remember both aws ses and sendgrid provide this service, but I’m sure there are more niche businesses providing this too.
>2- After that they’ll ask you a few questions on why you need it, just tell them its because you’re a growing startup who expects to send thousands of emails per month, (make sure to say this, they don’t crosscheck later, if you dont say something along the lines of this, they usually reject your application to avoid having to serve small customers who might not scale their business later. )
I have the same setup as you, relaying outbound mails through SES. I told exactly how I was going to use it and was accepted promptly. Maybe I just got lucky.
As others have mentioned, setting up SES is straightforward (as much as anything AWS) and for the rest of my setup I just use mailu.io containers. The config for setting up the relay is here: https://mailu.io/1.8/configuration.html?highlight=relay#mail...
...or for choosing the wrong VPS provider. Op talks about Digital Ocean. But, just like Linode or EC2, their IP blocks are inevitably on some undisclosed blocklist that livemail, yahoo, gmail randomly use.
This is a bandaid that avoids solving the harder problem of trust/spam. It is such design patterns that make a fundamentally open/federated protocol more centralized, exacerbating the problem.
Personally, I think the use of proof-of-work like methods can mitigate the problem by a large extent, making it computationally expensive to spam users. This was one of the original goals of what has now become the "blockchain" revolution. Is anyone aware of any projects that are still implementing similar (open) systems?
I am happy to be proven wrong here (not an expert) but IMHO there is not much hope or solving the open decentralised communication problem with email at all. It seems that something like Matrix.org presents much more promise in this area. I also host my own Matrix server, but sadly not everyone I need to communicate with uses Matrix....
OK I run several mail servers and I do have problems but of another kind that you describe. But this one is a bit ridiculous:
> When you need to explain to people why they can't send you that 100 MB video attachment which they sent to other people just fine but only your address is bouncing and why don't you fix your email already.
The maximum attachment size for Gmail is still conservative 25 MB and they basically dictate what is currently to be expected in terms of attachments going through.
That makes sense, because the attachment is going to be base64 encoded within the email and thus the size is inflated by 33%. This results in the size being roughly 47 MB in size.
You can easily search around that the Gmail limit for personal users is 50MiB, recently enlarged from 25 MiB. As the other poster says, even between Gmail accounts I still have trouble sending files larger than 30MiB.
BTW, there is soo much FUD in your comment, check http://www.postfix.org/ before claiming "someone will hack your email"
"""
First of all, thank you for your interest in the Postfix project.
What is Postfix? It is Wietse Venema's mail server that started life at IBM research as an alternative to the widely-used Sendmail program. Now at Google, Wietse continues to support Postfix.
Postfix attempts to be fast, easy to administer, and secure.
"""
It sounds less like FUD than a change in perspective.
When I was in my twenties, I would have empathized with your point. I used to host my own web servers, but back then, my main priorities were curiosity, privacy and independence.
Not just that, I opened accounts for friends and family.
A decade later, I made all my hosting someone elses‘ problem, because I had different priorities.
There‘s nothing like the sound of a friend shouting in your ear because he trusted you with his mail address and he‘s running into weird errors. Or trying to get an important email delivered after a 10h crunch shift when you just want to bring your kids to bed instead.
I‘m thankful for all those learnings, but nowadays, I‘m old enough to just want mail to frickin work, that‘s why Google does it for me on a custom ___domain.
As I got further into my thirties, I became much more aware of the concept of opportunity cost: by deciding to do one thing, I'm by definition deciding to not do others. Running my own mail server is one task that has not made the cut for being more worthwhile than other priorities in my life.
Sad to see all the negativism. To anyone considering it, just run your own email infrastructure and don't let the naysayers put you down. Email is far too critical to let any megacorporation own you on it.
I spend approximately no time at all in maintaining my email infrastructure which I set up around ~10 years ago.
Of all the things I self-host and self-manage, my email server and related parts is the one which requires the least attention and ongoing work, by far. Set up postfix, it'll take some work initially, then it'll chug along forever.
If other email providers are blocking you, smarthost through an email provider.
If you're getting brute forced, learn how to set up and run blocklistd or fail2ban.
Not getting 100 meg attachments is an issue that other email providers have, not people who run their own servers. If your server doesn't have any free disk space, that's on you. If it does, then set confMAX_MESSAGE_SIZE to whatever you want.
If by "standard X" you're talking about SPF or DKIM, there are lots of tutorials.
If your email software is vulnerable because of issues with your distro, you're doing things wrong.
The point is if you can't, don't. If you don't want to think about issues like these, then you shouldn't be running servers, anyway, so you're definitely not the target audience.
I get it, if you're genuinely interested in hosting email, it's fairly easy... but those who aren't will lose interest after a couple of major setbacks and that's that. E-mail is, whether you like it or not, pretty critical, so it's not the best place for hobbyists to start.
It means no longer hosting your own server for sending email, but rather use eg smtp.gmail.com to send outgoing mail. It would mean allowing Gmail.com to send email from your ___domain via SPF, thus allowing all with a Gmail account to spoof your from ___domain/header.
Ed: as sibling correctly notes, "smart hosting" specifically referes to setting your smtp server to relay via another (eg: Gmail) - eg exim or postfix allow setting a "smart host" so that rather than looking up the receiving smtp server, all mail is shunted to the smarthost to figure out.
Added: you might get good delivery to Gmail using Gmail smarthost (requiring you to have a Gmail account) - but you might need to use outlook.com as a smarthost to get good delivery to o365 accounts... Etc.
It means that "hosting" your own email really means hosting your inbox, and only your inbox. Because "the bigs" have basically monopolized sending email, you will tire of working through everyone blocking email from your 2-bit server. You will cave, and eventually use one of those "bigs" to do the sending half of "hosting your own email." Which means they will "see" and index all your email anyway, which WAS THE POINT most of us were TRYING TO PREVENT IN THE FIRST PLACE. I gave up about 8 years ago, and gave everything to Apple.
That's true, but to do it securely isn't as simple as one would like to believe. Setting up "DKIM/SPF/etc." simply provides auth/verify security on send, but I'm more specifically referring to the receiver side security.
If you're really properly securing your mail server, it would likely be isolated behind a firewall and only have a LAN ip of some kind and utilize UUCP for transport to another LAN machine that does not have WAN access, and then, only allow POP3/IMAP access to machines in the LAN or connected to the LAN via VPN tunnel. Finally, you would want to setup a backup system of some kind for this machine to periodically backup via rsync when the inotify/fswatch file modification triggers.
Next, you'd have a separate SMTP machine. For things like critical deliverability, you can't rely on SMTP to 'retry' albeit it's how they are supposed to act, so it would make sense to have multiple SMTP machines across multiple different backbones in different physical locations with backup power and the like with different MX priorities set.
The initial configuration and running a mail server are incredibly easy.
It's running it securely that increases the difficulty on order of magnitude (because you essentially have to setup a proper security protocol across multiple machines (a network) - defense in depth).
That said, it's easily doable if you're already running complex infrastructure. Hopefully, you're getting paid for your time and costs for doing so.
If not, then I hope you need to rely upon the protection of needing a home-invasion warrant vs a simple-subpoena since a machine at your home can't really just be 'subpoenad' while a machine at some datacenter business can. This of course assumes you're even running the machine at home because if you're doing all this on some VM the value of doing so diminishes ever so quickly.
EDIT: Just to be clear, you can't simply rely on fail2ban and some other on-machine script / snort / daemon / kernel feature to protect you. There are bugs in software/systems and 0days are very real (as well as the market places for them).
Well, then I must do something wrong. I run my personal email server based on Debian stable on my own SBC hardware in a colo for a couple of years now . Manual maintenance is just upgrade to the next stable Debian version every 3 years or so. I had zero of the problems you describe.
Yeah, me too. My life was basically set up and forget. Except for that time I upgraded my instance and the upgrade came with an unexpected and unwanted IP change. Unfortunately it seemed like the new IP has been used for spam and I had to spend some time clearing my IP status. But eventually everyone understood and my IP is clear and shiny now.
Just stick with Gmail then @avian, it’s not for everyone. I think that you might be constantly think of worst cases, where as most of the time it’ll just be fine? And even I it isn’t, then I’m learning more about the web and email. When did people stop doing something because it was challenging?
No, email is an especially brutal service. I had taken this path for 10+ years.
If my custom media server or private photo site setup fails, it is not a big deal. But if I can’t login to a shopping site or my family can’t checkin to a flight because the two-factor auth email disappeared in to thin air, I am the “horrible IT person” who spoiled Christmas - end of story.
What makes it particularly brutal, to me, is that the failures are typically silent. If GMail starts spamboxing my emails, I don't get feedback about that. If I'm not getting emails, I don't get feedback about that.
You can script checking those, but you'll have to re-implement it for each provider. And there are some that you can't; if I'm applying for a job, they're not going to give me an email account to test whether my stuff is deliverable.
HTTP services are absurdly easy to monitor in comparison.
In my experience, the brutality grows rapidly with the number of users. If it's just for you, and you have at least one alternative account, then it's not brutal at all.
A mailserver administrator is a sysadmin. Being a sysadmin is on the face of it unrewarding - nobody pats you on the back when everything is working normally. They only call you when there's a problem (and it's usually urgent, and sometimes critical, and they'd like to know who to blame). So if you run a mailserver with users, it starts to become a people job, and things begin to matter.
People certainly rely on email to a greater extent than other services - even mobile. I can order goods online without surrendering my mobile number, but I always have to provide an email address before I can complete the order.
But I've still enjoyed administering mailservers. Perhaps I liked the slight paranoia induced by trying to reconcile the boss's demands for functionality with his demands for security.
True user-controlled e-mail, the kind without third parties, contemplated in the original e-mail RFCs, is a "two-sided equation". Both sides need to run their own SMTP.^1
1. I have tested both sides running their own SMTP on small, i.e., less than 100, peer-to-peer overlay network and it works. "OTT email". Interesting question for the reader is how does spam enter this system. If the spammer is not a peer on the network, then they cannot send mail to the other peers.
If only one sides runs their own SMTP, that only solves one side of the equation. Almost every HN discussion of user-controlled e-mail focuses only on users controlling one side, while ignoring the other and leaving it to third parties. That can still have benefits such as not storing mail with a third party, but obviously only focusing on one side, e.g., the receiving side, will fall short of true "user-controlled e-mail".
User-controlled e-mail is a solvable problem for most users, i.e., those whose contacts in a given context, e.g., personal, school, or work, are under 100. User-controlled e-mail is an unsolvable problem for people who want to send e-mail to 100s of people, e.g., people they do not know, or people who want to recieve e-mail from any random person/organisation, treating their address like a phone number on a bathroom wall. Maybe, for most users, that is actually a good thing. (Given the attitudes toward "spam", it appears most users generally do not appreciate unsolicited mail.) There could be separate systems for unsolicited mail. We already have such systems in place.
I was thinking of stopping to use gmail and hotmail, and run my own webmail client on a droplet. I don't like the idea of all my documents being tracked.
Is there anything that competes with them that I could deploy.
Not really. Gmail is still one of the best webmail experiences IMO, so you have to prefer clients like Thunderbird if you want to go the self-hosted route. Some people really like RoundCube, but I only use it when it’s absolutely necessary. I did get some novelty out of using Squirrelmail as a teenager.
Roundcube has a built-in editor for Sieve scripts.
Sieve is pretty cool; it runs user-defined scripts server-side, on delivery to addressee's mailbox. The scripts are in the Sieve language, which is just for mail filtering. But it's a bit abstruse - it may be rudimentary, but most users don't want to tangle with a language at all.
Anyhow, the Roundcube UI includes natively a Sieve script editor made up of drop-downs, which makes it much clearer what filter-steps you're asking for. I'm rather minimalist; I prefeer Squirrelmail, if I have to use webmail. But this feature of Roundcube is really good.
Interesting. I did not know it was (mail) server side! I assumed that, like many PHP applications, those filters were just ran using a cron job that calls a PHP script periodically.
It runs server-side, on the delivery server. And the scripts are stored on the delivery server too.
This means that you don't have to synchronize your filters between mobile client A, laptop client B, webmail client C. Thy're all on the delivery server (which is usually the same as your IMAP server). The filtering happens before you log in to email, and before your client knows there is an email.
Sieve is a specification, not a product (and not some specific PHP script). It's part of the Dovecot package in the Debian distro. It's client-server, only because there is a protocol for a client to upload/update scripts. So there's a wire protocol for management. But it has nothing to do with retrieving mail. Sieve on the server only affects what happens on delivery.
I think Sieve is totally the sanest way to do per-user mail fitering.
> It was easier than I thought to create a mail server that works as well as Gmail’s
No it isn't and no you didn't.
The article doesn't even cover basic stuff like email rules and spam filtering (incl. tuning and spam learning). It doesn't "look after itself" like the author wanted (article doesn't mention any update strategy). The author acknowledges that email servers are "open to attack" but this setup doesn't seem to include any security improvements over traditional setups. In fact, maintaining this looks harder due to the amount of custom scripts and lack of good documentation.
And of course it doesn't cover any of the things that actually make Gmail special like labels, having a consistent set of apps for web and mobile, push notifications (esp. on iOS), really good spam filtering, really good search (incl. OCR for attachments), high availability, image proxying, smart suggestions, datacenter security, Google doing code and infrastructure audits all the time, using reproducible builds, ...
It's great that the author is experimenting and learning, but if I had any private data hosted by the author, I would be worried now.
> if I had any private data hosted by the author, I would be worried now.
Merry Christmas to you as well.
Such negativity for just showing something I knocked up in half an hour. - something that I thought might be helpful, with experiences on how to make it more Gmail like.
Attacking the writing is fine, but insinuating my custody of private data is at question is pretty shitty
> Such negativity for just showing something I knocked up in half an hour. - something that I thought might be helpful, with experiences on how to make it more Gmail like.
GP's feedback is direct but quite right imo. I trust the author had only best intentions in mind but "Knocking something out in half an hour" and sharing, but good privacy and security engineering requires probably much more time. Quite frankly, the wording of the article can be insulting even for folks that are working on that problem professionally for several years.
Were it presented differently, it would get different feedback I'm sure. More like "hey HN, i made the first three steps what would be next?" -- i.e. efforts towards trying to understand the problem better.
Docker, cloud volumes, SpamAssassin, Dovecot, ClamAV, fail2ban, DKIM, DMARC. Ask what these are to someone in 1996 and see what you’d get back. The article covered setting all of these up
However my main objection to the OC was attacks on my professionalism. Unless you’re going to defend that, I don’t really care
I think the author and submitter got exactly what they asked for by posting something to HN that is by their own admission low-effort as it took only 30 minutes to knock it out. Many of the commenters call this out, since in their opinion the content does not hold up to what the headline promises.
Also, one aspect of professionalism is also to be thankful for the feedback rather than trying to interpret it as attacks.
> if I had any private data hosted by the author, I would be worried now.
Is an attack on me personally, nothing about the article. The article also took much longer than 30 minutes to 'knock out' - more like 3 hours all in all
I didn't mean to insult you. I think it's great if you're experimenting and I fully support that. It's just that the headline set high expectations and the article reads like this is being used in production, which I would strongly advise against.
At $WORK we use Gmail and I get a lot of automated stuff (cron, etc). I want these types of message to go into folders. I don't want it in my "all" / archive area because they just clutter up searching for other things.
Perhaps labels work for other people / general public, but for me 'traditional' folders is how things work best.
For me personally, one of the most effective means of knocking out the first 95% of spam was using the S25R regex methodology [1] created by Asami Hideo which seems to keep the load on SpamAssassin and ClamAV really low. I've had to adjust the regex rules over the years a little bit but it's really low maintenance for my setup. There are also lists of IP addresses and networks you can block that are known to be malicious which also reduces the load and log volume. [2]
In my experience spamassassin works wonderfully. There are some few false negatives (1-2 mails per week), but I did not have a single false positive which is very important for me. For example Google is much worse in that regard, which forces me to check spam every few days to ensure no legitimate mail ends up in spam, so it's like no spam filtering at all, I have to read it all anyway.
I mean gmail has the most limited frustrating filtering (lack thereof) rules of any email system I've used. Any self-hosted solution will be infinitely better.
> Gmail special like labels
How is that special?
Also, gmail spam filtering is not very good. You know how every business has that "check your spam folder" bit? Because gmail is so terrible about it. It is easy to do much better with a self-hosted solution, put an end to the false positives of gmail.
Seems that you like gmail, but in my experience it's one of the worst mainstream email implementations ever. Doing better is a trivial bar.
Based on my experience running mail servers for a long time and today, I'd say OP is right.
gmail spam filtering is terrible. On my various gmail accounts, I both get spam and the good email goes to the spam folder. And there's nothing you can do, you can mark it not-spam a thousand times and it's still a crapshoot.
I don't have any of those problems with my self-hosted email.
Gmail spam filtering is top notch. I just stopped to care to obfuscate or hide my email adress (which I use since the beta invitation program of gmail) and I can count the spam I actually read in a year with one hand.
It's easy to bring down the number of false negatives if you allow the number of false positives to be arbitrarily large.
On my GSuite business email, I've had > 50 incoming business-relevant emails this year that were incorrectly classified as spam. My personal self-hosted email server [1] lets through a bit more spam than Gmail, but it also doesn't suffer this big false-positive rate.
"I can count the spam I actually read in a year with one hand."
This is partly because Gmail is good at classifying emails as spam/ham.
But it's partly because it's more tolerant of false positives (ham sent to the spam folder) than you or I would be if we were tweaking our own spam filter.
I occasionally check my spam folder, and there are usually some mailing list emails that I don't care about, but which I did actually subscribe to, and would have wanted to reach my inbox.
Seriously? I actually think gmail's spam filtering is brilliant - I probably average less than a single spam email a year that it doesn't catch.
Contrast that with every corporate email spam filter I've ever been subject to, which vary from "shit" to "OK", and Gmail is completely in another league.
My problem with Gmail is the false positives. (Or is it negatives?) They routinely send too much to the spam box and others tell me they have the same experience.
The worst is when they take email from one Google hosted ___domain and send it to spam in another Google hosted ___domain, even though the email didn't leave their network at all.
Still, I agree that the overall level is pretty good and hard to duplicate.
> even though the email didn't leave their network at all.
FYI gmail treats all of its children equally. Mail from one Google user to another is subject to the exact same treatment as mail received via SMTP (and, indeed, Gmail sends traffic to itself over SMTP). If you study the headers of messages in Gmail, you can form a picture of how they allocate and use the virtual IPs.
Have not had to deal with spam on my personal Gmail address in the 10 years I've been using it, and I'm having the same experience running a big Workspace organization. Their spam/fishing detection is making my job a lot easier.
I also have serious doubts about Google's spam fighting. While they catch a lot of spam in the spam folder, they are simultaneously overzealous, catching normal emails that I receive and read regularly, and underprepared, as if putting [email protected] and sending the email to Gmail servers isn't totally obvious spam.
EDIT 2: I don't understand why other comments are so agressive against the author for sharing how he runs his own mail server, I'm not sure if it comes from one's frustration, failures, unreasonable expectations about email, but I noticed that everything related to servers or email receives this hate (here on HN, eh?). Come on, let's start a new year where we appreciate someone sharing their experience in running a mail server :-)
Here is a true statement: "I speak English and have black hair, like Keanu Reeves. I'm looking into growing a beard."
If, based on the above, I were to tell people I looked like Keanu Reeves, would you consider that a reasonable claim?
BTW I'm not deriding your efforts. I'm just saying there's a big gap between 'setting up my first email server with webmail and IMAP access' and 'setting up something with the features and reliability of Gmail'.
Thanks and cheers for being nice! Happy holidays to you too - I’m not really that surprised how negative the comments are here, including attacks on me personally it feels like.
If they don’t like it, stay with Gmail, I don’t care. I would just rather live in a world where the internet isn’t controlled by 2 or 3 big companies. Hacking a server for email and making it work like gmail was the aim, and I did it in less than an hour. Some people on here are pissed that I didn’t consider every eventuality, and filtering, and spam and this and that. Fine, but attacks on me as a person reflect more on who you are as a person.
If you don’t like how I wrote or setup the server, do one and make one yourself - or just stay with Gmail
dang must have the day off or something. The number of rulebreaking "shallow dismissals" in the comments is staggering. Hope you don't let it get to you! This community often thinks it's the center of the internet in my experience, and by proxy any mistake that happens is some sort of crime against the internet that you should pay for.
Cheers, I haven’t - I’m old & ugly enough to read them and let it not affect me. A few have genuinely good points, and I’ve replied to those. Some are just annoyed i didn’t setup a 6 node Kube cluster with 24/7 AI intrusion bot detection with a brand new UI with OCR text detection for image uploads.
Experimenting on the internet is what i did, and it’s what I’ll continue to do, despite what other dullards might say is a waste of time
If youre not on the cloud (and even there on AWS and maybe Google Cloud…Azure is ok in a pinch… and iCloud everything if you also overlap with the Mac crowd) are you even an engineer, seems to be one popular strain of HN thought, which comes out particularly aggressively against e-mail servers because they are arguably the worst type of server to run on your own.
I think it's because many of us here also did run our own email servers at some point in time, until realization came how hard it is to implement and support all features one gets instantly and effortlessly with gmail and such
I recently got into running my own mail server on my NixOS instance using[0]. The server has a total of 1.5 GB RAM and 10 GB of disk space, but it was sufficient to get 10/10 on mail tester[1]. Here's my 12 line mailserver config[2]. It was quite liberating once everything was set up, because then you know you are in full control of your communications.
It was more annoying to set up DNS than the mailserver itself, is there a good way to automate that as well?
Terraform has modules for many popular DNS providers. But that's another tool with its own state to maintain. I've used terraform for both Route53 and Cloudflare.
siraben.dev doesn't seem to be registered anywhere so I don't know if there's one for your provider.
The reason that motivated me to run a mailserver is best described as an artisanal choice, not a practical one[0]. After all, I'm running this on a constrained system and certainly don't have ambitious to scale this to corporate scales.
Like some other commentors here the point is mostly to learn and have something semi-useful at the same time (I've had some pleasant exchanges over my own email already.)
Once your outbound mail leaves your server, it's destiny is out of your control. And you don't have full transparency into the processes your inbound mail has been subjected to. That's in the nature of email. It's orthogonal to running your own server.
> The blog article is the setup to make Docker Mailserver act like a Gmail server.
I'm not sure what a Gmail server is. I was expecting this to include a web ui, admin ui, and the things that actually make Gmail hard to move away from. The docker-mailserver container doesn't seem to include something like that or am I just not seeing it?
The killer feature for Gmail has always been the spam protection and the fact that the emails I sent actually get delivered.
This is nice, but even though I've administered email servers for a quarter of a century, I haven't got the foggiest clue what makes an email server "Gmail-like". What does "Gmail server" mean?
I would think, if anything, that what Gmail has that typical email servers do not is somewhat decent webmail, but that can't be it because webmail isn't even mentioned.
Or is this another one of those instances where people use "Linux" to refer to all things Unix? I genuinely would like to know.
I was referring to Archive/All mail working, instead of just deleting all email, I should have been clearer I think - I don’t use the web UI so don't need a full replacement for that, even though it might be helpful
I've been using mailinabox for years now, and it is really good in the sense that it gets out of my way.
I've included it in my ansible setup, so the basis, distro, os updates, firewalls, backups are cositent with my other servers.
That took some effort: mailinabox is opnionated (and that is good. It is the main reason it works well and is secure), which can be a bit confronting if your opninions are very different.
Many of us run our own small email servers quite successfully, even in 2021. Every time there's a post about it on HN, all these commenters come forward to say it's a fools errand, that it's nearly impossible, nobody should try it, anybody who says it's a good idea is a lying idiot, etc.
Sure, it's not for everyone and there are pitfalls that require effort and sometimes creative solutions to overcome. We should celebrate these projects like we do with other similarly challenging projects that get posted.
You can't replicate Gmail, but with Mailcow I've gotta say the whole process is pretty seamless. You can throw it onto a 5 euro VPS at Contabo, run docker-compose up and be done. Just regularly run the update and backup scripts to make sure you're up to date but that's it, really.
Exchange ActiveSync, multi ___domain + multi aliases with catchalls, (temporary) aliases, mail delivery rules, TLS requirements, you name it, all configurable in the web UI. There's even a built in DNS checking tool to verify that all the necessary records are set up right.
I'm using mailcow for year and a half now, moved to it from Kolab. It's really great and painless to manage it. The only thing missing for me is the LDAP auth, something I got used to over the years with Kolab.
The OP's solutions has built-in LDAP auth, so I'll give it a try.
If you just want your own address, iCloud+ now supports custom domains. You might already be subscribed to it and not know it. This also includes private relay and email hiding. It might be the easiest way to move your email out of gmail.
Important note - custom domains for iCloud is a very limited feature:
- you can't send mails from more than 3 addresses per ___domain
- it doesn't support catch-all
Since I am in a time of moving to other city to study on university, I decided to abandon my mail server and migrate to iCloud... so now I am moving every of my [email protected] to [email protected] (tagging system that doesn't parse properly on some sites). It's no fun, but at least I'll take off my head caring whether my server is on fire, as it's now Apple's issue.
I had to manage email infrastructure for years as part of my job and I really don't see how running your own email server can be a good idea for anyone. Setting it up superficially might be a quick and easy task but maintaining it stable takes hell of a lot of effort. I seriously cringe every time I see this type of guide and articles, it just makes me think that people who write them have zero experience running a mail server and have no idea what it takes to set up one that is secure and stable.
For majority of people best middle ground is to buy a cheap ___domain and a cheap cPanel/web hosting and just use that to host emails. You'll be done in 5min, it will cost you a cup of coffee and you won't have the headache maintaining anything other than passwords.
No need for negativity. It takes very little effort to maintain a highly functioning self-hosted email infrastructure. I don't get these posts saying it's some impossible effort. Been doing it for a decade+, mostly not-doing since there's nothing to do, it just works perfectly.
I run a production email server on Digital Ocean and have not experienced this issue.
An issue I have experienced is that one email provider (who provides a white label service so that small regional ISPs can include a free email account to their customers) has blocked anything coming from DO's IP block. Ultimately my solution is to route those emails (and only those emails) through mailgun.com. The other 99.9% of my outbound email gets delivered directly to the final email server with no issues.
incoming email is easy. Outgoing email is a whole different ball game. I tried setting up an SMTP server on digital ocean and found it to be impossible due to the fact that all Digital Ocean IP ranges are on various blacklists. I moved the server to AWS and was able to eventually get a running SMTP server, but it requires additional steps to gain the trust of AWS and outlook.com and other providers. It SHOULD be possible to set up an email server if you never never never never never send marketing email. If you DO send marketing email, then your stuff should be sent straight to the garbage bin and you should be on a blacklist. That's the whole point of spam.
What's a good way to monitor a self-hosted mail server? I can easily set up uptimerobot.com or similar and get alerted if my website fails, whether it's a DNS, IP, firewall, nginx, TLS^, application, or database issue. Is there a way to check my mail server and get alerted if it is not accepting emails for some reason?
^: uptimerobot.com specifically doesn't warn you if your site works but is using an expired certificate, be careful there
I use and recommend the prometheus blackbox_exporter. You can configure a TCP connectivity check with TLS validation and an SMTP "expect" transcript. If you ran this in the cloud from multiple probing regions/clouds you'd have a monitoring scheme on par with what Google uses to monitor Gmail.
Good article - seems comparing with Gmail upset some peeps but well done for having a go and trying to not just using off the shelf saas for every little things. Progress should have made it easier to host stuff ourselves not harder right?
Thanks! It works well for me, so people just are so negative - I didn’t really expect to upset their day so much… by ‘Gmail like’ I just meant Archive/All Mail working, not just deleting email
Well interface is an easy one.
All the available guis for email work like steaming garbage.
Another few are mail deliverability SSO, and security.
Back in my day, deliverability wasn’t an issue, security wasn’t an ever looming specter, and feature parity was mostly there.
I ended up throwing in the towel with Hey mail, and have really found love for email again.
Personally I find the Gmail UI to be unusable for serious work. The threading is awful (based on subject instead of headers and collapses everything into a linear sequence) quickly going through an inbox is painful because the split view is incredibly basic and the way it tracks read-status in threads is far too unreliable to find what you need to see. Plus every time you open a long thread you need to scroll forever to get to the new messages. It makes me understand why most of my coworkers don't read their email.
Let's say I'm tired of self-hosting my email (for all the reasons previously mentioned by others). What's a good option of privacy-conscious provider I can move my domains to?
I think ProtonMail (https://protonmail.com/) is the only one that comes close to being "privacy conscious," mainly because it's their entire focus. Any other non-major, non-free provider might be an alternative but they're probably not as focused on being proactive about the privacy element?
Now you have your own mail server. Great!
But if you don't know how it works or if you don't have something that will help you maintain it, sooner or later it will break.
You can deploy an email server almost instantly by getting a cheap vps with cPanel on it -- with everything you needed already configured, including spam filtering, security, etc.
It opens with some fair points why somebody might not, but the main reason —perhaps on par with the constant security headaches— isn't there. You will never block spam as effectively as Google, Microsoft, FastMail, etc.
They see [for lack of a better word] infinite times more spam and ham than you'll ever be able to train your little Spam Assassin database, and millions of users to sort through it.
Email without spam control is not a pleasant experience.
This is a valid point - The setup I used integrates SpamAssassin for the basic setup which would get a fair number. There’s a way to learn from listening to when you tag a message as spam that I didn’t cover in my article. But yeah, you’d never do as well as spam protection as the big companies, but is that really such a deal breaker? Maybe if more people like us ran their own infra for email then we’d have better and stronger tools for spam protection?
Not really. By definition, a blacklist is a list of things to block, and things not found on the list get a pass. Most use IP address or ___domain. Unless someone has reported something, they don't get added. The only real exception I can think of is "residential IP addresses, " as some blacklists will try to keep up with multiple ISPs' residential assignments to block them. Even then, they still have to he added to the list to block. Most have forms easily accessible to request review or unblock. Even Spamhaus has a very easy process, and I find them to be rather more aggressive then the others.
Zoho Mail has a free email hosting plan for up to 5 users (web and mobile/desktop app only, with no IMAP, POP, or ActiveSync access), but its paid plan (with IMAP, etc.) actually costs $1/user/month billed annually, which is $12/user/year. It's still less expensive than Google or Microsoft.
