Replies: 57 comments 3 replies
-
|
Hey there 😊 First of all, thanks for your article, and thanks for cleaning up with a few myths! That's a good thing, and please don't take what follows as criticism on your article, but rather as some further thoughts on it… As you said, running the mail server itself actually isn't that hard. What is hard is all the things around it:
As long as running a mail server is not your core business, I would avoid doing all that on my own. I'm happily paying for Office 365 (or any other hosted solution out there) that let's me focus on my actual job, which I get paid for. Because, running your own git server isn't hard, too. Or running your own messaging service. Or running … but it all sums up. And if in the end I have to spend hours per month or per week to manage my infrastructure, I'd rather pay for it – as said, unless it's my core competency. So yes, running a mail server isn't hard. Anyway, it's nothing I would want to do if I can avoid it. Just my 2 cents 😉 Golo |
Beta Was this translation helpful? Give feedback.
-
|
Look at this way, how much email do you really need to back up? could you not store those ones you need to back up on git/gDrive or I don't TrumMail or something? Second, If you use a tight container/VM/kvm/blehBlooh with tracked updates, what are the chances you gonna break you sh%$%t? and dont we all have the option of running gMail/MicrosoftMail/TrumpMail as back up anyway? |
Beta Was this translation helpful? Give feedback.
-
|
I 100% agree with @goloroden. I worked for a company previously with an IT admin who thought he can run our email in house. Our email server would go down at least once a month and on the outside it just seemed like he was incompetent. I knew better (that email is hard), but the CEO just fired the guy. The next guy, who wasn't nearly as knowledgeable just bought O365 and everyone was happy to not have emails sent to them bounced. Also you never touched on reputation. Without it most of your mail will end up in receivers spam folders. Good luck figuring out why outlook SMTP servers are not accepting your mail. Did someone somewhere in the world flag an email from your IP as spam? I'm not convinced. Sure it's easy to actually install postfix. Howver, not being able to provide 99.9999% uptime on email could be your ass. Why risk it? |
Beta Was this translation helpful? Give feedback.
-
|
I care enough about decentralization, data privacy and sovereignty, that I would gladly pay for a turnkey self-deployed solution in the form of an AWS marketplace offering or DigitalOcean droplet. BUT it seems to me that an even bigger problem than hosting mail is the lack of clients with Gmail's capabilities. If someone developed a Gmail clone frontend for such a solution, I would gladly pay a hefty monthly fee for it. So much of my life is on Gmail, and Google is so secretive about when and how they suddenly shut off accounts, that I would much rather manage my own email, but the alternatives lack feature parity. |
Beta Was this translation helpful? Give feedback.
-
|
amazon ip's are typically blacklisted. good luck. |
Beta Was this translation helpful? Give feedback.
-
|
its called roundcube its been there since before dinasours voted republican |
Beta Was this translation helpful? Give feedback.
-
|
@cayblood What feature of GMail has keeps you tethered to their service? Is it just the webmail interface? |
Beta Was this translation helpful? Give feedback.
-
|
@golorden
Hey,
Thanks, and don't worry, I wouldn't write and allow comments if I didn't want people to interact and contradict :-)
Yes, however backups, updates, failover, these are all not mail specific and you supposedly have to handle them for any service you run. I'm not saying mail is easy, I'm saying it's not hard, it's not harder than running other services, it's just... work. I have daily backups, they cover user home directories, mail, websites, databases, etc... mail is not handled any different. I monitor all services equally when it comes to updates and security, mail does not get a more special treatment than my web server which people will never mention as being "hard". Yes it is harder to maintain your services and ensure they are up than to outsource them, but harder doesn't equate to hard in my opinion.
I'm not saying you shouldn't, if this is the best option to you then go for it, I'm not against Big Mailer Corps. What I'm against is the trend that because mail is considered something hard, the immediate solution is to move everyone to Big Mailer Corps without assessing if they would be just fine part of a smaller provider, hosted on a shared server, self-hosted on the family server, etc... This gives them so much power that it's akin to giving them the power to decide what they want to do with that protocol disregarding what's in the interest of the community.
Yes, people have to make choice, some would rather outsource everything, others will want to host everything, and others will want to pick what they will work on, but my point remains: By claiming that it's hard, which it is not, people are discouraging others from even attempting and seeing by themselves, and what bothers me is when this is done by hearsay.
There are also many reasons that aren't technical, why this isn't a good thing for all. I'm sure the Iranians that got kicked out of Github because US decided that sanctions should apply aren't that thrilled about a world where e-mail is fully controlled by 3 or 4 US companies that could essentially not only kill their e-mail but also prevent them from communicating with most of the world. I dislike that idea profoundly and it's not far fetched.
Well, thanks for your comment :-) |
Beta Was this translation helpful? Give feedback.
-
|
@dubvfan87
On the other end of the spectrum is a company I worked for who bought O365 and switched to Gmail, and another who bought Gmail and switch to self-hosting. Heck, a month or so ago I helped a team unbreak their mail setup which was broken by proofpoint in front of O365, I got all of the pain from third-party hosting without any of the benefits. Different experiences, but in all of them I still don't thing mail is hard: it is work, yes, but work is not necessarily hard, it is not necessarily constant and it is not necessarily time consuming on the long run once you know what you're doing.
I didn't touch on reputation because by experience it is irrelevant to most people, reputation is only an issue when you send mail to larger volumes of people and my post wasn't about bulk sending. Outlook is a different beast, I accepted that it takes time to inbox from them. It is when you try hard to work around their spam that you actually make things worse, once you accept that for some time you're going to warn people about looking in their spam folders, it'll eventually get better. Other Big Mailer Corps are essentially no problem for small senders, I can inbox any Big Mailer Corps with a basic setup and, to be transparent, when I used to work in a borderline industry, I could easily inbox pretty much any Big Mailer Corps with not so much work even if they were actively blocking me. If I could do that with the volumes and kind of trafic I was sending, I think most people should easily inbox everywhere (but outlook, who will spambox for a while) given they do the minimum work.
Because I don't have uptime issues and I'd really rather have control over my mail and not depend on a company that can terminate my account the next day if they so wanted. |
Beta Was this translation helpful? Give feedback.
-
|
First, thank you poolpOrg for you contribution to the open source internet. I ran my own mail system from 2002-2015. Started out with just my own domains and then hosted domains for friends and family. Variations of Qmail plus postfix plus ASSP plus custom scripts and DoveCot IMAP etc. Ya'll know the drill. I've a few friends that have done the same thing over the years, and all of them have quit as well. They all say the same thing -- it got to be more work that it was worth. And as the base price of email is "zero" (even though we're now finding out that "free" isn't so "free) it's hard to make money on it. Then came protonmail. They do everything better (and I mean everything) than I could ever and for not much $ at all. Do I regret all those years of running an email system? No, the experience of keeping up with the technology and the internet kept me on my technical toes, a constant learning experience. That experience helped my career. So yes, you can run an email system. If you don't know how, but are interested in it, then by all means set one up. (The cloud is cheap). I have no regrets. |
Beta Was this translation helpful? Give feedback.
-
Thanks for reading and commenting ;-)
We should wonder if people are quitting because they're missing a bit of information that would help them understand why they find it hard. Like the fact that SPF/DKIM are mandatory, like the fact that you should have a valid rDNS + FCrDNS and in a ideal world a matching HELO name. How comes a lot of us, postmasters, manage to handle their mails for decades with minimum maintenance (the last time I had to deal with a block for my own server was over two years ago, otherwise I don't think I ever do mail stuff ... outside deploying new code for testing), while others seem to hit pretty much any blocklist, get blocked at every major host, etc... MOST blocks and junking come from a mistake to start with, something that degraded reputation or that you were not allowed to do (like contacting a spam trap). The way mail works requires a bit of doing something bad over and over again to actually be punished. Sometimes you are a collateral damage, like my block from two years ago, but this gets fixed easily and doesn't happen every two days.
If you feel like protonmail is the proper choice for you, then you made the good choice :-) I don't advocate for everyone to self-host, I advocate for people to give it a try if they want to do it rather than give up because others told them it's hard, and I advocate for people to spread across multiple hosts and not concentrate in the three or four top hosts that are all known for their monopolies in other areas.
I have a self-hosted address, I have addresses at various hosts, we need them all ! |
Beta Was this translation helpful? Give feedback.
-
I remember over the years, learning some new things the "hard way". (SPF record? What's that?) For me the learning part was the reward, for others it's independence and freedom from the "big mailcorps". And I think we agree - for whatever reason you decide to run one, actually running your own email server helps keep us all independent and free, so I salute all of you. It's not for everyone, but everyone benefits (except big mail corps :-) |
Beta Was this translation helpful? Give feedback.
-
|
It would be great if a bunch of mail server experts got together and put together a docker-compose or swarm that is well-refined! It would also help pool optimizations & recommended documentation between mail server tooling. I would be happy to help test it :) |
Beta Was this translation helpful? Give feedback.
-
|
@poolpOrg : very nice article. I agree with your assessments. With one minor difference: I'd rather deflect bad actors than see them continually show up in logs. This is one of the things I use (just updated to add explanatory comments). |
Beta Was this translation helpful? Give feedback.
-
Thanks !
I like seeing them in my logs myself because I test filters on them, they're my tamagotchi :-p |
Beta Was this translation helpful? Give feedback.
-
|
ok, so maybe this is a stupid thing to ask, but where are the installation instructions? https://www.opensmtpd.org/ Having built a mail server which runs on kubernetes (https://github.com/kubernetes-mail-server) I can say the biggest problem that I had was that there are so many working parts and none of them are really explained very well. The man-pages are either 90% of what you need and the 10% that's missing is what you really need, but nobody thought it was important to write down. Or that options are described in very technical terms, but that doesn't mean anything to you specifically, so you google around for weeks trying to find out, how this option affects me, what does it do which I can't glean from reading a highly technical explanation. Then you have all the programs, and ports, and pipes and files everywhere, written in different formats, each multiplying the problem of bad documentation (even after 20 years) that explains only the bare minimum. Then you have the problem of IP addresses, mail servers are quite sensitive to them and resolving to the correct one isn't necessarily so easy if you try to run behind a firewall or a proxy, then you have to take care that you accept email where the SOURCE IP and not the FIREWALL IP, that bit me a few times before I realised what was happening. But not because it was explained. But because I sat down and really drove into the problem of why spam was happening. Then you have the problem of restrictions, in postfix, which is the correct set of restrictions. Is there a page on postfix.org which says "PUT THESE RESTRICTIONS AND YOU'RE GOLDEN". Nope! It doesn't. But it does have a man page going into several hundred words explaining each option and what it does. But do you and have you the confidence to put them together in the right order and get it right? This also bit me in the ass a few times before I realised there is actually a right way and a wrong way. I think the problem comes that nobody wants to tell you what a good "policy" is because this is open source, here are a bunch of engine parts. Go make a sports car! Don't ask me the right way to build it. You do you and you'll be fine. Except this isn't true. There are sometimes right ways and wrong ways and sometimes making decisions which cover 90% of the situations is better than not doing this in the spirit of "not dictating to others what or how to do things". Does anybody know how to host multiple websites, with multiple SSL certificates per domain? Postfix says to run postfix-multi, but did you know that dovecot supports submission now? But have you configured it before? It has very little docs on it, but when it works, it's great. Then you can add as many domains as you want with as many SSL certs as you want without all the complexity of running one MTA per SSL cert. But I might be out of date cause I'm not certain whether it's the only way to do it. It even works nicely with LetsEncrypt certs that you can reuse for the domain website if you configure it properly. Then when I managed to finalise a working mail server from all of these engine parts. I encoded it and allowed you to change a few of the options, many others you can only change if you edit the code. I'm dictating policy because I know that other people can't and other people don't have 1000 hours to read every single page on postfix or dovecots website. So I don't entirely agree that mail isn't hard. I think it gets easier when you spend time with it. But if you try with zero experience to set-up a mail server. You'll fail for weeks before you succeed. Either that or you use somebody else's preconfigured solution and that solves your problem and you never really built it yourself in the first place. |
Beta Was this translation helpful? Give feedback.
-
OpenSMTPD is an OpenBSD software, it is distributed with the system. On other systems, the portable archive should come with a README providing details on how to install: https://github.com/OpenSMTPD/OpenSMTPD/blob/portable/README.md Note however that OpenSMTPD depends on LibreSSL as of latest stable release, so if you want it to use OpenSSL, you'll need to get the development branch of wait for next stable release which is due in a few weeks.
OpenBSD projects are fully documented in their man pages which are often reworked to make things clearer, provide examples, and such: https://opensmtpd.org/manual.html The smtpd.conf man page will provide multiple examples of common setups.
I don't understand this, sorry. Myself, I have multiple simple setups with 10 lines configuration files and I have complex setups which involve multiple machines with segregated roles, relaying to each other, with configuration files that don't exceed 10 lines either. They all use the same software, there's only one file to control the software, it's in a straightforward format.
I think we have a different terminology. When I say it's not hard, I don't mean that it's a two click thing that doesn't require work. I mean that it's not hard in the sense that "you can get it running relatively fast and it won't need you to spend an hour a day on it". You still need to learn whatever software you chose, some being harder than others, you still need to know basic networking and some of the key points behind the protocols you're going to deploy. The same is true for HTTP, the same is true for DNS, the same is true for anything you setup to face Internet. Setting up a mail server requires work, it requires preparation, none of which is hard, but all of which is mandatory to get things going. I have seen people that have gone from zero to running in a few hours and that can now do it in a few minutes.
I don't get that, the rules are very widespread:
the first two points are trivial, the third one requires a google search to know how to generate a DKIM key. I can literally do that in less than 2 minutes and this is not because I'm particularly skilled. Sure you'd take some time doing it the first time, but does it qualify as hard ?
I'm not a Postfix user and generally you will always find cases harder than others, but:
I disagree with you: I've seen people failing for hours before succeeding, they now run servers that don't require maintenance and that plain works. work != hard |
Beta Was this translation helpful? Give feedback.
-
So far, the setup is simply: Postfix, Dovecot, Spamassassin and Postgrey.
Not sure I'm going to work on that soon. |
Beta Was this translation helpful? Give feedback.
-
Ya, I'm more trying to stoke some brainstorming. For example, if such a thing would make life easier for all of you veteran mail server admins... Or just make more of a headache; easier for us all to find our own way? |
Beta Was this translation helpful? Give feedback.
-
|
I just leave this here: |
Beta Was this translation helpful? Give feedback.
-
and what is your point ? :-) Regarding the first link: "I asked on different forums and tried to follow the advices I got. Someone did something wrong for a long time (according to his own words) until his reputation got impacted and pushed him to spam box. SPF and DKIM are not trade secrets, if you look up "delivering mail to gmail" on google, the first link will point you to their troubleshoot guide for senders which tells you the following (which also applies to other big mailers):
So now, just adding these will not fix his issue, because he did something bad for a long time, he has to do something right for a while to regain reputation. As for the ycombinator link, unless you have one in mind, I won't comment all the comments. There's a lot of people who successfully run their mail server, including people who set their first one after this article with no prior experience, so what does it tell about those who keep saying it's hard beyond the fact they probably did not do things right, like on your first link, and are assuming that the whole thing is hard rather than question their mistake ? |
Beta Was this translation helpful? Give feedback.
-
|
I think you're missing the point. Which is that it's easy to say that it's not hard, but when there are countless examples of how people have tried and failed, sometimes because of their own fault, other times because the documentation is just so utterly terrible. It's not such a difficult thing to understand why email has such a bad reputation when it comes to configuration and maintenance. You think it's easy because you are seeing things from your point of view, with your skills, your knowledge, and your experience. Just declaring something as easy doesn't make it so. I know how to setup a mail server, but I'm fully aware of all the problems you can have. That's why we are stating your claim that it's not hard is not accurate. It's based on collective experience. Lets take an easy example, setting up smtpd restrictions, please do not even attempt to tell me that it's easy, you have multiple filters, each where you can select the options and where it'll succeed or fail. Now take a cursory glance around the internet and try to tell me where I can see a default setup, a common setup, a nice filtering setup, a nice explanation of all those options. It just doesn't exist. The docs are decades old. Yet they are absolutely terrible. That's why email is hard, because the docs are terse, uninformative, require knowledge which is not explained. This is also why postfix and dovecots mailing lists are full of people who had problems, why the internet is littered with documents on how to setup an email server and you only get a working setup when you take various parts of each document and combine everything together because lets face it. There is always something wrong with one document which doesn't match what you want and then you have to hunt and then later combine the knowledge from multiple documents together to get what you want. Email is actually hard. Not hard in that each software is hard. But hard in that the configuration, the CORRECT configuration of each component, with each combination, is hard. |
Beta Was this translation helpful? Give feedback.
-
|
Well thanks for this, I’m now seriously considering self-hosting my email again. I did it a few years ago using Zimbra. For the most part it was generally easy once I had scripted the updates/backups etc. This may be fun. 😊 |
Beta Was this translation helpful? Give feedback.
-
|
Some big providers just blocking IP addresses from cloud providers even if the IP is not on any public blocklist. I tried it several times over the years on different providers and there were always mails that didn't get delivered. Without any commercial SMTP relay service I think sending mail from a VPS doesn't work reliable. Maybe use an automatic fallback relay in case the target provider blocks my IP? Is that possible with OpenSMTPd? |
Beta Was this translation helpful? Give feedback.
-
What about just using a commercial SMTP relay? Do you guys think that this is an easy - albeit non-free - way to guarantee self hosting email works as well as big providers? If so, then it could be a fallback for anyone who hits an issue - or a default for anyone who wants to self host without the potential confusion/problem of getting blocked. |
Beta Was this translation helpful? Give feedback.
-
|
I've got a setup on a VPS working for years. The problem is whether you have ticked all the boxes related to DMARC, DKIM, got the correct DNS entries, etc, etc. Then you'll have no problem. But if you're missing any of these, you'll easily fall into a grey zone where things work for a while then stop. I'm even able to send to hotmail addresses from my setup. So I think it's important to have all the correct security information setup and configured for your server so other email servers can properly validate your email server is legit and not used for spamming |
Beta Was this translation helpful? Give feedback.
-
Which tools/sources do you trust to verify the correctness of your DMARC, DKIM, "etc" setups? |
Beta Was this translation helpful? Give feedback.
-
|
I used this to validate some information: https://dkimvalidator.com/ Another great tool is: https://www.mail-tester.com/ But you can only use it so many times a day before they tell you that you must pay ;) but perhaps you can find a way around that, I didn't, but then again I didn't need to check myself so many times a day, I just waited until tomorrow to try some other things and used it for free, I previously got 10/10 score with my setup, which I have here if you're interested: https://github.com/kubernetes-mail-server The problem is that there is no standard way to check down the list. You just gotta go through the tools and play things by ear. That's why I was complaining about email so much in my previous messages. Cause there is no standard way to setup things in the right way, there are just 1000 websites with varying ideas of how to do things and nobody agrees on the proper way to do it |
Beta Was this translation helpful? Give feedback.
-
This doesn't help at all, if the IP address is blocked. |
Beta Was this translation helpful? Give feedback.
-
|
Unfortunately not, some hosting companies are safe though. I've used Hetzner and Contabo without issue, but other hosting services which are cheap, no-frills, and borderline sketchy have the problem you've mentioned with their IP Addresses being blocked. The biggest problem is that email is just fundamentally broken. It's too complex and too difficult to properly set-up, configure, and maintain. It's not easily scalable, many people don't know how it works and there isn't really a lot of accessible information that newbies can read and learn about. Instead of fixing problems and simplifying things, the mail server authors of various types just layered solution upon solution until it's all a quite shaky house of cards, with so many things that could and often do go wrong. But there isn't an alternative, one that you could drop it and replace. So it lingers around when in reality, somebody should try to replace it with a more modern solution using modern techniques and technology. |
Beta Was this translation helpful? Give feedback.
-
|
Finally taking the plunge. I wanted to get some final input from those of you who have analysed the quality of the full systems out there. From my analysis so far, the best is this: Is it missing anything? I suppose future lock-in isn't much of an issue, but switching to a different system would be a pain. Some honorable mentions go to: https://www.proxmox.com/en/proxmox-mail-gateway -- also may be of interest, and curious to hear your alternatives to it. |
Beta Was this translation helpful? Give feedback.
-
|
@poolpOrg Thank you so much for this post and emphasizing that "hard" is not the same as "putting in the work". (I believe this is similar to confusing motivation with discipline.) Quantum mechanics is objectively hard (even according to Feynman), but an open source software with extensive documentation, even if it does not spell out every common scenario, is "just" work. I would argue that open source projects without (or bad, spotty, etc.) documentation would fall into the same category - it's another matter entirely of how long it would take to become productive. Yes, it is daunting, especially if one is not versed in ancillary topics such as system administration, networking protocols (of all colours of the rainbow), programming, and so on, but the information is out there somewhere (and are mostly free): books, blog posts, standards, forums, videos, online courses, and more. On the other hand, it looks as the argument is mostly about semantics (religious wars over a letter come into mind), but the point of your post stands: to set up a mail server is complex/time-consuming/hard/etc. but it is a solved problem and can be replicated by others just the same, and it is not an insurmountable obstacle that would render one's life's work obsolete, irreparably demolish one's reputation (... blabla) and so it should be left to "higher powers" (oh yeah, thanks also for advocating for decentralizing the web and for open source:). |
Beta Was this translation helpful? Give feedback.
and others
-
Beta Was this translation helpful? Give feedback.
All reactions