I think the biggest pain in hosting your own mail server is getting your outbound mail delivered into the mailboxes of the large providers without being marked as spam. Especially if you don't actually send a lot of mail, so you can never really build up a good IP reputation.
That's why I generally recommend a hybrid setup: Host inbound mail completely by yourself so that you have full control, but ship off outbound mail to a trusted relay of a privacy oriented provider. For example https://posteo.de/en doesn't filter any outgoing messages by sender, so you can send mail from your own domains. Your local Postfix can DKIM sign your mail before sending them to Posteo and if you add include:posteo.de to your SPF record all your mail will be DKIM signed, SPF authenticated and coming from a reputable IP, so all your deliverability issues will be gone.
> I think the biggest pain in hosting your own mail server is getting your outbound mail delivered into the mailboxes of the large providers without being marked as spam
Why do you think that?
The article mention exactly this could just be a myth that perpetuates because people repeat it without actually trying it.
In fact, I used to think exactly like you when I originally set up my personal mail server and opted for the hybrid setup you suggest.
However, when the third party service I used to send mail shut down a couple of years ago, I quiclky set up outbound mail on my own server. None of my emails are considered as spam, and I haven't touched the mail configuration since.
> None of my emails are considered as spam, and I haven't touched the mail configuration since.
Sounds like you never actually tried to measure the delivery rates of your outgoing email. No email server is actually able to get 100% of its outgoing mail past spam filters.
I've ran my own email server for a few years and I can confirm that the article is making false claims. It's incredibly difficult to deliver email as a low volume sender. The article falsely claims that it's not difficult and that people simply haven't tried it. Well guess what, I did try it - for years - with countless hours sank into it.
I wrote a blog post documenting my numerous, unsuccessful efforts to get my email delivered. I wish people would stop spreading these baseless falsehoods.
> Why do you think that? The article mention exactly this could just be a myth that perpetuates because people repeat it without actually trying it.
Not sure about OP, but many people have tried it and it is a PITA. Even once you have everything setup right (SPF, DKIM, etc.), you still have to deal with IP issues. I ran my own mail server for years (actually, still do for incoming) and it worked fine for about 10 years until I needed to switch providers and got a new IP. Then I found I had to deal with an IP with no reputation and IP whitelist/blacklists. The deal breaker was finally dealing with blacklists that you had to pay to keep off of.
UCEPROTECT was the one at the time. They would randomly add my provider's entire IP space to one of their lists and during those times all email to a couple ISPs would bounce and the only way to get my IP whitelisted from those blanket bans was to pay $$. Switched to using an external provider for outgoing mail and haven't had a problem since as they take care of that crap.
UCEPROTECT was the pits and its operators were unprincipled lowlifes. Luckily they (like all the pay-to-delist BLs) never got more than minimal traction, so ignoring them and their BS mostly worked ('course you needed a /19 to be effective at doing that)
You use past tense... why? They still exist and seem to be continuing their blacklist/blackmail BS. The ISP in question was AT&T and their subsidiaries (I mostly got hit sending to bellsouth.net), which I wouldn't consider to be that small of an ISP and was enough that I eventually gave up.
Past (more than 5 years ago as I reckon) is when I locked horns with them. They may have switched policies, I may have gotten smarter than them, who knows.
I was the ISP, & they had BL'd one of my servers for delivering NDRs. They used to blacklist /16 netblocks wholesale for this kind of, "sins", and demand ransom for getting out.
No - the article contradicts itself. The author says it's "a myth" and "just set up spf/dkim", and then later says "This is already happening with one specifically requiring mails to be sent from another Big Mailer Corp to hit the inbox, or requiring that senders be added to the contacts for others. Any other sender will hit spambox unconditionnally for a while before being eventually upgraded to inbox."
Which is it? Those of us who have run mail servers know that you get unconditionally spammed all the time.
Not GP, but it matches my experience: with reverse DNS, DKIM, SPF, being in DNSWL (and not in DNSBLs) some of my messages end up in Gmail spam folders (very recently received a reply to a message I sent 3 years ago because of that, for instance). A while ago there were strange issues with Google-hosted mailing lists as well, with some messages not being visible even to an ML administrator, while Gmail servers accepted mail just fine (maybe it still happens, I'm just not writing to those). I hear similar stories from other mail server administrators too.
I think it's more of a Gmail/BigMailerCorp issue, and still using a private mail server myself, but I don't think it's fair to say that it's all easy and works flawlessly, even with strangely behaving mail servers on the other end.
I recently checked my Trash folder in Gmail. I had emails in there that were several years old, right below the text that says Gmail deletes emails in the Trash folder older than 30 days.
Indeed, a quick search suggests that spam is automatically removed every 30 days there. The wording was "Google hid it from me", so either I have wrongly assumed that it's about the spam folder, or it is possible to disable autoremoval.
Edit: or, as the sibling comment suggests, it behaves strangely with respect to removal as well.
Or user accidentally archived it. “X did Y” is often a user’s way of saying “I accidentally made X do Y” without having to accept blame. It’s the “dog ate my homework” of excuses.
I think that because I tried it and my outgoing emails weren't being delivered.
Big mail companies have blacklisted entire C-blocks of IP addresses at VPS providers, because parts of those blocks have been used to send spam in the past, and you won't know until you actually check with people you are sending mail to, to ask if they've received them. It sucks.
New domains and new IPs are penalized heavily by most services' spam filters.
My guess is that a lot of people try it, have poor delivery rates, and abandon the effort before they have enough time to develop decent domain & IP reputations.
I've been running my own personal mail servers since 1995, on my current IP addresses since ~2016. If I add a new domain to my configuration, it takes 1-2 weeks before Gmail will reliably accept mail from it. Once I wait that out, delivery is mostly flawless.
As a person who does run a number of (smallish) mail servers, some outgoing mail does get classified as SPAM and I have to fix it. It's very rare - every 3 months or so maybe, but it does happen. (Possibly the most irritating thing about this is they won't tell you what emails made them think you are sending spam, lets alone why they thing it's spam which makes root cause analysis damned near impossible.) The reverse happens too - we occasionally block non-spam emails.
But, and it's a damned big but, AFAICT, my email servers make less of those mistakes than the big email providers like Google or Microsoft, so the overall reliability of my home grow system is higher. It's got to the point that when someone complains they didn't get an email I got, my first question is "do you use gmail?".
The other big but it is it complex. It isn't just a single server - they are a network of them that distribute email across a geographically dispersed organisation. Despite being distributed it archives copies of all email that passes through it (as required by law here). There are some 2.5K of configuration lines driving it. So yes, it requires some effort to set it up. However, like your case those 2.5K lines haven't changed overly in over a decade.
The flexibility the system provides us in the way we handle email is invaluable. As a consequence of that flexibility a lot of emails are never seen by humans - they routed to a place they can be automatically processed.
Personally I wouldn't say email particularly hard, or particularly easy. It seems no more difficult than all the other things you have to do to manage a cluster of servers. The rewards for getting do it inhouse are pretty big, because just like everything else a programmer or sysadmin has control of, over time it will be scripted so heavily it will become almost invisible.
I tried it a couple years ago. Delivering to large providers wasn't the problem. The problem was smaller ISP, one of them flat-out banned all IPs from my host provider (Digital Ocean), others for no reason I could find banned my specific IP. (I was not on any public block list BTW)
Getting off their block list was either impossible or involved digging some old forum posts to find an email address that you could try your luck with, usually with no result as it seems nobody was reading it anyway.
I work in the industry and it's no myth. Even using expensive relay servers doesn't guarantee the big email providers will deliver your or mail or care much to explain why it isn't being delivered. Big email won't even respond to their own customer complaints that they aren't receiving email.
I would also recommend setting up a dmarc record so you can monitor your deliverability.
Spam however is still a hassle and you're never going to match Gmail or Outlook when you're small. Say it takes a few hundred identical spammy emails to trigger a filter. For Gmail it means the chance of receiving that spam is less than 1 in a million. If you're small and managing say a 1000 users - a large percentage will end up with receiving the spam.
You can't actually measure deliverability with dmarc reports. They will only tell you the portion of emails that are bounced, but they don't differentiate between email in inbox and email in spam folder.
SPF and DKIM are best practices to get delivered to the inbox and DMARC reports are the richest source of aggregate email authentication performance results.
However, you are correct, DMARC reports will not provide the disposition of an email getting delivered to the inbox or spam folder. The report is generated based upon the transaction at the MTA level before the message is handed off for delivery.
In essence this is one of the principles, the thing is the emails don't need to be perfectly identical, just identical-ish with the identicalish matching implementation being not public (as this would immediately lead to spammers adding just enough randomness to whatever they are sending)
The article handwaves this by pointing out that it’s “proof-of-work”. Which is a fancy way of saying “it requires a bunch of work, continuously as the rules change over time”.
Which is why I’d consider it “hard” to run my own mail server. The complexity of running a daemon with a config file isn’t the issue, nor is it that any individual task of SPF/DKIM/etc is complex. It’s that I generally want to put 0% of my time into making sure my emails are being received, and I’m receiving other peoples’ emails.
In my experience of over 15 years, the "rules" have only changed 3 times since I started. First to require good reverse DNS, then to require SPF, last to require DKIM. The transition on each of those was generally several years with plenty of publicity.
That hardly covers it. The problem is that who "you" are, your identity as a sender, is not entirely within your control. The big mail operators figure in a ton of stuff, like the reputation of your subnet - /24 maybe, or /96, it's an approximation - or the reputation of your AS, your domain registrar, all kinds of stuff. If the guy on the next IP over is sending spam that's going to pollute your reputation.
If you have your own ASN and your information is registered with reputable parties in lawful countries, you might not have much to worry about. If you are using a cheap cloud VPS service whose network is registered to a Romanian holding company, nobody will deliver your mail.
It would be nice if these big mail services accepted actual proof of work as a way to guarantee that your mail gets through. Maybe someone could extend SMTP with a bitcoin-like challenge where the server sends the client a nonce and a difficulty factor, and the client has to reply with a suffix that you can append to the nonce so that the whole string hashes to something with the corresponding number of zeroes at the end. People sending personal emails will be happy to burn a penny of CPU time as "postage" but spammers won't be able to send their spam profitably, especially if you increase the difficulty factor based on how many emails you've sent in the last hour.
> People sending personal emails will be happy to burn a penny of CPU time as "postage" but spammers won't be able to send their spam profitably
The problem with this idea is that spammers don't use their own computer, they borrow other people's computers to send the spam. It also heavily penalizes people who legitimately need to send bulk email.
One of the big corner cases I've noticed is transaction-oriented email-- the password resets, "welcome to your account" emails, order details, shipping notifications. People want these, but they're probably harder to deliver than personal email or much spammier bulk mail sent directly via outsourced experts.
These have a lot of iffy issues.
* Since they're going to be generated by the ecommerce service, it may not be running on the same server or IP block as your main mail infrastructure, so you might have to specially configure around that.
* The content-- hundreds of messages a day, at seemingly scattershot recipients and highly templated-- screams spammy nto the wrong algorithm.
Yeah, there's services like Mandrill, but the whole ecosystem feels broken. It's not serving mailers. It's not serving the recipients very well. It seems designed to please a small, delicate, vocal audience. The people so upset at the prospect of false negatives (spam in the inbox) that they rally around providers who are prone to aggressive false positives. Who are these people and what makes them tick?
I could see this backlash in a situation where it's high risk or cost, but users can manage their own filters in any half-decent mail system, and frankly, clicking through a few pieces of spam a week is not a big deal-- arguably far less a hassle than spending an hour on the phone to get a tracking number that a spam filer ate.
(Note: deleted and moved here because I put it in reply to the wrong comment)
I work for an email marketing provider (opinions my own, obviously), and I was extremely surprised how many people genuinely want to receive lots of commercial bulk marketing email. And by "want to" I don't mean "don't mark as spam", I mean "continually sign up for different marketing newsletters, and consistently use links in newsletters/marketing emails to make purchases from multiple companies".
It's an alien phenomenon to me, and it may be a minority of bulk email, but it definitely is useful to a lot of people. Maybe it's their equivalent of coupon-clipping-based shopping, or maybe they just like learning about things to buy in that format? I have no idea.
That aside, the last thing an email marketing platform wants to do is send spam. Every time someone marks an email such a provider sends as spam, the reputation of their sending addresses is damaged, and therefore their ability to deliver mail to aforementioned people who want to receive them (and therefore the ability of the marketing platform to make money by facilitating that kind of desired email) is jeopardized.
TL;dr there's bulk marketing email and then there's spam, and surprisingly little overlap (and overlapping incentives) between the people who send each.
Sure, I do this. A lot of stuff people buy is often a lot cheaper with promotions. Just the other day, I bought a $600 set of tires for $300-380 (depending on whether or not I remember to do the mail in rebate). That alone is well worth the few clicks it took to filter mails from that list out of my inbox and into a label for car-related marketing.
I have a couple of broad categories for these filters like car stuff, computer stuff, other-hobby-related-stuff and of course clothing, and just take a look at the appropriate one when I need or want something. Takes virtually no time, and saves a lot of money.
Of course I'm not gonna lie, I do end up buying things that I wouldn't have bought otherwise by doing this. But some of those things are pretty cool, too. :) I do have a few more cheap ARM boards of various types than I can probably use, though.
The paper's argument assumes that the PoW scheme is required though. It could be an option which just guarantees that individual mail can get through, while still keeping the usual reputation-based spam filtering for clients which don't want to participate.
If there's one lesson from the history of spamming, it's that spammers are among the fastest adopters of anti-spam technologies. If PoW meant that spammers could only send 1/100th the email, but the email was 1000× more likely to actually show up in an inbox, spammers would jump on it in a heartbeat.
The value to spammers isn't in email delivered, it's the actual user engagement at the end of the process. A million messages delivered to the inbox is worth far more than a billion messages delivered to the spam folder.
A million messages delivered to the right inboxes are valuable.
The quantity-above-quality model in the spam industry probably lead to low-quality or vague list criteria. Broad demographic groups and domains. A lot of third party data of questionable legitimacy. If you've been running the list a long time, some "email address foo@bar.com clicked on campaigns A and B" data. You might be able to say on a large scale "List A is likely to convert better than B", but it still comes down to spray-and-pray at the individual address level.
If you went to a spammer with a few "sucker lists" of 100k emails each, and told him "you can only send 1k per day", will he have the data to triage his list to get a decent return on it anymore?
This would also snowball over time. Without being able to do high-volume campaigns initially, they'd have a harder time building up the knowledge they can use to manage their lists, and the quality would decline over time.
Yeah, but if the marginal cost of the next message is higher than the marginal expected increase in revenue then they won’t. It doesn’t matter optimizing inbox delivery rates if per-delivery you lose money.
Keep increasing the PoW cost and think about where that line of argument leads you. "One message delivered to the inbox is worth far more than a billion messages delivered to the spam folder." One single delivered message is going to be profitable for the spammer? Clearly not. This proves the existence of some break-even point at a lower cost. Mail providers just need to set the cost above this point.
Given that spammers mostly use others' computers to blast their email out, I suspect that the price point that would truly throttle spam would be higher than the price legitimate users of email would be willing to pay. I'm not sufficiently knowledgeable in spam and black market economics to throw any actual numbers around.
It's an interesting idea, but wouldn't this be easy to abuse though, like Gmail could force Fastmail to do huge proofs of work before accepting their mail?
> Which is a fancy way of saying “it requires a bunch of work, continuously as the rules change over time”.
Wouldn't this be a good task for an open-source package to handle? If you'd just updated the package regularly, your mail would always be sent according to the rules. No need for any third-party handling your mail.
>Wouldn't this be a good task for an open-source package to handle? If you'd just updated the package regularly, your mail would always be sent according to the rules.
Some of the "rules" for reliable outgoing email delivery cannot be encoded inside the source code files of a github repo, or email setup bash script, or a Docker image, or a EC2 virtual image, etc.
An example of an unspecified "rule" outside the boundaries of a local machine holding the email server is recovering from an unexpected blacklisting of ip addresses. Example.[1] A preconfigured Docker image of Dovecot isn't going to magically ask Amazon support staff why the ip address is blacklisted and/or move the server to a different ISP etc. If a bad actor (that you are unaware of and have no control over) happens to share your ip address block and sends out spam which then causes Gmail/Hotmail/Yahoo to reject your server's emails, there's no open-source programming code that can detect and fix those problems happening outside of your control.
Or put another way, if HonestJoeBlow could download a constantly updated Docker image that has the so-called "correct rules" for sent emails to always be accepted by Gmail, it would mean the DishonestSpammers could also download that same Docker image to get their emails delivered.
The issue is that "trust" in the email ecosystem is an emergent property among participants and therefore, it can't all be embedded inside email server configs, or in a DIY blog article trying to explain the exact 12 steps to make email delivery work perfectly with all receivers. Eventually, something can break (because a receiver changes their idea of "trust" and "spam") and it requires troubleshooting & debugging to get email delivery working again.
A tldr would be: You can't put "sender reputation" into a downloadable software package.
This would allow spammers to evade the rules all the same. For anything that is standardized or agreed upon, we can of course have packages like this. Time zone data (tzdata), IP Geolocation databases, etc.
This is true, I run a small mailserver for myself, friends and family. I eventually gave up trying to get mail delivered to the major mail services and now just send postfix mail through Amazon Simple Email Service.
That's why I generally recommend a hybrid setup: Host inbound mail completely by yourself so that you have full control, but ship off outbound mail to a trusted relay of a privacy oriented provider. For example https://posteo.de/en doesn't filter any outgoing messages by sender, so you can send mail from your own domains. Your local Postfix can DKIM sign your mail before sending them to Posteo and if you add include:posteo.de to your SPF record all your mail will be DKIM signed, SPF authenticated and coming from a reputable IP, so all your deliverability issues will be gone.