Rarely have I seen such concise and respectful way of describing the typical rants I see on HN way to often, bravo! I can only wish I had your writing skills.
> It's insulting to the folks that are pouring their time replying to what they think is a user in need of technical assistance, only to eventually stop replying when it becomes obvious that he only replies with questions of morality and treats most answers as if they were platitudes specifically attempting to dismiss his efforts to communicate.
Yes, I have been bitten by this many times. Every time I see a topic like this one I try to help that person by offering technical advice, basically doing my consulting job for free for everyone to read. And many times the post gets downvoted till its dead.
Slowly I learned that people don't want a solution to their problem, they just want to vent their (understandable) frustration about $BIGCORP. But it still hurts to get downvoted when you just try to help.
This is essentially HN with every single Google related post nowadays. Good or bad, people hate Google, they hate it the way they used to hate Microsoft, they see everything it does as being nefarious.
People ignore the email spammers, SEO spammers, automated password crackers and blame google for the solutions they put in place to combat those things.
It's impossible to reason with them, honestly I think it's just best to ignore posts like this.
Have you considered using a full TLD and hosting and/or relaying through a reputable service. I've used mailgun and sendgrid both for moderate accounts for outbound email and didn't have any issues. There's also hosting options for mail ranging from fastmail, to google, MS and others.
In the end, TFA is from a subdomain that seems to have a lot of spam, and to avoid it may mean using A commercial option, if not a specific commercial option. It sucks.. but the mail relays I mention above are relatively cheap and easy to properly setup with limited issues on the other side.
If you were doing everything right it would work, unless you sent email from an IP known to send spam in the past. That doesn't necessarily mean you did, but it does mean your IP did at some point. Given the saturation of the ipv4 space this isn't exceptionally unlikely. In any case, your post is contrarian and doesn't offer any substance aside from the fact that you are angry. That's kind of saddening because that was already pointed out as exactly the problem.
Believe it or not, its possible to carefully look into things and check them, but I understand its much easier to be smug and assume that its my fault.
Judging by your username I don't expect you to share the domain name here, but if you need a second opinion, send me a PM and I'll analyse the domain for you.
There is usually some misconfiguration somewhere that goes overlooked. Happens to the best of us :-)
I've thought about setting up a mail server on a DigitalOcean or Vultr VPS without any experience with mail servers (for person use). Whenever I read opinions about operating a private mail server, there's usually a few people that express great satisfaction in doing so, and say that it's not too difficult. However, there are also many more people that say it's not worth the effort since there's maintenance to consider, complex setup, and the cost of email providers is usually the same or less than running your own server.
What is your opinion (or others' if they'd like to chime in)? If you enjoy running your own, are there any guides in particular that you'd recommend? Does the future of email look even more prohibitively complex for self-hosting?
Self hosting has become harder due to email abuse (spam, phishing, impersonation, etc) and the measures against this that have been introduced.
Following a guide to setup a basic email service is quite easy to do, but after a while the lack of proper implementation and maintenance of the standards will start to cause deliverability issues. Those are frustrating and most end up writing a long rant on why Google is evil because their email ended up in spam. When in fact, it usually is a misconfiguration, but those can be really hard to detect.
IMO there is still great satisfaction in running any internet service yourself. But, if you rely on this service to work, it just no longer makes financial sense to do so.
It takes resources to keep any internet service running correctly. Even the simple stuff can quickly get overlooked, I have seen so many self-hosted email servers that have expired certificates (like with the domain of the OP) or lack support for any of the modern security extensions such as DMARC or MTA-STS.
For most businesses my recommendation is not to run it yourself. Like you wouldn't run your own DNS service anymore. Of course it can be done, and in some cases it makes sense to run stuff yourself (usually when operating at a very large scale). But it almost always makes more financial sense to use a hosted email solution.
We haven't written a guide (yet) on setting up an email server with all the bells and whistles. But if you want to read more on all the email security standards, have a look here: https://www.mailhardener.com/kb/
Thank you so much for the detailed reply. I'm fairly new to programming, development, Linux, the command line, etc etc, and so am always on the lookout to learn something new. My day job is web development, but I'm looking for something that I really want to dive deep into. Email seems to be a whole universe of its own! From what you've said, maybe I'll try it once to learn a bit more someday, but for now I'll let sleeping dogs lie and keep G Suite as my custom domain email host. Thanks again!
Honestly, running an E-Mail is so tedious that I just allow Migadu to manage it for me. You have to deal with configuration (as the original poster said) and run into issues with systems just blocking you outright (especially Outlook). There are some utilities or Docker images that make it easier, but in the end its not worth the hassle. I still think domain-based email is way better than allowing G-Mail or Outlook to host your email though.
When you say domain-based, you mean using a email-hosting provider with a custom domain, is that right? I currently have my custom domain set up with G Suite for email and am thinking that's probably good enough.
I ran my own for a number of years... it was a pain in the ass and took too much time for what it is/was worth. I'm a frugal man, but it's just not worth it to me. I now have several relays pointing to my gmail and/or a custom outlook.com domain (grandfathered free account). I've thought about putting something up, but it's a headache and a half.
For delivery, if you are really just using it for personal use a mailgun or sendgrid account can allow you to handle most delivery issues, while still handling inbound.
Which still leaves the other side, actually dealing with inbound mail issues which is just painful with either too much or too little filtering. The likes of google and MS have teams of brilliant people to handle dealing with spam, and despite some little guys caught in the middle, is still far better than I could ever do on my own.
Thanks for your take on it. I'm fairly new to the world of programming, development, Linux, etc., so I'm always looking to learn things and make something of my own. A mail server sounds very neat in that regard, like building your first computer, but it sounds like in reality it's best left to the experts. I currently have a G Suite account for my custom domain, so that's probably good enough.
I don't mean to discourage anyone who wants to experiment... It's actually really interesting with a lot of options in terms of running your own. It helps to learn things from the security aspects, to the use of tools for spam, greylisting, dns lists etc. On the flip side, I wouldn't do it for my primary email again.
In the end, if you want a career in IT, or find it interesting, I'd say go for it. I've often thought about building my own end to end open-source mail service aimed at ease of administration. If I was ever rich enough to not have to work, that's one of the things I'd probably do after a few months off.
I'd suggest starting with a secondary domain and using Mail-In-A-Box on a VPS or smaller Cloud host. It's a decent starting point, and there are many other options. One of my favorites is SmarterMail (commercial, windows only though). I also ran a BBS for a number of years using Synchronet, which does email/pop/smtop and even nntp for group messages. I did have it configured with SendGrid at the time for outbound for a while, which handled delivery issues for me.
You will have headaches if you take it seriously and/or use it as a primary service without using a delivery service like MailGun or SendGrid, and even then you probably will have other headaches. My point isn't really do discourage so much as let people have a more realistic understanding. The issues the OP has are real. However, there is so much junk from bad actors that the well is poison.
If this is the kind of technical advice you provide, which merely assumes that Google can do no wrong, and the user on the other end is incapable of pressing a button on a yet another website that checks email configuration, merely to confirm that the configuration of their mail server is not in any way incorrect, then it's not a surprise that you get downvoted into oblivion, according to your own statement earlier in this thread.
I can't provide any technical advice unless the op shows their config. I am downvoted because this community is already biased thinking Google must be at fault and the configuration can't be wrong at all ?.
I would say the community is biased the other way around, where everyone always thinks that it's the little guy that has invalid configuration, instead of Google being at fault.
What sort of config do you require from the OP? Their domain name is public and uncensored right in the top of the post. Their configuration has been cross-checked by many people so far. A lot of folks claimed that it has issues (like a `?all` in SPF, or `p=none` in DMARC, or some other nonsense that any spammer would easily get right), but those issues have been rebuked as not being significant to the issue at stake, plus, there's absolutely no confirmation that fixing these "issues" would resolve the problem, either.
Filtering SPAM mail with an acceptable false positive and false negative rate is part of the service.
Unfortunately for you, the e-mail providers' customers (the users receiving the mail) get to decide what is an unacceptably high false positive/negative rate --- and not you. If e-mail recipients get too much SPAM or if they get mail that they want landing in their Spam folder too often, then they will switch. This provides the economic incentive for mail providers to Get Things Right.
Unfortunately, there is so much spam out there that if you are too relaxed with letting marginal e-mails (or e-mails from marginal network neighborhoods through) the false negative rate could result in so much spam in users' inbox that they will get mad, and switch.
However, it can happen that the email provider can be too sensitive, and there can be too high of a false positive rate. For example MIT recently tried relying Microsoft's spam filtering system. Way too much stuff landed in the spam filter, and while you could go to the spam filter and manually mark e-mails as "not spam" and eventually Microsoft's ML algorithms would hopefully figure it out, the false positive rate was too high and the MIT community (especially the ones with the real power, e.g., the professors) rose up with an outcry, and MIT abandoned using Microsoft's Spam filtering service.
My experience is that the email provider's customers have very little say in the matter. I deal with complaints from Customers who use Office 365 and Gmail about false positives regularly. They aren't MIT-profile Customers, so their complaints are ignored. The complaint usually turns into "Why can't this sender just use Google or Microsoft for their email like 'everybody' else?"
It depends on the customer. If you're an Office 365 customer who is a major enterprise with huge numbers of paid users, and your CIO goes golfing with Microsoft board members, you have a huge amount of power. Similarly, if you are an MIT Professor you have vast amounts of power.
They can be pile driving right next to the machine room, and causing disk drive errors, and Physical Plant won't care if the I/T folks complain. But if some Sloan School professors complained that it was disturbing their cogitations, it immediately stopped and Phys. Plant called a "stakeholder meeting" and it was really obvious who had the power.
All customers are equal, but some customers are more equal than others.
"People irrationally hate X" may be perfectly true, but is not a reason why any particular, or most, or the most significant, criticisms of X are wrong or should be ignored.
Even the particular people who irrationally hate something are obviously likely to also run across the good reasons for hating it.
It's not reasonable to say "criticism X doesn't count anymore because we've heard it too many times".
Even technical people are people. This tip can help defuse the frustration, making the recipient more willing to listen to the technical solution.
People forget sometimes that the person on the other end of the screen isn’t a computer that is completely unmoved by silly things like emotion and frustration.
> It's insulting to the folks that are pouring their time replying to what they think is a user in need of technical assistance, only to eventually stop replying when it becomes obvious that he only replies with questions of morality and treats most answers as if they were platitudes specifically attempting to dismiss his efforts to communicate.
Yes, I have been bitten by this many times. Every time I see a topic like this one I try to help that person by offering technical advice, basically doing my consulting job for free for everyone to read. And many times the post gets downvoted till its dead.
Slowly I learned that people don't want a solution to their problem, they just want to vent their (understandable) frustration about $BIGCORP. But it still hurts to get downvoted when you just try to help.