Hacker News new | past | comments | ask | show | jobs | submit login
How to prevent email spoofing, using an unholy combination of silly standards (simonandrews.ca)
392 points by simon360 on Aug 15, 2021 | hide | past | favorite | 125 comments



I disagree with the idea that SPF, DKIM and DMARC are somehow hard. They're all one-liners in DNS config that can easily be generated with some online tools if you don't want to read the standard. In my experience, correctly configuring mail server software is a lot harder than configuring the DNS to make your email arrive as reliably as possible. Obscure configuration formats are par for the course for almost every mail server setup.

I've personally had a much harder time getting DNSSEC to work than I ever had issues with the various email DNS records. Even that is manageable if you're willing to Google around for a while.

My solution to getting self-hosted email right is using Mailcow. A cheap VPS (Contabo FTW) plus Docker is all you need to get it to work, and it spits out all the DNS records you might need to add. That also includes autoconfig records for tons of software, which is nice.


Ummm... DKIM is NOT a one-liner in DNS. It requires actually setting up, well, DKIM. The DNS part is simple, yes, but the DKIM isn't, and equating it to a one-liner in DNS is just simply not true.


It’s not trivial, but it’s definitely something I’d expect to be within the grasp of most visitors to this forum.

Importantly you can also verify the correctness of the configuration, which isn’t the case for all anti-spam and anti-spoofing measures.


For most services it is. But you’re right, if it’s self-hosting it’s a real pain in the d**.


would agree..It can also be hard finding selectors down the line that you had already set up because they can often be anything and largely depends on the service and how well they document their standards (and assume they dont change).

I have had to revisit these a few times for companies that change their setup standards and then dont tell their customers (same has happened to SPF but that is much easier to audit/fix).


Opendkim as a daemon as something added to an existing smtpd with postfix is not that complicated.


This feels like shifting the goalposts. From "one line in DNS" to "just install extension into your email daemon and configure it."


Your e-mail daemon should already have configuration for DKIM, and the same tool that generates that key should also print out the complete DNS entry you need. Configuring DKIM in your daemon is a bit of work; The DNS portion of it is simple.


If a person is trying to self-host their own MX and doesn't want to get into the details of configuring SPF, DKIM and DMARC, I would highly recomment they go with something like gsuite or office365 instead. Or some other managed email hosting provider that will do the DKIM setup for them.


> They're all one-liners in DNS config

$1 - turn the screw

$9,999 - knowing which screw to turn


Exactly that. Plus, if you have to configure it, you're already doing more with e-mails and custom domains than 99% of average people who just use an address from a free provider or their ISP.

It's nontrivial to move away from e-mail-as-a-service. It shouldn't be, but we're stuck with decades of legacy and abuse. Years ago there was a statistic doing the rounds that either 80% or 99% of e-mail traffic was spam; I'm confident that is still the case, but thanks to technologies and big providers, end-users only see a fraction of it ending up even in their spam box, and even less in their actual mail. There was one in my non-spam gmail inbox the other day, the first one in years.

I run a wordpress blog with comments enabled as well, Akismet has stopped a million spam comments (99% of comments attempted to be poasted).


This is the exact reason I keep my hotmail account. Which now seems to get me this 'you still you thhhhhhaaaat?'. 'would it make you feel better if I gave you an outlook one or a gmail?'

They filter like 99.99% of the junk. Usually 200-1000 a week (had a low of 20 a few weeks ago). They even show me what they are filtering. That is just 1 account. I have toyed with the idea of setting up my own domain and having my family have all their own emails. But that would mean managing it. Not terribly hard but just one more thing I do not want to mess with. 30 years ago I would have done it. Now I would rather it 'just work'.

Managing spam is a pain. I have been using the 'unsubscribe' links recently. You say then you are just verifying it is real. Well yeah, not like they are going to stop anyway. Worth a shot, but many times they actually respect it, unlike 15 years ago.


Email-spoofing-related records will be checked for you probably dozens of times a week if you offer a public bug bounty through any of the popular platforms. You can get all the advice on them you want for under $100 in payouts. They are among the lowest of low-hanging fruit.


What are the popular platforms for this?


Off the top of my head, HackerOne and BugCrowd.

If you're thinking of setting up a program, it's worth your time to read over some existing program policies - those policies encapsulate a lot of experience that the existing programs have had on their platforms. Your ideal policy will probably differ, but it's worth thinking about why the other program policies are the way they are.

https://hackerone.com/twitter?type=team


Thats why you use one of the all in one email software packages like mailinabox. They will configure all of this for you and show a dashboard telling you everything is good or if something is wrong.


DNSSEC is in a whole different league. But I'd be very skeptical of putting a one-liner from a generator into my DNS without understanding, at least at a basic level, _what_ it was doing. Understanding that also means you can do some manual verification - take a look at email headers to see how mail servers are responding to your configuration, etc. etc.

As for configuring a mail server, yes, that's definitely way harder. But these days, most companies outsource that to the likes of Google or Microsoft, until they get large enough to justify administering their own. There's exceptions to every rule, etc. etc., but every company I've worked at has either used G Suite or Office 365.

The result being: many companies have email services running, but don't have anybody whose day-to-day is understanding how it works and how it's secured.


> But I'd be very skeptical of putting a one-liner from a generator into my DNS without understanding, at least at a basic level, _what_ it was doing

You're completely right, of course. In my experience, these generators usually explain what the generated code means, though. These records are a lot easier to read than they are to write if all you've got is a manual and a technical specification.

I haven't used any hosted mailing services myself, but I can't imagine their control panels don't have either an option to generate the necessarily policies for you or an extensive guide on how you should configure these records and why. These records are the only part of the mail ecosystem these hosted platforms can't manage (unless you also let them do DNS) so they're a crucial step of the onboarding process.


For SPF and DKIM, yes, you're right - most hosted services will generate those DNS records for you, and validate that they're set correctly in your DNS. But in my experience, they often don't mention DMARC at all, present DKIM as a "hey just do this don't worry about it" (which, given that it's cryptography... yeah, I don't necessarily disagree with that approach, don't scare people off), and often don't provide good SPF practices (~all vs. ?all vs. -all).

On generators, a lot of them left me in an uncanny valley. They were supposed to make everything easier, but didn't explain the basic concepts, so I didn't know the values to even put into the generator. After I nailed down some of the basics, the generators just started making sense.

YMMV. There's an infinite combination of mail hosts/servers, DNS providers, and record generators. Collating all that information the first time can be overwhelming, if DNS or email aren't an area of expertise, but I'm sure there's a low-friction path out there. A Northwest Passage of email security. I'd love to find it.


Did … did you just write a blurb followed by pushing a service to do it for you exactly as the author lamented?


You're right - they're not really hard. Dealing with their consequences can be though. The real problem is that they are tacked on solutions...and that's why you end up with things like ARC (https://en.wikipedia.org/wiki/Authenticated_Received_Chain) which as far as I can see is a way of getting around the problems that DMARC creates for so many setups.


ARC addresses a very specific subset of DMARC problems, and in doing so introduces another set of problems.


They are not too hard in principle, but w.r.t implementation, even Google fails in some respects today with DMARC and its own Google Calendar (invite notifications bouncing within the same GSuite):

https://www.reddit.com/r/sysadmin/comments/c5p7e8/dmarc_bloc...


Calendar (and some other stuff) can be fixed by telling google to use the g suite domain for DKIM at https://admin.google.com/ac/apps/gmail/authenticateemail


There's a vast majority of users who struggle even with simple things like setting up an email account in Outlook. Those users will struggle with simple things like deliverability, and for them, an article like this is a godsend.

If you're capable of running your own mail server then you know more about this stuff than most people who use e-mail do, and the article isn't for you.


https://www.iredmail.org/ is another longstanding mailserver-in-a-box if you want to do self-hosting.


And yet still many many many business have SPF records that do not pass validation which IMO is worse than not having them at all


GSUITE wraps it up all nicely so they're really just a option toggle.

I wonder if any other domain providers do this?


Gandi and OVH.


Can someone suggest a Node.js mail server thay does everything for you?


Why does your mail server need to be written in node?


Because that is the ecosystem we write our code in


i use mailinabox for almost a year now and the process has been pretty painless. i did have problems with gmail for the first 2-3 months but that is mostly gone now.


+1 for Contabo. I've been a happy customer for years.


These guys wanted a passport scan and utility bill to open a tiny VPS with them. No thank you.


Actually that is a sign of a provider that takes responsibility for people abusing its service.


> Normally, this means that spf.messagingengine.com has its own SPF DNS record, which will probably list some valid IP addresses that emails can be sent from. There's also one for Mandrill, for transactional emails. You can have as many of these as you want.

This isn't strictly true (that you can have as many as you want), because SPF has a (IMO) very silly hard limit of 10 DNS lookups per record. From RFC 7208:

> Some mechanisms and modifiers (collectively, "terms") cause DNS queries at the time of evaluation, and some do not. The following terms cause DNS queries: the "include", "a", "mx", "ptr", and "exists" mechanisms, and the "redirect" modifier. SPF implementations MUST limit the total number of those terms to 10 during SPF evaluation, to avoid unreasonable load on the DNS. If this limit is exceeded, the implementation MUST return "permerror".

We see this not infrequently at Fastmail, when customers report DMARC validation problems, and the answer turns out to be that they've got too many includes in their SPF records, so SPF always fails.


That limit of ten is extremely easy to meet when someone casually says "Hey we've started using Freshdesk for ticket tracking, setup DNS please". Ok so you include:email.freshdesk.com. That record itself includes four other freshemail.io DNS lookups, and sendgrid.net, which includes another one. So you're seven DNS lookups in just for that. Gmail's recommended include:_spf.google.com includes four more, at which point the limit is exceeded and SPF is broken.


Yeah, for sure! Outlook is also very bad about this, and usually when we see it people have added include:spf.messagignengine.com (Fastmail's SPF, which is all IP addresses) and one other include, which does a bunch of recursive lookups. This is often very difficult to explain to people, and there's not even really a way to work around it, short of using different domains for different sending needs (which many people are unwilling or unequipped to do).


why can't you just periodically fetch and cache the records yourself? i.e. an ALIAS record for SPF. if the customer doesn't want to use dynamic dns, then you could even provide this service for them, of course charging a premium for the "enterprise feature".


There are services which do that, but it doesn't work in every situation. It can't (for example) cope with SPF macros, which is where the server looking up the record can replace parts of the record it is requesting with variables from the SMTP session (for example, by inserting the connecting IP address into the record lookup domain). In short, as stated in the article "email is complicated"


> why can't you just periodically fetch and cache the records yourself?

The only way this will help you is if you recursively de-reference all the records, and replace them with a list of allowed sender IPs. Now you have a new problem. You only get 255 characters in a single string in DNS. You can chain multiple 255 character strings together into a single record though, so you can get up to 4K characters. But, if you have a bunch of authorized spoofers (e.g., mailchimp) that different departments of your organization uses, even 4K characters may not be enough (source: I went through this exercise a couple years ago and our organization would have needed closer to 8K characters for the de-referenced records).

Our solution is to include our own outbound MTAs, and ?all. On inbound, we don't use SPF as a signal at all, as there are too many sites with misconfigured records.

Long term, we are trying to get each department to send spoofed mail using per-department subdomains, e.g., info@marketing.example.org, data@research.example.org, etc.

DKIM + DMARC is designed to handle the above without any of the issues SPF has.


I didn't know this! Is there an easy tool to check how many chain-on dns lookups happen?

I also typically have at least 2 for clients, including evidently google's 4. so this is important to know!


> Is there an easy tool to check how many chain-on dns lookups happen?

I use https://www.kitterman.com/spf/validate.html


https://fraudmarc.com/spf-record-check/

If your spf is too heavy, add universal spf and then watch it work by either sending a test message or rechecking your spf with https://vamsoft.com/support/tools/spf-policy-tester


thank you!!


> they've got too many includes in their SPF records, so SPF always fails.

Any domain can toss "v=spf1 include:UniversalSPF.org -include:x.UniversalSPF.org" in front of their broken SPF record to automagically clean and fix it. This authorizes mail that the domain owner expects to pass, and fails the mail they expect to fail.

Source: One of my startup's covid projects was creating and giving away https://UniversalSPF.org. We'd already been providing SPF Compression commercially since c. 2015.

It's free to use and already trusted by several hundred businesses.

Here's a good spf evaluator in case you want to see universal spf fix your domain's policy: https://vamsoft.com/support/tools/spf-policy-tester and a more technical deep dive for you other command line geeks: https://fraudmarc.com/introducing-universal-spf/


Outsourcing security is a very stupid thing to do. Of course copying a line you don’t understand into your DNS isn't that much better but at least you control it.

In this case configuration is handled by an unknown entity that you no contractual obligations to you. Don’t do it.


How does it work?

Based on a cursory inspection, it looks like UniversalSPF makes use of a custom DNS server and the the %{i} and %{o} macros for SPF.

Before this, I wasn't even aware that SPF had macros!


Quick, somebody prove it's Turing-complete!


It works by inactivating spf terms that cause errors then serving out a cleaned up and compressed policy for you on the fly.


Holy, I thought that there was a draft to change the limit to 10 different base domains, it seems to have died.


Not to overly shill my own free solution here but quite a few people use Universal SPF (a little string you put in front of a traditional spf record) to overcomes the 10 dns lookup limit.


Related: if you have a domain from which you never want to send email, you can also set up these records to tell receivers to reject everything.

Here is a guide: https://www.gov.uk/guidance/protect-domains-that-dont-send-e...


I like that concrete mail advice is two subpath levels deep into "the" UK government website. High text contrast, loads fast. Kudos to them.


This is worth doing, as certain "vendor security vetting" services will check these settings on vanity/convenience domains they manage to associate with your organization.


I'm going to refer to this post whenever says "running your own mail server is easy!".

I suppose that if you already know how to run your own mail server and are familiar enough with all the moving parts, sure, it's easy. But if you have a problem that needs to be solved, and you think the answer is running your own mail server, you now have two problems.

The "unholy combination of silly standards" is an excellent way of putting it, but it goes back to the 1970's. SMTP was originally based on FTP. Anonymous FTP, at that. It was designed to send text- nothing more. And really that's all it can do still. We can thank Base64 for binary attachments that are at 30% bigger than their unattached size.

Yeah, mail is a problem. And it always will be because it's a 50 year old technology that was designed for a collection of a few hundred servers ran by technologists who trusted each other. SPF, DKIM, and DMARC (and literally everything after except for "send text to so and so") is an afterthought.


These aren’t even "run your own mail server" territory. It’s stuff that you already need to worry about when simply using your own domain.


Running a mail server is very easy. Setting it up to begin with is the hardest part. Email isn't exactly moving fast, so it's basically set and forget if you don't have to deal with it on behalf of untrusted parties.

That said, a technically inclined person would probably be able to set up a server in a day or three and understand the basics of how it all ties together.

Just because not that many people understand how to set it up doesn't make it hard. But it's definetly showing it's age and could be much simpler to use.


It’s definitely not set and forget. In the past year alone there where multiple 0day attacks against Exim and MS Exchange servers.


Use sovereign instead of setting everything up by yourself. You should know the basics of linux system administration, however


Thank you. We need more of this. Too many "how-tos" are written with too many assumptions and not enough common sense, straightforward explanations.

I've set up and run all of these, but that doesn't mean that it doesn't frustrate the heck out of me whenever I have to deal with them. A good, handy reference like this will be quite useful.


Also infuriating: Not all email services treat DMARC in the same way, or way you'd expect (eg: O365 will by default deliver a p=reject to spam https://docs.microsoft.com/en-us/microsoft-365/security/offi...)


The title seems a bit dramatic but I can see how this sort of stuff would throw anyone without a history of network or systems administration for a loop.

This is one thing I really enjoyed about setting up ProtonMail with my personal domain. They make it really simple to validate that SPF, DKIM, and DMARC are set up properly.


I didn't realise that ProtonMail had DMARC built in. I'm kind of surprised more email providers aren't including DMARC report analysis out-of-the-box, to be honest.

If Google Workspace/G Suite started pushing admins to use DMARC, and provided tools to make it easy, so many more domains would be protected, particularly in startup world.


Ok, we've changed it to the (slightly less dramatic?) subtitle. Not sure which is the better choice here...


I agree about Protonmail. Never used their service, helped a client straighten their records for a project. No issues.

Tutanota are just as straightforward, but they're both in the security game so not much of a surprise there.


When implementing spam filtering for my email forward app (https://hanami.run) I was super confused at DMARC and as the author said. The DMARC keyword were all SEO-focus content by companies that aim for that keyword and didn't clearly give a definition of DMARC.

Another confusion is the ARC, the acronym was alike so I though ARC has some relation to DMARC where they are totally different :).

I think to help new people understand them, we should stop group DMARC with SPF/DKIM. SPF and DKIM are the thing you set up and you control. But DMARC is just a policy to tell the recipient what they should do if something failed SPF or DKIM. And it's totally up to recipient's mail server to support and follow standard of DMARC.


DMARC basically breaks mail intermediaries and is fundamentally problematic, hence the low adoption rate.

ARC is a much better alternative that builds on DKIM and allows intermediate custodians of mail.


No it doesn’t, unless said intermediaries are modifying contents and are not DMARC aware.

Many European countries are forcing DMARC adoption for government infrastructure and it works just fine.


What does intermediate custodian mean here? Something like sendgrid? Is there an issue for normal peoples email on custom domains (eg fastmail or gmail hosted)?


I think he is talking about mailing list servers. When you email mailman, it'll add a line to the bottom of the email, manipulate some to: and cc: headers, and forward it along. This changes the hash of the email which breaks some crypto signatures.


> And never forget to use -all

And yet even Googles docs recommend ~all.

If you're using 3rd party bulk senders you probably want to consider the cost and benefit of -all vs ~all

https://support.google.com/a/answer/10684623?hl=en


3rd party bulk senders likely know more about mail delivery than you do. Also DMARC aware recipients will treat ~ALL in SPF exactly like -ALl.


> nothing demonstrates this more simply than this PHP script

Arguably there's nothing more clear than writing it manually over telnet. If you haven't done so yet it's kind of fun to try it out once.


The thing to remember is that these are useless unless the recipients MTA is configured to take action against failures. Another thing to understand is that very few services, commercial, public or otherwise, send DMARC reports, and many that do send very sparce reports that are as bad as not recieving anything at all. Fundamentally, catching spoofing, phishing and spam and not catching 'ham' is actually really hard.


It's been years since I did it, but I added DMARC to a high volume domain and the reports were useful while I was adding it, to help make sure I didn't forget any authorized senders, but once I got that finalized, the reports were totally unactionable.

I can't do anything about the attempted spoofs; I'm not going to track down everyone's open relays, and if I would, DMARC reports aren't really enough anyway.

At the time, Yahoo, Google and Microsoft all sent reports, which is a good portion of email, although certainly missing a lot. I think there were a few other smaller names, which I no longer remember.

Of course, it almost doesn't matter. Spam/phishing mail to our users still continued, they just stopped spoofing our address. It's not like very many people look at the domain mail claims to be from anyway; modern clients hide it, too.


> they just stopped spoofing our address.

This also means that emails from your domain is less likely to get marked as spam, which can be a significant win for smaller domains.


The article is good, but misses an important point (that adds yet another confusing thing about email): the difference between Header.From and Envelope.From and how DMARC also solves this by requiring identifier alignment.

Without it, generally only partial validation of the envelope address happens by i.e. SPF.


>Then, I encountered a second problem: I had no idea what those were, and seemingly nobody has written about SPF, DKIM, or DMARC in a way that a human can understand, not to mention implement. Every article I found was either highly technical, trying to game SEO to sell me something, or too high level to be useful.

I guess TFAuthor hasn't done a cursory google search, because if they had they'd have found the ISPmail tutorial[0] which is clear, concise, mostly correct, and approachable if you aren't a total novice to Linux.

[0] https://workaround.org/ispmail


A controversial headline and a bit of opinion creates controversy and gains views. Once this sort of information would be written in a plain text HOWTO and posted to usenet but now we have to clickbait everything so it gets posted to HN.

SMTP is hopelessly naive by the standards of today as it was designed to work in a much smaller and more trusted environment. The additional reputation protocols are well intentioned and work well if the receiver acts on them and are not at all silly or unholy. They are not difficult to configure but neither are they as simple as a redesigned protocol might be.

If you don't need to setup your own mail servers or have a desire to develop and maintain skills in that area then use a pre-configured service.

Otherwise I am fairly sure that configuring these things is far simpler than most trade skills. I don't think there is any comparison between the time it would take to read some docs and add a few lines to a config file and learning to become a proficient welder. If you work with servers and want to be good at your craft I think email is still one of the services people should know how to configure and secure.


This is a really nice article! Just so you know, it says that Postmark doesn't offer a DMARC service, but they actually offer two: Free DMARC Monitoring (https://dmarc.postmarkapp.com/) and DMARC Digests (https://dmarcdigests.com/)


Well this is embarrassing. Thanks for pointing that out - I've corrected the article and cited you.


My pleasure! Sorry to be pedantic :)


There’s a minor technical error in the description of DKIM “happy path”. Here’s what step 4-6 should be:

> 3. The Fastmail SMTP server generates a signature using the secret key, and attaches it to the email, along with a ‘selector’ (a string specifying which which of many possible secret keys key it used), then sends the email to example.com's receiving server.

> 4. Google email server receives this email. It's from sadl.io and has a fastmail DKIM ‘selector”, so it gets the DNS records for that selector in that domain.

> 5. The email has a signature and selector embedded, and the DNS records for the selector in the sadl.io domain declares a public DKIM keys which can be used to verify that signature.

> 6. That DKIM keys matches the one used to make this signature. So the DKIM test passes.

Crucially, there is no way to list all existing DKIM keys for a domain without knowing all the selector strings. Of course you could do a brute-force search and find a lot of them, but you could never know you found them all without doing a full zone transfer (or if DNSSEC is used with the old NSEC records, which allows enumeration).


A note on the PHP script: it will only work if that PHP server has been configured to send email properly.

If you just have a random Linux server that someone has run "apt-get install php" on, it probably hasn't been configured and so that command won't do anything. eg https://www.quackit.com/php/tutorial/php_mail_configuration....

Ps. Because you can't rely on PHP built-in email functions being configured properly and because apps generally want more control over how email is sent most modern apps will use their own library that makes SMTP connections itself. For example here is Symphony - note the first section on setting up transports: https://symfony.com/doc/current/mailer.html#using-built-in-t...


From what I understand, this is incorrect:

> SPF and DKIM are used as indicators of whether an email is spoofed or not. But if you added an SPF record on your domain, and you forget to add one of your email systems - say, Postmark, which you use to send mission-critical notifications from your application to your customers - then your customers could stop getting emails. If you added DKIM keys to your domain, but one of your email services doesn't support DKIM - or you forgot to add DKIM keys for that service - your customers could stop getting emails.

Setting up DKIM alone for one email service should still allow mails sent by any other email service (i.e. one not using DKIM) to be delivered. (That is, of course, if DMARC is not also set up with a strict policy.)

That is to say, DKIM is not that risky to set up and use, by itself.


SPF = This list of servers are authorized to send email for my domain.

DKIM = This specific email can by verified/authenticated by my DNS.

DMARC = What to do with email that doesn't comply with above. And how to tell me about it (if you want to).

SPF is basically useless due to the centralization of email providers. It was created to be locked to a company specific email server. E.g. Only accept mail from exchangeserver.company.com, and reject everything not from that server. Now everyone authorizes Microsoft or Google to send their mail.


Technically SPF is still useful if you trust that Microsoft and Google aren't allowing arbitrary email to be sent from non-verified email addresses, which it seems both do (although neither seem to require periodic re-verification).


SPF (alone) is useless because it actually does not check the sender's domain in the From header (as one might naively think). Instead it only verifies Envelop Sender which can differ (intentionally) from the mail seen in From header.


I use a service called OnDMARC (https://ondmarc.redsift.com/)for managing email spoofing/delivery. They have a suite of tools for managing and monitoring spf/dkim/dmarc. Nothing you couldn’t do yourself, but they make it easy. They also have a couple of cool extras like “dynamic spf” which lets you expand beyond the standard spf record length.


Can SPF, DKIM, and DMARC be CNAMEs? It’s a little bit easier when one can move all this stuff onto one host, away from the rest of ones infra, especially when it comes to distributing DKIM keys.

I wouldn’t be surprised if there’s also a mail daemon that also did DNS for you. Everything in one simple MTA. Begone, Exim4 mega config and update-exim4.conf.conf.conf!


DKIM and DMARC can be CNAMEs, but not SPF, since SPF lives at the apex domain name, which can not be a CNAME. (Of course, if your email address is on a subdomain, then that can be a CNAME. But then the MX records will also have to be moved to the CNAME target.)


Not even then, RFCs state that a CNAME may not exist with any other RR type, and your apex domain needs at least SOA and NS records. A CNAME on the apex domain may kind of work, but it will present as broken in subtle and unexpected ways.


Um, what? I think you misunderstood me. Or you are unaware that DMARC and DKIM records live on subdomains.


DKIM and DMARC can be CNAMES, many services already use this, at Fastmail for example for DKIM you setup CNAMES, and we manage DKIM keys for you. SPF needs to be on your (root/organizational) domain, and you don't want (can't have) a CNAME there.


Some companies just mark incoming email with something like [external] to solve the problem used as an example. In the end you are going to know if an email came from your own server, you are the one sending them.

If it is really that important, businesses should be signing their emails.


Eh, DMARC is meant to solve different problem. E.g. when your accountant receives spoofed mail with a fake invoice supposedly coming from a legit supply chain provider. They might be trained to check domain that was used to send email but without DMARC non-techies won’t be able to notice well made spoofed email.


Assuming that the email client used even shows the domain. If you can train your employee to do an obscure technical check you can for sure train them to verify invoices. After all, if you don't then the legit supply chain provider can just generate fake invoices.


I still run my own router at home, and my own servers of various kinds. I put a lot of thought into my packet filter configuration for example, and overall my handwritten router and server stuff comes at a maintenance cost that I am happy to pay.

Not so for email. It just stopped being worth it about 10 years ago or so. Regularly fine-tuning your configuration to not only not be marked as spam in originating mails, but to also filter incoming spam, walking the thin line between filtering too much and too little, was just no fun.

I looked out for an email provider that provides the flexibility that I'm used to (multiple domains, catchall for subdomains, sieve filtering etc.), pay them a few dollars per month or year, and haven't looked back. My DNS is still homegrown and I handwrite my zones, but the MX just points to my provider.


Does using SPF -all mean that if you send a mail to someone, they won't be able to resend/bounce/redirect that mail to someone else since the resending process uses their mailserver instead?


It breaks forwarding services that don't use SRS [1], for example OVH's. Like you mention, the next hop (example: gmail) will see it coming from the redirecting domain which is not valid per SRF rules. A SOFTFAIL will send a lot of emails to spam, a HARDFAIL means the email might be rejected outright.

[1]: https://en.wikipedia.org/wiki/Sender_Rewriting_Scheme

I need to move my domains off of OVH...


That should be fine. Forwarding means you’ll become the sender so the SPF record of the original sender will no longer apply.


I'm not talking about forwarding, but resend/bounce/redirect, which means the original sender stays in the From header and the entire email is identical to the original one.


I'm less upset with SPF, DKIM, and DMARC and more irritated that you can't find any 5 minute guide on how to set it up. As this article states, it's all SEO bullshit and DNS validators that don't explicitly say, "Add this TXT record to your DNS set. This directive means X. That directive means Y."

Unfortunately, as much as I sympathize with this article, I can't pass this around to colleagues as technical reference, because it's just a rant. A rant with helpful information, but a rant nonetheless.

I need reference documentation that isn't going to disappear when the author decides to change blogging platforms, etc. So, below are the RFCs, with sections, that actually matter.

DKIM and DMARC are not required to have a functional piece of software that sends emails today, though. Just use SPF and you should be fine. A TXT record with @ for the current origin[1] or the relevant host with the following is sufficient[2]:

    v=spf1 include:example.com ~all
You can use multiple includes.

If you're pointing to your own DNS records, though, don't use include. Use a. As far as I understand it: include[3] is others, or as RFC 7208 states it, 'independent domains.' a[4] is you (A records).

In this case, it looks like this:

    v=spf1 a:mydomain.com ~all
That's it. Now your mailer should work.

So for example, you've got a domain name on Namecheap, and you're hosting on a VPS.

You've pointed your A record for @ to your VPS' IP address. That's all you need there, and all you need for SPF is a TXT record pointing for the current domain, @, and the string above, with the a mechanism pointing to your domain name.

SPF-compliant software will look up your TXT record containing the SPF string, see the domain, lookup its IP address, and match it to the server where your email is coming from, then allow it in transmission.

Popular mailers should tell you all this these days, but they don't, and it really sucks.

[1]: https://datatracker.ietf.org/doc/html/rfc1035#section-5.1

[2]: https://datatracker.ietf.org/doc/html/rfc7208#section-3

[3]: https://datatracker.ietf.org/doc/html/rfc7208#section-5.2

[4]: https://datatracker.ietf.org/doc/html/rfc7208#section-5.3


> That's it. Now your mailer should work.

Work... technically... yes. If the definition of 'works' is: "I press send and sometimes my email reaches it's destination".

The spirit of the article IMO implies a definition akin to "I press send and my email reaches it's destination. And... my emails have some protection against spoofing attacks".

So in the spirit of the article, I would recommend your 1-liner takes a small yet significant change:

v=spf1 include:example.com -all

Now... let's talk about biggest barrier to entry for n00bs (myself included) in the self-hosted email game: non-blacklisted ip-address space. And blacklist here refers not just to public lists (SPAMHAUS etc) but private ones too (e.g., outlook).


Your configuration provides no protection from email spoofing though.


In practice, that's not true. Major email providers will not respond any differently to ~all versus -all, despite what the RFCs state you should do.


That’s not what I meant. SPF checks Envelope Sender domain which does not have to be the same as the address seen in From header.

Your configuration is trivially spoofable by setting Envelope Sender to something attacker controls (and thus it will pass SPF), while still placing your domain in the From header.


Can someone explain why you can assume a recipient would respect DMARC when dealing with SPF but not check your DKIM? It never made sense to me why we need anything other than DKIM honestly.


Sure. Parsing DMARC requires understanding DKIM as well, so what you’re asking is a non issue.

That said DKIM is not enough because that standard does not have a way to signal recipient that your domain has DKIM set up in the first place (and you promise that all your mails should always be signed).

It’s kind of funny but essentially hacker does not need to spoof DKIM because they can just omit it and recipient won’t be able to know that it should have been present in the first place. Btw, there was proposal to add a feature that could be used for this signaling but it didn’t get adopted, so DMARC is the only practical solution right now.


> Btw, there was proposal to add a feature that could be used for this signaling but it didn’t get adopted, so DMARC is the only practical solution right now.

Wow, thanks. That definitely explains this mess.

Do you know why it didn't get adopted? I'd have thought that in a sane world all you'd need is to literally have a DNS record that specifies the DKIM info... pretty simple. Why on earth did people find it more convenient to do it in such a convoluted fashion instead? Especially now that there's ARC too, which I'd have thought shouldn't be necessary either.


Corporate politics and death by committee. These standards were created by many parties with conflicting interests. My understanding is that they couldn't agree on an exact way From header should be checked and thus deferred all policy aspects to further standard.

If you read DKIM RFC it actually only specifies how to sign and verify the email signature, but does not mandate exact checks recipient should do to figure out authenticity. E.g. technically Microsoft could send mail from @gmail.com address and use "d=outlook.com" in the signature - the signature would be valid, but obviously it wouldn't make it authentic (meaning such mail wouldn't be authorized by Google). Although common sense dictates that there should be some sort of connection between domain seen in From header and the one that's DKIM-signing the mail (and most real-world implementations will do some kind of checks), RFC deliberately does not standardize these steps.

The intended standard that should have clarified these issues was ADSP, but it wasn't well received and now we have DMARC which handles both SPF and DKIM together (this is better for deliverability as mails that fail one of these checks might still pass another).


Do you know any free email providers for custom domains that who provide DKIM?


lol, that's cause it's mostly for mail coming from your server. You probably don't see any of it


SPF, DKIM, DMARC... all useless.

I somehow live without all those for what, since before they appeared. 2001 in fact. With my own MX.

They are just ineffective bandaids.


As long as you don't email anyone who uses a gmail address or uses google's email hosting (which is the majority of internet users now?) then you can get away with that. I did too (also having run an MX since around 2001)... but a year or so ago google started filtering all email which didn't comply into spam.. so.. had no choice but to set it up.


What's with Google? It's not like Google is the Internet.

About everyone I feel a need to connect with understands that, so they don't use gmail, at least exclusively.

If a company trying to hire me can't receive email from me "because google" - then to hell with them, period. They are incompetent.

Although I must say there were no lost, rejected or diverted to spam messages from my deliberately ignorant MX to gmail for the last year.


Everyone I know who isn't tech-literate uses Gmail because (a) Gmail is/was better than their ISP's email in many ways, including UI and spam filtering, and (b) it's hard to switch off without dedicating a few hours to changing your email on a bunch of services (and there are a lot of services that don't offer email changes at all) - also, if someone hasn't use a password manager religiously, chances are they don't even have a convenient list of all the services they're signed up for.


83% of spam I received last month failed at least one of these checks, while less than 2% of ham did. They're certainly useful.


IDK maybe it's my tendency to keep a low profile. Or some other perceptual bias.

But in last 5 years levels of spam never even remotely approached levels what I remember from 2000s, when my job was actually fighting it (at an ISP).

All this tech (SPF, DKIM, DMARC) came and went ... somewhere, and there was exactly zero impact in spam rates on the handful of domains I now maintain, when I did take some time to implement them.

Nowadays I have a couple of my own domains, with nothing antispam-configured, and an gmail account, and you know what -- spam amounts almost exactly match.

Hence the conclusion - it's not worth it to invest time into it.


Maybe try running a business?


The point of these is to tie email reputation (aka. How much spam you send) to the domain sending the mail instead of the IP address doing the sending. You can still send mail without these set up, but real spammers might see it missing and impersonate your domain in their spam and tank your reputation.


Yet without DMARC, we would be far worse off.


I doubt that.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: