Andrew Ayer

I recently heard from someone, let's call them Alex, who was unable to email comcast.net addresses. Alex's emails were being bounced back with an MTA-STS policy error:

MX host mx2h1.comcast.net does not match any MX pattern in MTA-STS policy MTA-STS failure for Comcast.net: Validation error (E_HOST_MISMATCH) MX host mx1a1.comcast.net does not match any MX pattern in MTA-STS policy MTA-STS failure for Comcast.net: Validation error (E_HOST_MISMATCH)

MTA-STS is a relatively new standard that allows domain owners such as Comcast to opt in to authenticated encryption for their mail servers. (By default, SMTP traffic between mail servers uses opportunistic encryption, which can be defeated by active attackers to intercept email.) MTA-STS requires the domain owner to duplicate their MX record (the DNS record that lists a domain's mail servers) in a text file served over HTTPS. Sending mail servers, like Alex's, refuse to contact mail servers that aren't listed in the MTA-STS text file. Since HTTPS uses authenticated encryption, the text file can't be altered by active attackers. (In contrast, the MX record is vulnerable to manipulation unless DNSSEC is used, but people don't like DNSSEC which is why MTA-STS was invented.)

The above error messages mean that although mx2h1.comcast.net and mx1a1.comcast.net are listed in comcast.net's MX record, they are not listed in comcast.net's MTA-STS policy file. Consequentially, Alex's mail server thinks that the MX record was altered by attackers, and is refusing to deliver mail to what it assumes are rogue mail servers.

However, mx2h1.comcast.net and mx1a1.comcast.net are not rogue mail servers. They are in fact listed in Comcast's current MTA-STS policy:

version: STSv1 mode: enforce mx: mx2c1.comcast.net mx: mx2h1.comcast.net mx: mx1a1.comcast.net mx: mx1h1.comcast.net mx: mx1c1.comcast.net mx: mx2a1.comcast.net max_age: 2592000

This means that Alex's mail server is not consulting comcast.net's current MTA-STS policy. Instead, it's consulting a cached policy which does not list mx2h1.comcast.net and mx1a1.comcast.net.

This can happen because mail servers cache MTA-STS policies to avoid having to re-download an unchanged policy file every time an email is sent. To determine whether a domain's policy has changed, mail servers query the domain's _mta-sts TXT record (e.g. _mta-sts.comcast.net), and only re-download the MTA-STS policy file if the ID in the TXT record is different from the ID of the currently-cached policy.

The obvious implication of the above is that if you ever change your domain's MTA-STS policy, you have to remember to update the TXT record as well.

A more subtle implication is that you have to do the updates in the right order. If you update the TXT record before changing the policy file, and a mail server fetches the policy in the intervening time, it will download the old policy file but cache it under the new ID. It won't ever download the new policy because it thinks it already has it in its cache.

This pitfall could have been avoided had MTA-STS required the ID to also be specified in the policy file instead of just in the TXT record. That would have prevented mail servers from caching policies under the wrong ID.

There's some evidence that this is what happened with comcast.net. The ID in the _mta-sts.comcast.net TXT record appears to be a UNIX timestamp (seconds since the Epoch):

_mta-sts.comcast.net. 7200 IN TXT "v=STSv1; id=1638997389;"

That timestamp translates to 2021-12-08 21:03:09 UTC.

However, the Last-Modified time of https://mta-sts.comcast.net/.well-known/mta-sts.txt is three minutes later:

Last-Modified: Wed, 08 Dec 2021 21:06:05 GMT

If the ID in the TXT record reflects when the TXT record was updated, there was a three minute gap between the updates. If Alex's mail server fetched comcast.net's MTA-STS policy during this window, it would have cached the old policy under the new ID, causing the errors seen above.

Recommendations for Domain Owners Who Use MTA-STS

You should automate MTA-STS policy publication to ensure that your MTA-STS policy always matches your MX records and that the TXT record is reliably updated, in the correct order, when your policy changes. If your policy file is served by a CDN, you have to be extra careful not to update the TXT record until your new policy file is fully propagated throughout the CDN.

I further recommend that you rotate the ID in the TXT record daily even if no changes have been made to your policy. This will force mail servers to re-download your policy file if it's more than a day old, which provides a backstop in case something goes wrong with the order of updates.

It may be tempting, but you should not reduce your policy's max_age value as this will diminish your protection against active attackers who block retrieval of your policy. Having a long max_age but a frequently rotating ID keeps your policy up-to-date in mail servers but ensures that in an attack scenario mail servers will fail safe by using a cached policy.

It's quite a bit of work to get this all right. If you want the easy option, SSLMate will automate all aspects of MTA-STS for you: all you need to do is publish two CNAME records delegating the mta-sts and _mta-sts subdomains to SSLMate-operated servers and SSLMate takes care of the rest.

Recommendations for Mail Server Developers

You should assume that domain operators are not going to properly update their TXT records and you should always attempt to re-download policy files that are more than a day old, regardless of what the ID in the TXT record says.

Is this Really Better than DNSSEC/DANE?

Thanks to MTA-STS' duplication of information and requirement for updates to be done in the right order, there is a high chance of human error when MTA-STS is deployed manually. Unfortunately, it's very likely to be deployed manually because there's a dearth of automation software, and on the surface it looks easy to manage by hand. To make matters worse, MTA-STS' caching semantics mean that the inevitable human error leads to hard-to-diagnose problems, such as a subset of mail servers being unable to mail your domain. I suspect that many problems will never be detected - email delivery will just become less reliable than it was before MTA-STS was deployed.

Meanwhile, DNSSEC is increasingly automated, and if you use a modern cloud provider like Route 53, Google Cloud DNS, or Cloudflare, you don't have to worry about remembering to sign zones before they expire, which was traditionally a major source of DNSSEC mistakes.

However, not all mail server operators support DNSSEC/DANE. Although Microsoft recently added DNSSEC/DANE support to Office 365 Exchange, Gmail only supports MTA-STS. Thus, there is still value in deploying MTA-STS despite its flaws. But we should not be happy about this state of affairs.