Secure Email Transport: STARTTLS and DANE

How STARTTLS works

A sending server looks up your MX records, connects to one of the hosts on port 25, and starts a plain-text SMTP conversation. If your server lists STARTTLS among its capabilities, the sender asks to upgrade, both sides run a TLS handshake, and the rest of the session, including the message, is encrypted.

S: 220 mx1.example.com ESMTP
C: EHLO mail.sender.example
S: 250-mx1.example.com
S: 250-PIPELINING
S: 250-SIZE 52428800
S: 250-STARTTLS
S: 250 8BITMIME
C: STARTTLS
S: 220 2.0.0 Ready to start TLS
   … TLS handshake; the session restarts inside the encrypted channel …
C: EHLO mail.sender.example

You can run the same conversation yourself to see the certificate an MX host presents:

$ openssl s_client -starttls smtp -connect mx1.example.com:25 -servername mx1.example.com -brief
CONNECTION ESTABLISHED
Protocol version: TLSv1.3
Peer certificate: CN = mx1.example.com
Verification: OK

STARTTLS on port 25 is different from the ports mail clients use. Clients submit mail on port 587 with STARTTLS or port 465 with TLS from the first byte, and they always check the certificate. The weaknesses below apply to the server-to-server hop on port 25.

Why opportunistic TLS is not enough

Port 25 STARTTLS is opportunistic by design. A sender uses TLS when it can and quietly delivers in plain text when it cannot, because refusing would have stranded mail to the many servers that once had no TLS at all. That leaves three gaps an attacker on the network path can use:

Stripping is not theoretical. Researchers and the EFF documented networks and security appliances removing STARTTLS from SMTP sessions in the mid-2010s, which is what prompted both DANE for SMTP and MTA-STS. Each of the three attacks has to be closed by information the sender gets before it connects. DANE gets it from DNSSEC; MTA-STS, covered in MTA-STS and TLS-RPT Explained, gets it from an HTTPS policy file.

How DANE for SMTP works

DANE, short for DNS-Based Authentication of Named Entities and defined in RFC 6698, stores a fingerprint of a server's certificate or key in a TLSA record. For mail, the record sits at _25._tcp under each MX host name. A sending server that validates DNSSEC follows these steps for every delivery:

If any step fails after a signed TLSA record has been found, the sender does not deliver to that host. It tries your next MX host, and if none of them pass, it keeps the message queued and retries, the same way it would after a temporary network error. That closes all three gaps: the MX answer is signed, the TLSA record tells the sender that TLS is mandatory, and only the published certificate is accepted.

$ dig +dnssec +noall +answer +comments TLSA _25._tcp.mx1.example.com
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
_25._tcp.mx1.example.com. 3600 IN TLSA 3 1 1 8d02536c887482bc34ff54e41d2ba659bf85b341a0a20afadb5813dcfbcf286d
_25._tcp.mx1.example.com. 3600 IN RRSIG TLSA 13 5 3600 …

The ad flag shows the resolver validated the DNSSEC chain. Without it, a sender would ignore the record.

Reading a TLSA record

The three numbers before the hash say what is being matched and how.

A 3 1 1 record pins the MX host's public key. With usage 3, the sender checks only that key, not the certificate's names, issuer or expiry date, so an expired or self-signed certificate still works. That is deliberate: DNSSEC is the trust anchor, not the public CA system. To produce the hash from the certificate your MX serves:

$ openssl x509 -in mx1.pem -noout -pubkey \
    | openssl pkey -pubin -outform DER \
    | openssl dgst -sha256
SHA2-256(stdin)= 8d02536c887482bc34ff54e41d2ba659bf85b341a0a20afadb5813dcfbcf286d

To check the key the MX host actually serves against your published record, give the record data to openssl s_client. OpenSSL then runs the same DANE check a sending server would:

$ openssl s_client -starttls smtp -connect mx1.example.com:25 -brief \
    -dane_tlsa_domain mx1.example.com -dane_ee_no_namechecks \
    -dane_tlsa_rrdata "3 1 1 8d02536c887482bc34ff54e41d2ba659bf85b341a0a20afadb5813dcfbcf286d"
CONNECTION ESTABLISHED
Protocol version: TLSv1.3
Peer certificate: CN = mx1.example.com
Verification: OK
DANE TLSA 3 1 1 ...db5813dcfbcf286d matched EE certificate at depth 0

A match prints the DANE TLSA … matched line. A mismatch prints verify error:num=65:no matching DANE TLSA records and no Verification: OK. -dane_ee_no_namechecks matters: RFC 7672 tells SMTP senders not to check the certificate's names for a usage 3 record, and without it OpenSSL reports hostname mismatch for a deployment senders would accept. Pass one -dane_tlsa_rrdata for each TLSA record you publish, so a "current plus next" pair is checked as senders see it.

Malformed records are a common failure, and they fail quietly. RFC 7672 treats a record with a usage outside 0 to 3, a hash of the wrong length, or non-hex data as unusable. When every TLSA record for a host is unusable, a DANE sender still insists on TLS but no longer checks the certificate, so mail keeps flowing while the protection DANE was meant to add is gone. Only a mismatch against a usable record makes senders hold mail. The record syntax is shared with HTTPS, as described in DANE for HTTPS.

Rolling keys without losing mail

Because senders refuse to deliver on a mismatch, the TLSA record must always match the key the server is presenting, including during a change. The safe pattern is to publish the next key before you use it, sometimes called "current plus next":

1. Generate the new key pair and compute its 3 1 1 hash.
2. Add a second TLSA record with the new hash, keeping the old one.
3. Wait at least two TLSA TTLs so cached copies of the old record set expire everywhere.
4. Switch the MX host to the new key and certificate.
5. Remove the old TLSA record once the new key is confirmed working.

DNSSEC itself is now part of your mail path. An expired signature or a broken DS record makes TLSA lookups fail, and DANE senders will hold mail until it is fixed. DNSSEC Key Rollovers and Debugging DNSSEC Validation Errors cover how to keep the chain healthy.

DANE, MTA-STS, or both

The two standards solve the same problem with different trust anchors. DANE protects even the first delivery and does not depend on the public CA system, but it requires DNSSEC on every zone involved. MTA-STS works without DNSSEC but trusts a policy fetched over HTTPS on first use. Different large senders implement different subsets, so publishing both gives the widest coverage, and when a sender supports both, a DANE result takes priority.

Whichever you choose, publish a TLS-RPT record first. TLS-RPT is a separate standard, so only senders that also implement it will report, but several large providers do. Their daily reports of failed TLS sessions can show you a broken certificate or TLSA record before it becomes lost mail. The DANE for Email checklist is a short reference for the records involved.

How DNS Panopticon helps

Domain Visualizer looks up TLSA records at _25._tcp under each MX host found in its live scan. It reports when MX records exist without any SMTP TLSA records, when a TLSA record is malformed, for example with an out-of-range usage or a hash of the wrong length, and when valid records are present. The same scan reports DNSSEC signals for the domain and checks for an MTA-STS record, so you can see both transport protections in one place.

Domain Visualizer reads DNS; it does not open SMTP sessions. Use the openssl s_client DANE check above, with -dane_tlsa_rrdata, to confirm that the key each MX host actually presents matches its TLSA record.