Problems with .de domains: What is known so far

The evening of May 5th was not a pleasant one for many administrators managing services and websites with .de domains: Shortly before 10 PM, monitoring systems sounded the alarm, customers and employees triggered support cases, and troubleshooting began – websites were unreachable, apps malfunctioned, and VPN connections failed. The cause, however, was not with the service operators, but at a central point: in the Domain Name System (DNS) of the .de zone, specifically in its DNSSEC configuration.

DENIC eG is responsible for the configuration, which has so far only provided brief information about the incident[Link auf https://blog.denic.de/technische-storung-bei-de-domains-behoben/]. Although the dust has settled by Wednesday, the disruption is resolved, and some details about the events are becoming clearer. However, a look at the DNS data also shows: only DENIC can explain the exact cause, and a statement is still pending. To reconstruct the events, we examined the historical DNS entries recorded by the service dnsviz.net.

What happened

DNSSEC's task is to protect DNS responses from manipulation with digital signatures. Without DNSSEC, attackers could forge responses if they intercept and alter the traffic between client and resolver. DNSSEC works with asymmetric cryptography, meaning with key pairs of public and private keys. The public keys are stored in an entry of type DNSKEY in the DNS. For signing responses, there are longer Zone Signing Keys (ZSKs), which are in turn signed with a shorter Key Signing Key (KSK).

The integrity of a DNS response is checked step-by-step, starting from the back, beginning with the root zone, whose keys the requester must trust. The root zone, digitally signed, points to the responsible name servers of the top-level domain – in this specific case, .de. If they provide a valid signed reference to the name server responsible for that domain, it is queried. If a signature is faulty along the way, the entire chain is considered faulty, and name resolution fails. This is the desired behavior that protects against manipulation.

At regular intervals, the Zone Signing Keys at the top-level domain level are exchanged. Because this is a central step with far-reaching consequences, it happens in several stages: On May 2nd, DENIC, as the entity responsible for the .de zone, announced a new public key with ID 33834. This was done in a timely manner so that the new entry could propagate through the DNS. Initially, signing was not done with the new key; the old key 32911 handled that. Key 33834 first appeared as a signing key on May 5th at 9:43 PM (7:43 PM UTC) in a signature (RRSIG) for the SOA record of the .de zone. SOA stands for "Start of Authority," and the record contains information about the zone itself. This signature was invalid for reasons yet to be clarified. The data from dnsviz shows: all 6 responsible name servers delivered this defective signature with key 33834 at that time.

Around 9:59 PM, the first countermeasures were observed: one of the name servers, n.de.net, started delivering a new RRSIG record for the SOA record with a valid signature, signed with the new key 33834. The other five servers continued to propagate the incorrect signature.

By 10:27 PM, a new picture emerged: n.de.net again delivered an invalid signature, and now a.nic.de and z.nic.de provided valid entries with the old key 33834 – but different ones via IPv4 and IPv6. z.nic.de also showed a defective signature concurrently. At 10:31 PM, the next state occurred: five servers were now capable of correctly signing with the new key 33834, and only n.de.net, which had previously managed this feat, was incorrect with its entry. Just three minutes later, all servers delivered incorrect responses for a change, with different combinations coming from the servers at short intervals, all of which were invalid.

By 10:50 PM, most servers had temporarily agreed on a common valid signature with the old one; only n.de.net remained incorrect. This changed for the first time at 1:15 AM on May 6th (11:15 PM UTC), when all servers again had correct responses ready – albeit not yet perfectly: a.nic.de and z.nic.de delivered two signatures in parallel, both of which were valid. At 1:17 AM, the desired state was finally achieved: six name servers could generate a valid signature. Because it was not possible to convince all six name servers to sign with 33834 simultaneously, all had reverted to key 32911 at that point, meaning the key exchange was rolled back. Nothing has changed in this regard until the afternoon of May 6th; there has been no renewed attempt to reintroduce key 33834.

What follows from this

The look at the historical DNS data shows: something was wrong with key 33834, at least with SOA records. Other DNS records could be signed successfully at any time. Countermeasures began 15 minutes after the first invalid signature. Despite some confusion, it was not possible to use the key on all six servers for SOA signatures. To get the outage under control, the decision was made to sign them again with key 32911.

For .de domains, such an outage due to a DNSSEC error is unprecedented so far. In 2022, Sweden experienced problems with the .se domain that were attributed to DNSSEC. In 2024, Russia had a DNSSEC problem with .ru domains. The cause then was a duplicate key tag ID.

DENIC is now responsible for identifying the cause of the problem and explaining what countermeasures were taken. The question of why the signature problems were not detected in a test environment also remains unanswered.

(jam)