When DNSSEC goes wrong: how we responded to the .de TLD outage (opens in new tab)
On May 5, 2026, DENIC published invalid DNSSEC signatures for the `.de` zone, causing validating resolvers to return `SERVFAIL` for affected domains. Cloudflare’s 1.1.1.1 mitigated much of the impact by serving expired cached records and temporarily bypassing DNSSEC validation for `.de`. The incident demonstrates both the value of DNSSEC and the operational risks of a failure at a top-level domain. ## How DNSSEC Protects DNS - DNSSEC adds cryptographic authentication to DNS records through `RRSIG` signatures. - It protects integrity and authenticity, but does not encrypt DNS traffic. - DNSSEC creates a chain of trust from the root zone to child domains: - The root delegates trust to `.de`. - `.de` delegates trust to domains such as `example.de`. - A failure anywhere in the chain causes validation to fail below that point. - Zones generally use: - A Zone Signing Key (ZSK) for signing records. - A Key Signing Key (KSK) for signing the ZSK. - Errors during key rotation can produce signatures that resolvers cannot validate, forcing them to return `SERVFAIL`. ## Impact of the `.de` Outage - Around 19:30 UTC, DENIC began publishing invalid DNSSEC signatures for `.de`. - Any DNSSEC-validating resolver, including Cloudflare’s 1.1.1.1, had to reject the responses. - The failure spread across domains under `.de`, potentially affecting millions of German websites and services. - `SERVFAIL` responses increased gradually as cached records expired and resolvers requested fresh, invalidly signed data. - Query volume also rose because clients commonly retry failed DNS queries multiple times. ## How “Serve Stale” Reduced the Damage - Recursive resolvers normally serve cached records only until their TTL expires. - Cloudflare’s 1.1.1.1 implements RFC 8767, allowing it to serve expired records when authoritative resolution fails. - Cached `.de` records from before the incident continued resolving successfully after their TTLs expired. - This kept the overall `NOERROR` rate relatively stable, even though fresh lookups increasingly failed. - Without stale serving, successful responses would have declined steadily from the start of the outage. ## Temporarily Bypassing DNSSEC with an NTA - RFC 7646 defines Negative Trust Anchors (NTAs), which allow resolvers to treat a broken signed zone as temporarily unsigned. - NTAs are intended for situations such as a TLD operator publishing invalid signatures. - Bypassing validation can restore availability because the failure originates in the parent zone, not necessarily in individual domains. - The tradeoff is reduced security: while the exception is active, `.de` domains are exposed to DNS spoofing or other attacks that DNSSEC would normally prevent. ## Cloudflare’s Mitigation - Cloudflare’s Big Pineapple resolver did not yet have a native NTA implementation. - Instead, Cloudflare used an existing override mechanism to mark `.de` as an insecure zone. - This caused `.de` queries to be resolved without DNSSEC validation, functionally providing the same result as an NTA. - Combined with stale serving, this reduced the outage’s effect while DENIC worked to correct the zone. Cloudflare’s response illustrates a practical incident strategy: preserve cached answers where possible, then use a narrowly scoped DNSSEC exception when a parent zone is demonstrably broken. Such overrides should be temporary and carefully monitored because they restore availability at the cost of DNSSEC protection.