A broken DNSSEC rollover took down .AL. Now 1.1.1.1 tells you when validation is bypassed (opens in new tab)
A failed DNSSEC key rollover at Albania’s `.AL` registry caused validating resolvers to return errors for every `.AL` domain. Cloudflare temporarily installed a Negative Trust Anchor (NTA) to restore access, suspending DNSSEC validation while the registry fixed the issue. To make this bypass visible, 1.1.1.1 began returning a new Extended DNS Error (EDE) code alongside affected responses. ## What Happened to `.AL` - Around 14:15 UTC on July 3, the registry published a new DNSKEY and stopped serving the old key. - The root zone’s DS record still referenced the old key (`id=26319`), breaking the DNSSEC chain of trust. - Around 17:00 UTC, the new DNSKEY was also removed, leaving `.AL` with no DNSKEY records. - At approximately 19:15 UTC, the DS record was removed from the root zone, restoring normal resolution but leaving the entire TLD unsigned. - The outage affected government services, banks, media, and any other `.AL` domain accessed through DNSSEC-validating resolvers. ## Why Negative Trust Anchors Were Used - An NTA, defined by RFC 7646, tells recursive resolvers to treat a zone as unsigned and skip DNSSEC validation. - Cloudflare applied an NTA to `.AL` at 17:15 UTC, restoring resolution for 1.1.1.1 users. - The measure was considered acceptable because the failure was public, confirmed, and affected validating resolvers broadly. - Communication with the registry was difficult because its contact addresses were themselves hosted under `.AL`. - The NTA was removed the following day after the DS record had been removed from the root zone. ## The Security Tradeoff - NTAs prevent widespread `SERVFAIL` responses but remove cryptographic protection against DNS spoofing. - Previously, clients could not distinguish an NTA-served response from a fully DNSSEC-validated response. - Public status pages provide disclosure, but applications, monitoring systems, and users cannot reliably discover the bypass from DNS responses alone. ## EDE-Based Transparency - Extended DNS Errors, defined in RFC 8914, let resolvers attach explanatory information to DNS responses. - Cloudflare implemented a proposed EDE code, “Negative Trust Anchor,” developed with Quad9 contributors. - During the incident, `.AL` responses included: - EDE 9: `DNSKEY Missing`, explaining the broken DNSSEC chain. - EDE 33: `Negative Trust Anchor`, explicitly stating that validation had been bypassed. - A query such as `google.al` could therefore return a successful answer while clearly indicating that it was not DNSSEC-validated. Resolvers and applications should use EDE information where possible, while DNS operators should disclose NTAs and remove them promptly once the underlying DNSSEC problem is resolved.