dnssec

3 posts

cloudflare

Natural disasters and government interference: examining Q2 2026’s major Internet disruption event (opens in new tab)

Cloudflare’s Q2 2026 outage review shows how dependent Internet connectivity remains on physical infrastructure, government policy, and complex technical systems. Disruptions ranged from typhoons, earthquakes, and power failures to shutdowns, war-related data-center damage, DNSSEC errors, and submarine cable cuts. Despite these failures, regional networks generally remained resilient, with Cloudflare Radar revealing both the scale and distinctive patterns of each outage. ### Natural Disasters and Power Failures - Super Typhoon Sinlaku passed north of Guam in April, causing power and water disruptions. - Internet traffic fell as much as 80% below expected levels on April 13–14. - Two earthquakes in northern Venezuela on June 24 produced an immediate drop in HTTP traffic. - The decline was especially visible at Fibex Telecom, CANTV, and VNET. - A nationwide power outage in Tanzania on June 27 caused Internet traffic to collapse for at least five hours. - These events demonstrate that storms, earthquakes, and electricity failures can produce similar connectivity impacts, reinforcing the need for redundancy in power, routing, and physical network paths. ### Government Shutdowns and Geopolitical Conflict - Iran began restoring Internet access on May 26 after an 88-day near-total blackout. - Traffic initially recovered to about 40% of pre-outage levels, later reaching 90% before settling near 59%. - Connectivity returned closer to the country’s recent pre-shutdown baseline, rather than fully normal levels. - AWS’s `me-central-1` region in the UAE continued to experience reduced traffic after drone strikes damaged infrastructure in the UAE and Bahrain. - The disruption affected applications hosted in the region even when those applications themselves remained operational. - Iraq imposed three exam-related shutdowns, while Sudan imposed ten. - Sudan’s outages generally lasted about 3.5 hours. - Iraq’s shutdowns lasted roughly 90 minutes. - These incidents illustrate how governments can deliberately switch off, throttle, or selectively restore national connectivity. ### DNSSEC Failure in Germany - On May 5, a DNSSEC key rollover at DENIC caused invalid signatures for Germany’s `.de` domain. - DNS resolvers validating DNSSEC rejected `.de` responses and returned `SERVFAIL`, making affected websites unreachable worldwide until service was restored at 23:15 UTC on May 5. - Cloudflare observed an increase in `.de` queries because failed responses could not be effectively cached, forcing repeated lookups and retries. - Users experienced the incident as widespread website unavailability rather than as an obvious cryptographic or DNS problem. ### Cable and Infrastructure Vulnerabilities - A submarine cable cut in Saint Lucia further demonstrated how regional connectivity can depend on a small number of physical links. - Together with the German DNSSEC failure, the incident shows that routine infrastructure maintenance and single physical-path failures can have effects far beyond the location where the fault occurs. Cloudflare’s findings underline the importance of network redundancy, careful operational procedures, and continuous monitoring. Internet outages may have very different causes, but their effects on users can look remarkably similar: sudden loss of access to communication, applications, and essential information.

cloudflare

A broken DNSSEC rollover took down .AL. Now 1.1.1.1 tells you when validation is bypassed (opens in new tab)

A failed DNSSEC key rollover at Albania’s `.AL` registry caused validating resolvers to return errors for every `.AL` domain. Cloudflare temporarily installed a Negative Trust Anchor (NTA) to restore access, suspending DNSSEC validation while the registry fixed the issue. To make this bypass visible, 1.1.1.1 began returning a new Extended DNS Error (EDE) code alongside affected responses. ## What Happened to `.AL` - Around 14:15 UTC on July 3, the registry published a new DNSKEY and stopped serving the old key. - The root zone’s DS record still referenced the old key (`id=26319`), breaking the DNSSEC chain of trust. - Around 17:00 UTC, the new DNSKEY was also removed, leaving `.AL` with no DNSKEY records. - At approximately 19:15 UTC, the DS record was removed from the root zone, restoring normal resolution but leaving the entire TLD unsigned. - The outage affected government services, banks, media, and any other `.AL` domain accessed through DNSSEC-validating resolvers. ## Why Negative Trust Anchors Were Used - An NTA, defined by RFC 7646, tells recursive resolvers to treat a zone as unsigned and skip DNSSEC validation. - Cloudflare applied an NTA to `.AL` at 17:15 UTC, restoring resolution for 1.1.1.1 users. - The measure was considered acceptable because the failure was public, confirmed, and affected validating resolvers broadly. - Communication with the registry was difficult because its contact addresses were themselves hosted under `.AL`. - The NTA was removed the following day after the DS record had been removed from the root zone. ## The Security Tradeoff - NTAs prevent widespread `SERVFAIL` responses but remove cryptographic protection against DNS spoofing. - Previously, clients could not distinguish an NTA-served response from a fully DNSSEC-validated response. - Public status pages provide disclosure, but applications, monitoring systems, and users cannot reliably discover the bypass from DNS responses alone. ## EDE-Based Transparency - Extended DNS Errors, defined in RFC 8914, let resolvers attach explanatory information to DNS responses. - Cloudflare implemented a proposed EDE code, “Negative Trust Anchor,” developed with Quad9 contributors. - During the incident, `.AL` responses included: - EDE 9: `DNSKEY Missing`, explaining the broken DNSSEC chain. - EDE 33: `Negative Trust Anchor`, explicitly stating that validation had been bypassed. - A query such as `google.al` could therefore return a successful answer while clearly indicating that it was not DNSSEC-validated. Resolvers and applications should use EDE information where possible, while DNS operators should disclose NTAs and remove them promptly once the underlying DNSSEC problem is resolved.

cloudflare

When DNSSEC goes wrong: how we responded to the .de TLD outage (opens in new tab)

On May 5, 2026, DENIC published invalid DNSSEC signatures for the `.de` zone, causing validating resolvers to return `SERVFAIL` for affected domains. Cloudflare’s 1.1.1.1 mitigated much of the impact by serving expired cached records and temporarily bypassing DNSSEC validation for `.de`. The incident demonstrates both the value of DNSSEC and the operational risks of a failure at a top-level domain. ## How DNSSEC Protects DNS - DNSSEC adds cryptographic authentication to DNS records through `RRSIG` signatures. - It protects integrity and authenticity, but does not encrypt DNS traffic. - DNSSEC creates a chain of trust from the root zone to child domains: - The root delegates trust to `.de`. - `.de` delegates trust to domains such as `example.de`. - A failure anywhere in the chain causes validation to fail below that point. - Zones generally use: - A Zone Signing Key (ZSK) for signing records. - A Key Signing Key (KSK) for signing the ZSK. - Errors during key rotation can produce signatures that resolvers cannot validate, forcing them to return `SERVFAIL`. ## Impact of the `.de` Outage - Around 19:30 UTC, DENIC began publishing invalid DNSSEC signatures for `.de`. - Any DNSSEC-validating resolver, including Cloudflare’s 1.1.1.1, had to reject the responses. - The failure spread across domains under `.de`, potentially affecting millions of German websites and services. - `SERVFAIL` responses increased gradually as cached records expired and resolvers requested fresh, invalidly signed data. - Query volume also rose because clients commonly retry failed DNS queries multiple times. ## How “Serve Stale” Reduced the Damage - Recursive resolvers normally serve cached records only until their TTL expires. - Cloudflare’s 1.1.1.1 implements RFC 8767, allowing it to serve expired records when authoritative resolution fails. - Cached `.de` records from before the incident continued resolving successfully after their TTLs expired. - This kept the overall `NOERROR` rate relatively stable, even though fresh lookups increasingly failed. - Without stale serving, successful responses would have declined steadily from the start of the outage. ## Temporarily Bypassing DNSSEC with an NTA - RFC 7646 defines Negative Trust Anchors (NTAs), which allow resolvers to treat a broken signed zone as temporarily unsigned. - NTAs are intended for situations such as a TLD operator publishing invalid signatures. - Bypassing validation can restore availability because the failure originates in the parent zone, not necessarily in individual domains. - The tradeoff is reduced security: while the exception is active, `.de` domains are exposed to DNS spoofing or other attacks that DNSSEC would normally prevent. ## Cloudflare’s Mitigation - Cloudflare’s Big Pineapple resolver did not yet have a native NTA implementation. - Instead, Cloudflare used an existing override mechanism to mark `.de` as an insecure zone. - This caused `.de` queries to be resolved without DNSSEC validation, functionally providing the same result as an NTA. - Combined with stale serving, this reduced the outage’s effect while DENIC worked to correct the zone. Cloudflare’s response illustrates a practical incident strategy: preserve cached answers where possible, then use a narrowly scoped DNSSEC exception when a parent zone is demonstrably broken. Such overrides should be temporary and carefully monitored because they restore availability at the cost of DNSSEC protection.