cloudflareJan 14, 2026

What came first- the CNAME or the A record (opens in new tab)

A memory-optimization change in Cloudflare’s 1.1.1.1 resolver accidentally reordered DNS records, placing CNAMEs after A/AAAA records. Although DNS record order is generally considered irrelevant, some clients—including glibc’s `getaddrinfo`—process answers sequentially and require CNAMEs to appear first. The resulting failures affected users globally until Cloudflare reverted the release. ## Incident Timeline - **December 2, 2025:** The record-reordering change was added. - **December 10:** It reached the testing environment. - **January 7, 2026:** Global deployment began. - **January 8, 17:40 UTC:** The change reached 90% of servers. - **18:19:** The incident was declared. - **18:27:** The release was reverted. - **19:55:** The revert completed and the impact ended. ## How CNAME Chains Are Resolved - A hostname may point through multiple aliases before reaching an A or AAAA record: - `www.example.com → cdn.example.com → server.cdn-provider.com → 198.51.100.1` - Each record has its own TTL and may expire independently. - If only part of a chain expires, 1.1.1.1 can reuse the cached portion and resolve only the missing records. - The resolver then combines the cached CNAME records with newly resolved address records. ## The Memory Optimization That Changed Ordering - Previously, the resolver created a new list: - Insert the existing CNAME chain first. - Append the newly resolved A/AAAA records afterward. - The optimization avoided allocations and copies by appending new CNAME records directly to the existing answer list. - This caused some responses to place address records before CNAME records. ## Why Some DNS Clients Failed - Many clients treat answer-section ordering as irrelevant, but some parse records sequentially. - These clients: - Start by looking for records matching the original queried name. - Update the expected name when they encounter a CNAME. - Accept the corresponding A or AAAA record only after that update. - With the expected order, the client sees the CNAME first and then accepts the address record. - With the address record first, it ignores the address because it does not yet match the expected name. After encountering the CNAME, there are no records left to process, so it reports an empty response. - The affected implementation included glibc’s `getaddrinfo`, widely used for DNS resolution on Linux. The incident demonstrates that even seemingly insignificant DNS response ordering can matter in practice. Resolver implementations should preserve CNAME-before-address ordering, and DNS clients should avoid assuming that record order is meaningful unless the protocol explicitly requires it.

glibcrustdnsnetworking+3

datadogJan 23, 2017

The trouble with mounting (opens in new tab)

Datadog found that some agents stopped reporting all metrics because they became stuck in an unkillable state during disk checks. The root cause was `os.statvfs`, whose glibc implementation can hang while inspecting NFS mounts configured with hard-mount behavior. Since agents run in unpredictable customer environments, Datadog isolated the call in a separate thread and allowed the main process to continue after a timeout. ## Detecting the Hang - Customers reported gaps across every metric, indicating that the agent—not an individual check—had stopped functioning. - Logs showed the agent sometimes hung without producing an error. - A watchdog failed to terminate it because the process was stuck in an unkillable system call. - Developer-mode timing data identified `os.statvfs` as the consistently slow operation. ## How NFS Causes Unkillable Processes - `os.statvfs` calls the Linux `statvfs` function through CPython and glibc. - `statvfs` can hang when examining a remote directory mounted through NFS. - NFS hard mounts retry indefinitely and do not time out system calls. - Soft mounts eventually return an error, while the `intr` option allows interruption of the calling process. - Hard mounts may be appropriate when reads and writes must eventually succeed, but they are risky with unreliable NFS connections because they are the default in many configurations. ## The `/proc/mounts` Complication - Glibc’s `statvfs` implementation checks each directory listed in `/proc/mounts` until it finds the requested mount. - Consequently, a disconnected NFS mount can block `statvfs` even when the agent is checking a different filesystem. - This made changing NFS mount options impractical as a universal fix because Datadog cannot control customers’ system configurations. ## Datadog’s Workaround - The agent now runs `statvfs` on a separate thread. - If the call exceeds a timeout, the main agent thread continues operating. - This approach avoids total metric loss across heterogeneous environments. - The trade-off is a modest increase in memory usage on systems with hard-mounted NFS volumes. The practical lesson is to treat filesystem statistics as potentially blocking operations, especially in environments with NFS. Isolating such calls behind timeouts provides more reliable monitoring than assuming system calls will always return promptly.

glibcdatabase-designdatadogpython+3