cloudflare·May 1, 2026·3 min readCurated summary

Code Orange: Fail Small is complete. The result is a stronger Cloudflare network

Cloudflare’s “Code Orange: Fail Small” initiative rebuilt key parts of its infrastructure to prevent repeats of the November 18 and December 5, 2025 global outages. The work focused on safer configuration rollouts, smaller failure impact, stronger emergency access, and improved incident communication. Cloudflare concludes that the network is now more resilient, though reliability remains an ongoing effort. ## Safer Configuration Changes - Configuration changes affecting customer traffic are now deployed progressively rather than instantly across the network. - Health monitoring can detect problems early and automatically roll back faulty changes. - Cloudflare introduced **Snapstone**, a unified system that: - Packages configuration changes. - Releases them gradually. - Monitors system health in real time. - Automatically rolls back unsafe deployments. - Snapstone supports different configuration types, including data files and global control flags. - New high-risk configuration pipelines have been identified and brought under the health-mediated deployment process. ## Reducing the Impact of Failure - Product teams reviewed failure modes and removed unnecessary runtime dependencies. - Systems now use the last known good configuration where possible, following a **“fail stale”** strategy. - Where stale configuration is unavailable, teams choose between: - **Fail open:** Continue serving traffic with reduced protection or functionality. - **Fail close:** Stop processing when that is safer than continuing. - The Bot Management outage scenario would now be detected during an early rollout stage, affecting only a small amount of traffic before rollback. - Services are increasingly segmented into independent systems serving different customer cohorts. - For example, the Workers runtime deploys first to less-critical segments, such as free customers, before reaching more critical traffic. - This approach limits the blast radius of faulty deployments and adjusts rollout speed based on customer criticality. - Cloudflare plans to extend cohort-based deployment to more systems. ## Revised Break-Glass and Incident Procedures - Cloudflare audited tools needed for visibility, debugging, and emergency production changes. - It created backup authorization paths for **18 key services**, along with emergency scripts and proxies. - These pathways are designed to remain usable if Cloudflare’s own Zero Trust infrastructure is affected by an outage. - More than 200 engineers participated in an organization-wide emergency drill on April 7, 2026. - Repeated exercises are intended to ensure engineers can use emergency access procedures effectively under pressure. - Cloudflare also began improving how technical incident observations are converted into clear customer communications. Cloudflare’s changes make configuration rollouts safer, reduce failure blast radius, and improve emergency response. The practical recommendation is to treat these safeguards as ongoing operational practices rather than a one-time project, continually testing them and extending them to additional systems.

Read original(opens in new tab)

Observability ToolsCloudflareZero TrustIncident Management+4

discord·Mar 4, 2026·2 min readCurated summary

Tracing Discord's Elixir Systems (Without Melting Everything)

Discord runs each guild independently using Elixir’s concurrency model, helping chats and reactions feel instantaneous at scale. When a guild becomes overloaded, metrics and logs can reveal activity spikes but often fail to show the actual user experience or downstream effects. To fill this gap, Discord built distributed tracing for its Elixir services and integrated it without downtime. ## Guild-Level Isolation and Outages - Each Discord server, or “guild,” runs independently from others. - This isolation supports high concurrency and limits failures to individual guilds. - A guild may become laggy or go offline when user activity exceeds its processing capacity. - If it cannot recover automatically, on-call engineers investigate the incident. ## Limits of Metrics and Logs - Engineers inspect metrics showing: - How often each user action type is processed. - How long processing takes. - These metrics can identify bursts of activity, such as sudden waves of reactions or messages. - However, they do not clearly show how those conditions affected users. - Metrics are comparable to a car dashboard: they expose internal conditions but not necessarily the consequences. ## Guild Timings - Discord’s custom “guild timings” tool records the amount of each minute spent processing different action types. - The data is stored in memory and provides more detail than standard metrics. - Its high volume makes long-term storage impractical, so data is frequently rotated. - The tool also focuses on guild-local processing and does not capture downstream effects or complete end-to-end request experience. ## Building Distributed Tracing for Elixir - Distributed tracing shows how long each part of an operation takes across services. - Other Discord teams had already benefited from tracing and application performance monitoring. - Typical tracing systems propagate operation context through metadata such as HTTP headers. - Elixir’s built-in communication mechanisms do not provide an equivalent metadata layer. - Discord therefore built its own mechanism for propagating tracing information between services. ## Deployment Without Downtime - Although the tracing system changed how Discord services communicate, it was integrated without taking the platform offline. - The result gives engineers a more complete view of request paths, helping them understand both the source of guild problems and their impact on users. Discord’s experience suggests that detailed distributed tracing is essential when local metrics and logs cannot explain end-to-end behavior. For highly concurrent systems, investing in tracing infrastructure tailored to the platform can significantly improve incident diagnosis without requiring disruptive deployment changes.

Read original(opens in new tab)

Observability ToolsDatabase DesignApplication Performance MonitoringDistributed Tracing+3