cloudflareMay 1, 2026

Code Orange: Fail Small is complete. The result is a stronger Cloudflare network (opens in new tab)

Cloudflare’s “Code Orange: Fail Small” initiative rebuilt key parts of its infrastructure to prevent repeats of the November 18 and December 5, 2025 global outages. The work focused on safer configuration rollouts, smaller failure impact, stronger emergency access, and improved incident communication. Cloudflare concludes that the network is now more resilient, though reliability remains an ongoing effort. ## Safer Configuration Changes - Configuration changes affecting customer traffic are now deployed progressively rather than instantly across the network. - Health monitoring can detect problems early and automatically roll back faulty changes. - Cloudflare introduced **Snapstone**, a unified system that: - Packages configuration changes. - Releases them gradually. - Monitors system health in real time. - Automatically rolls back unsafe deployments. - Snapstone supports different configuration types, including data files and global control flags. - New high-risk configuration pipelines have been identified and brought under the health-mediated deployment process. ## Reducing the Impact of Failure - Product teams reviewed failure modes and removed unnecessary runtime dependencies. - Systems now use the last known good configuration where possible, following a **“fail stale”** strategy. - Where stale configuration is unavailable, teams choose between: - **Fail open:** Continue serving traffic with reduced protection or functionality. - **Fail close:** Stop processing when that is safer than continuing. - The Bot Management outage scenario would now be detected during an early rollout stage, affecting only a small amount of traffic before rollback. - Services are increasingly segmented into independent systems serving different customer cohorts. - For example, the Workers runtime deploys first to less-critical segments, such as free customers, before reaching more critical traffic. - This approach limits the blast radius of faulty deployments and adjusts rollout speed based on customer criticality. - Cloudflare plans to extend cohort-based deployment to more systems. ## Revised Break-Glass and Incident Procedures - Cloudflare audited tools needed for visibility, debugging, and emergency production changes. - It created backup authorization paths for **18 key services**, along with emergency scripts and proxies. - These pathways are designed to remain usable if Cloudflare’s own Zero Trust infrastructure is affected by an outage. - More than 200 engineers participated in an organization-wide emergency drill on April 7, 2026. - Repeated exercises are intended to ensure engineers can use emergency access procedures effectively under pressure. - Cloudflare also began improving how technical incident observations are converted into clear customer communications. Cloudflare’s changes make configuration rollouts safer, reduce failure blast radius, and improve emergency response. The practical recommendation is to treat these safeguards as ongoing operational practices rather than a one-time project, continually testing them and extending them to additional systems.

traffic-managementcloudflarezero-trustincident-management+4

airbnbOct 9, 2025

From Static Rate Limiting to Adaptive Traffic Management in Airbnb’s Key-Value Store (opens in new tab)

Airbnb evolved Mussel’s QoS system from static, per-client QPS limits into adaptive traffic management designed to maximize goodput. The newer approach accounts for the actual cost of requests, prioritizes critical workloads under stress, and detects hot keys or attack traffic before they overwhelm storage. Together, resource-aware quotas and real-time load shedding provide stronger protection against traffic spikes, uneven workloads, and DDoS-like bursts. ## Why Static QPS Limits Fell Short - Mussel is a multi-tenant key-value store serving millions of point and range reads across Airbnb. - Its original Redis-backed limiter assigned each client a fixed requests-per-second quota. - Requests exceeding the quota received HTTP 429 responses. - This model worked when backend effort roughly matched request count. - As usage grew, it could not account for: - The difference between a cheap one-row lookup and a 100,000-row scan. - Hot keys accessed by many clients simultaneously. - Localized storage-shard overload that affected unrelated traffic. - Sudden events such as bot floods, DDoS attacks, or large uploads. ## Resource-Aware Rate Control - Mussel replaced raw request counting with request units (RU), which represent estimated backend work. - RU calculations incorporate: - Fixed per-request overhead. - Rows and payload bytes processed. - Request latency, which distinguishes cached operations from disk-heavy ones. - The system uses calibrated linear formulas for reads and writes, with weights based on compute, network, and disk-I/O measurements. - Dispatchers debit a local token bucket according to each request’s RU cost rather than charging every request equally. - Periodic RU refills preserve simple, static quotas while making them more proportional to actual resource consumption. - Requests are rejected with HTTP 419 when the RU bucket is exhausted. - Load shedding remains separate, allowing latency-based protection to react dynamically without changing the underlying quota-refill mechanism. ## Load Shedding Under Sudden Stress - RU rate limiting smooths normal traffic but may react too slowly to rapidly changing workloads. - Mussel adds a load-shedding layer based on: - Traffic criticality. - A real-time latency ratio. - A CoDel-inspired queue-management policy. - Each dispatcher compares long-term p95 latency with short-term p95 latency. - A ratio near 1.0 indicates stable performance; a drop toward 0.3 signals rapidly increasing latency. - When stress crosses the threshold: - The system raises the effective RU cost for a designated lower-priority client class. - That class’s token bucket drains faster, causing its traffic to back off. - If conditions worsen, the penalty expands to additional classes. - Critical workloads, such as customer support and trust-and-safety traffic, can remain responsive while less important traffic is reduced. - The latency estimate uses the constant-memory P² algorithm, avoiding raw sample storage and cross-node coordination. ## Hot-Key Detection and DDoS Protection - Client-level quotas cannot prevent overload when many clients request the same popular key. - Mussel therefore detects skewed access patterns in real time. - When duplicate requests target a hot key, the system can protect storage by: - Serving responses from cache. - Coalescing identical requests before they reach the backend. - This approach protects the underlying shard whether the traffic comes from legitimate popularity, automation, or a DDoS burst. Mussel’s experience suggests that mature multi-tenant services should move beyond fixed QPS limits. Combining resource-based accounting, priority-aware load shedding, and hot-key mitigation provides a more effective way to preserve reliability while maximizing useful work during unpredictable traffic conditions.

traffic-managementk8sdistributed-systemsredis+4