Cloudflare

196 posts

blog.cloudflare.com

Filter by tag

cloudflare3 min readCurated summary

Enforcing the First AS in BGP AS PATHs

Forged BGP AS_PATHs can let attackers impersonate legitimate networks, misdirect traffic, and conceal their true origin. Recent hijacks suggest some providers accept routes whose first AS does not match the customer peer that advertised them. Enforcing this “First AS” relationship check is a straightforward defense that complements RPKI and ASPA. ## Route Hijacks Involving Forged Paths - A reported hijack of Orange’s `90.98.0.0/15` included the path: `48237 1299 199524 270118 17072 41128`. - The path implied implausible commercial relationships: - `AS41128` was an unused Orange ASN. - `AS17072` was a Mexican ISP. - `AS270118` was a Mexican hosting provider. - `AS199524` was Gcore. - `AS1299` was Arelion, a Tier 1 provider. - Another hijack included Cloudflare’s ASN in the path: `199524 270118 17072 13335 36429`. - Cloudflare confirmed it had no adjacency with the unused `AS36429`, indicating that the path was fabricated. - The routes appeared to terminate behind Gcore rather than actually traversing the networks listed in the forged path. - The suspected attack sequence was: - Announce parked or unused prefixes. - Completely forge the AS_PATH without including the attacker’s ASN. - Send the route to Gcore. - Rely on the provider accepting the route without verifying its first AS. - Once accepted, the forged route could be propagated to upstream providers and peers. ## Why First AS Checking Matters - The BGP AS_PATH records the networks a route is expected to traverse. - It supports: - Route selection. - Loop prevention. - Operator routing policies. - BGP permits AS_PATH manipulation for legitimate purposes, such as AS prepending, but the same flexibility enables attackers to shorten or fabricate paths. - First AS enforcement verifies that the first AS in an advertised path matches the ASN of the customer or peer that sent the update. - Without this check, a customer can advertise a path that appears to originate from another network. ## Limits of RPKI and ASPA Alone - RPKI ROAs validate whether an origin ASN is authorized to announce a prefix. - ASPA validates provider-to-customer relationships. - However, an attacker may still bypass these mechanisms by: - Claiming an RPKI-valid origin ASN. - Including a legitimate ASPA provider in the forged path. - Omitting the attacker’s own ASN when the receiving provider does not enforce First AS. - In the example, a route forged to appear as though it originated from `AS64506` could remain RPKI-valid and attract traffic if `AS64502` accepted it without checking the peer’s first AS. ## Practical Recommendation Network operators should enable strict First AS checking on BGP sessions, alongside RPKI route-origin validation and ASPA deployment. These controls address different parts of the attack: RPKI validates the origin, ASPA checks authorized provider relationships, and First AS enforcement prevents a peer from presenting a path that does not begin with its own ASN.

Read original(opens in new tab)
cloudflare3 min readCurated summary

How we reduced core unit boot time from hours to minutes

Cloudflare traced a dramatic increase in core-server boot times—from minutes to nearly four hours—to a firmware update that caused UEFI to probe network boot interfaces sequentially. Each failed IPv4 HTTPS or iPXE attempt waited roughly five minutes before the firmware reached the working IPv6 interface. By explicitly configuring the correct boot interface and adapting their automation to firmware limitations, Cloudflare reduced later boot times to under a minute. ## Network Boot in Cloudflare’s Core - Core servers use automated network booting because they are bare-metal machines distributed across many environments. - The fleet relies primarily on: - PXE, often through the open-source iPXE firmware. - Native UEFI HTTPS boot on hardware that supports it. - iPXE enables scripted provisioning and can retrieve operating-system images over HTTP or HTTPS. ## The Linear Search That Caused Four-Hour Boots - After a firmware update, affected machines completed POST and hardware initialization normally but stalled before reaching the operating system. - UEFI tried network interfaces in sequence: - IPv4 HTTPS boot - IPv4 iPXE - Repeated attempts - IPv6 HTTPS boot, which eventually succeeded - Each failed attempt consumed about five minutes waiting for a timeout. - Multiple attempts added roughly 20 minutes to every boot. - Firmware upgrades required several reboots, causing the delays to compound to nearly four hours per server. ## Declaring the Correct Boot Interface - Cloudflare eliminated the linear search by configuring the preferred network boot interface order in advance. - The configuration was moved earlier into the PXE pre-boot stage and tailored to each hardware type and use case. - This removed repeated interface probing during firmware-upgrade reboots and reduced total upgrade time by about an hour. - Subsequent boots fell from approximately 20 minutes to less than one minute. ## Handling Firmware Limitations - Older UEFI versions did not support boot-order configuration. - Firmware upgrades could reset the boot-order settings. - Cloudflare added state validation to its automation: - Check the configuration after changes or upgrades. - Reapply the desired settings if they were lost. - Reboot when necessary. - The first boot could take slightly longer, but future boots became substantially faster. ## Working Around Lazy-Loaded Firmware Settings - The network boot configuration was represented by an `EFI_IFR_REF3` structure that firmware loaded only when accessed through a graphical BIOS callback. - Because the structure was not instantiated during programmatic scans, automation could not see or modify the network boot priorities. - Cloudflare worked with hardware vendors to enable tokens in the fixed “Boot Order Module.” - This forced the firmware to expose the network boot interface during startup without requiring manual BIOS interaction. ## Practical Conclusion The incident demonstrates that boot delays can come from firmware behavior rather than hardware or operating-system failures. Explicitly declaring boot interfaces, validating configuration persistence, and coordinating with vendors to expose hidden firmware settings turned repeated timeout waits into predictable, automated boot cycles.

Read original(opens in new tab)
cloudflare3 min readCurated summary

How we built Cloudflare's data platform and an AI agent on top of it

Cloudflare built Town Lake to unify data scattered across production databases, analytics systems, streams, and object storage behind one governed SQL interface. The platform combines Trino, Iceberg on R2, DataHub, and custom access-control and PII-detection services to make data fresher, more discoverable, and safer to use. Skipper extends Town Lake with a natural-language AI interface intended to provide fast, accurate, and auditable answers without requiring users to write SQL. ## The Data Sprawl Problem - Cloudflare processes over a billion events per second across a network spanning more than 330 cities and 120 countries. - Relevant data was distributed across: - Postgres - ClickHouse - BigQuery - Kafka - Google Cloud Storage and R2 - Numerous pipelines and production databases - Users needed separate credentials, query languages, retention expectations, and system knowledge for each source. - Sampled analytics data worked for dashboards but was unsuitable for billing, usage calculations, and security investigations. - External vendors created cost and dependency concerns. - Important data was difficult to discover because table locations, schemas, joins, and customer-ID mappings depended on tribal knowledge. - Data infrastructure had historically been treated as a back-office service rather than core company infrastructure. ## Goals for the New Platform Cloudflare wanted a single place where authorized employees could answer questions about customers, traffic, billing, security events, and support activity. - Support both: - Fresh, accurate, unsampled data for billing and investigations - Fast, downsampled data for dashboards and exploration - Provide built-in governance: - Automatic PII detection - Sensitive tables locked down by default - Auditable access - Time-limited permission grants - Build the system using Cloudflare’s own products, including R2, Workers, Access, and Workflows. - Eventually let employees ask questions in plain English rather than requiring SQL knowledge. - That natural-language interface became Skipper. ## Town Lake’s Lakehouse Architecture Town Lake is a lakehouse: a query engine combines data from object storage and operational systems while a metadata layer makes the data behave like a unified database. - **Trino** serves as the query engine. - A single query can join Postgres, ClickHouse, and Iceberg tables stored on R2. - Trino pushes filters into source systems and combines results without requiring intermediate materialization. - **R2 Data Catalog and Apache Iceberg** store warm and cold data. - Iceberg provides schema evolution, time travel, partition evolution, and compaction. - Data can be rolled from per-minute to hourly and eventually daily granularity as it ages. - Older data becomes cheaper to store while remaining queryable. - Parquet files on R2 cost less than retaining equivalent data in an OLAP database. - **DataHub** provides the metadata catalog. - It stores table and column descriptions, owners, lineage, and glossary terms. - Users can discover what a table contains, which teams maintain it, and how it relates to upstream and downstream data. ## Access Control and Privacy - **Lifeguard** manages access policies. - Rules are stored in D1. - User and group memberships are retrieved dynamically from Cloudflare’s internal access-management system. - Lifeguard produces JSON policies that Trino reads over HTTP. - It also supplies access information to Skipper and the Gateway, allowing users to be blocked before queries execute. - **Skimmer** continuously scans tables for PII. - It samples rows from columns across the data platform. - Workers AI classifies whether columns contain personally identifiable information. Cloudflare’s overall approach is to combine unified querying, durable low-cost storage, rich metadata, and policy enforcement so data can be broadly useful without sacrificing accuracy or governance.

Read original(opens in new tab)
cloudflare2 min readCurated summary

Iran's Internet is partially restored, Cloudflare Radar data shows

Cloudflare Radar data indicates that Iran’s Internet access began a partial restoration on May 26, after nearly three months of near-total shutdown following the February 28 military strikes. Traffic and DNS activity increased significantly, but remained far below normal levels, and the recovery could still be temporary. IPv6 connectivity remains effectively absent. ## Iran’s Two Internet Shutdowns - The first nationwide shutdown began on January 8, with traffic falling nearly to zero. - Limited connectivity briefly returned on January 21 and January 25 before recovering more substantially on January 27. - A second shutdown began on February 28 as military strikes escalated. - Traffic dropped to less than 1% of previous levels and stayed there for nearly three months. ## Signs of Partial Restoration - Around 11:00 UTC on May 26, Cloudflare observed sharp increases in traffic and DNS queries. - Transferred data briefly spiked at 11:45 UTC and then rose steadily from 12:00 UTC. - Traffic reached roughly 15 times the levels recorded during the previous week. - Activity followed expected daily patterns, declining around 21:00 UTC before rising again the following morning. - Increased DNS queries suggested that more users were successfully attempting to access websites and online services. ## Tehran and Major Providers Lead the Recovery - Tehran accounted for 91.6% of HTTP requests during the increase. - Other regions experienced only modest gains. - Traffic increased across several major providers, including: - TCI - IranCell - RighTel - MCCI ## Connectivity Remains Well Below Normal - Peak traffic on May 26 reached only about 40% of the maximum activity recorded in 2026 before the disruptions. - Future measurements will determine whether connectivity returns to pre-shutdown levels. - The January shutdown demonstrated that temporary restorations can quickly disappear. ## IPv6 Is Still Unavailable - Announced IPv6 address space from Iran remains effectively at zero. - IPv4 announcements have stayed relatively stable throughout both shutdowns. - This contrast suggests the disruptions were likely implemented through mechanisms such as application filtering or whitelisting rather than by withdrawing IPv4 routes from global networks. The data supports cautious optimism: Iranian users are regaining some Internet access, particularly in Tehran and through major providers, but service remains incomplete and potentially unstable. Continued monitoring is necessary to determine whether this is a lasting restoration.

Read original(opens in new tab)
cloudflare3 min readCurated summary

Announcing Claude Compliance API support with Cloudflare CASB

Cloudflare is adding Claude Compliance API support to CASB, giving security teams visibility into Claude usage without endpoint agents or inline traffic inspection. The integration scans Claude organizations, projects, conversations, files, and artifacts for sharing issues and sensitive data, then surfaces findings in the Cloudflare dashboard. It also connects those findings to Cloudflare Gateway policies so teams can move from detection to enforcement. ## The Security Challenge of Enterprise AI - AI adoption has outpaced governance, leaving organizations with limited visibility into sanctioned tools. - Traditional controls may block unauthorized applications but cannot inspect activity inside approved AI platforms. - AI-specific risks include: - Employees entering customer or confidential data into prompts - Developers exposing API keys - AI-generated content containing company secrets - Files and data being shared through persistent conversations and agent workflows - Effective protection must cover the full lifecycle of AI data, including API usage, content handling, and data stored within applications. ## Cloudflare’s Layered AI Security Model - **Cloudflare AI Gateway** monitors requests, token usage, and model performance while supporting rate limits, caching, and routing controls. - **Cloudflare Gateway and Data Loss Prevention** inspect AI traffic and can block prompts containing personally identifiable information or confidential material. - **Cloudflare Access with MCP server portals** protects connections between agents and corporate systems, with centralized access control and audit logging. - **Cloudflare CASB** scans data stored inside Claude for misconfigurations and sensitive content through API integrations. ## Claude Compliance API Findings Cloudflare CASB connects to Anthropic’s Compliance API and displays findings alongside those from applications such as Microsoft 365, Google Workspace, and Salesforce. - **Projects:** Detect projects shared with an organization or selected users and groups. - **Project attachments:** Identify files and documents violating DLP policies. - **Chat files:** Scan user-uploaded and provider-generated files. - **Chat messages:** Inspect prompts and provider responses for sensitive data. - **Artifacts:** Detect sensitive information in AI-generated documents and files. - Findings are categorized, prioritized by severity, and handled through existing triage, assignment, and remediation workflows. ## Coverage for Claude Enterprise and Platform - For **Claude Enterprise**, CASB retrieves information about organizations, projects, chats, roles, messages, and uploaded files using read-only endpoints. - For **Claude Platform**, it continues to monitor member and workspace changes, API key creation, and file creation or download events. - Support for the Claude Platform Activity Feed is planned for a future release. ## From Detection to Enforcement - A finding such as a sensitive file upload can be converted into a Cloudflare Gateway policy. - Administrators can: - Block uploads to Claude for specific users - Restrict access to Claude entirely - Limit application functionality until the issue is resolved - This combines CASB’s visibility into stored data with Cloudflare’s inline policy enforcement. ## Getting Started - Organizations need a Claude Enterprise account. - They must request Compliance API access from Anthropic. - Once access is granted, the integration can be connected through Cloudflare CASB. Cloudflare’s recommendation is to combine CASB monitoring with Gateway, DLP, AI Gateway, and Access controls to govern AI usage across both traffic and stored data.

Read original(opens in new tab)
cloudflare3 min readCurated summary

Announcing Claude Managed Agents on Cloudflare

Cloudflare and Anthropic have integrated Claude Managed Agents with Cloudflare Sandboxes, separating Claude’s agent reasoning from the infrastructure that executes code. The integration adds customizable security controls, sandbox observability, private-service access, browser auditing, email, and custom tools. It supports both full microVMs for complex workloads and lightweight isolates for fast, large-scale execution. ## Claude Managed Agents and the “Brain vs. Hands” Model - Claude Managed Agents run on Anthropic’s platform and can: - Read and write files - Run commands and code - Browse the web - Use prompt caching, compaction, and agent-focused optimizations - The integration decouples: - **The brain:** Claude’s agent loop on Anthropic’s infrastructure - **The hands:** Code execution, tools, sandboxes, and connected services on Cloudflare - Self-managed execution gives organizations more control over security, compliance, infrastructure, and performance. ## Cloudflare-Based Agent Environments - A Workers-based control plane creates a sandbox for each Claude Agent session. - Sandboxes support: - Code execution and file operations - Application development and CLI tools - Persistent state across session sleeps - Developers can customize: - Sandbox instance sizes - Container images for VM-based sandboxes - Cloudflare provides detailed metrics and logs, dashboard monitoring, external log shipping to services such as Datadog or Splunk, and SSH access to running sandboxes. - A built-in UI helps track sandbox state and open interactive shell sessions. ## Scaling with Isolates and MicroVMs - Full microVMs are appropriate for agents that need Linux environments, developer tooling, or complete application stacks. - Cloudflare also offers lightweight V8 isolate sandboxes using Agents SDK, Dynamic Workers, and Codemode. - Isolates provide: - Millisecond-level startup - Lower infrastructure costs - File-system support and arbitrary code execution - Much higher concurrency than VM-based systems - Developers can select an “isolate” backend when configuring an agent. - Isolates are intended for workloads reaching tens of thousands of concurrent agents, while Cloudflare Containers provide microVM-based execution when stronger environment fidelity is required. ## Security and Agent Connectivity - The default deployment routes agent traffic through customizable outbound proxies. - Proxies can help: - Inject credentials outside the sandbox - Prevent agents from accessing raw secrets - Reduce data exfiltration risk - Monitor interactions with external services - Agents can connect to private internal services without exposing those services directly to the public Internet. - The integration also includes browser session controls, recordings, audit trails, and human-in-the-loop workflows. ## Built-In Agent Capabilities The deployment template includes several capabilities without requiring additional infrastructure: - Detailed sandbox metrics, logs, and SSH access - Custom sandbox images and resource sizing - Browser automation with observability - Individual email addresses and outbound email for agents - Custom tools implemented as functions and deployed directly - Flexible execution through either isolates or microVM-backed sandboxes Cloudflare’s recommendation is to use isolates for inexpensive, highly concurrent workloads and microVMs when agents require full Linux environments or complex development workflows. The integration is designed to let teams keep Claude’s reasoning on Anthropic while retaining control over execution, connectivity, security, and observability on Cloudflare.

Read original(opens in new tab)
cloudflare3 min readCurated summary

Project Glasswing: what Mythos showed us

Project Glasswing found that Anthropic’s Mythos Preview represents a major advance in AI-assisted vulnerability research. Unlike conventional scanners, it can combine multiple low-level bugs into a credible exploit chain and generate working proofs by writing, compiling, and testing code iteratively. However, inconsistent refusals and a high rate of speculative findings mean capable models still require strong safeguards and human-led validation before large-scale deployment. ## Exploit Chain Construction - Mythos Preview can combine several seemingly minor vulnerabilities into a complete attack. - It can reason from primitives such as use-after-free bugs to arbitrary read/write access, control-flow hijacking, and ROP-based system takeover. - Earlier frontier models often identified individual bugs but failed to connect them into a working exploit. - This ability can elevate low-severity findings that might otherwise remain ignored in vulnerability backlogs. ## Automated Proof Generation - The model does more than describe suspected vulnerabilities: - Writes proof-of-concept code. - Compiles it in a scratch environment. - Executes it and checks whether the expected behavior occurs. - Revises its hypothesis when testing fails. - This feedback loop distinguishes plausible speculation from demonstrated exploitability. ## Inconsistent Model Refusals - Mythos Preview lacked the additional safeguards used in generally available models, but still developed emergent refusals around some offensive security tasks. - These refusals were inconsistent: - The same research task could succeed after an unrelated environmental change. - The model might confirm serious memory bugs but refuse to create an exploit. - Rephrasing the request or repeating it could produce a different result. - Organic model guardrails are therefore not reliable enough to act as a complete safety boundary. - Future publicly available cyber-capable models will need additional, deliberate safeguards beyond their learned behavior. ## The Signal-to-Noise Problem - Vulnerability research still requires determining which findings are real, exploitable, and urgent. - AI tools increase the volume of speculative findings, making triage more difficult. - Two major factors affect noise levels: - **Programming language:** C and C++ expose developers to memory bugs such as buffer overflows and out-of-bounds access, while memory-safe languages such as Rust eliminate many of these classes at compile time. Memory-unsafe projects produced more false positives. - **Model bias:** Models tend to report possible vulnerabilities even when evidence is weak, using qualifications such as “possibly” or “could in theory.” - Exploratory over-reporting may help discover novel issues, but it is costly in a production triage queue because each speculative finding consumes analyst time and model resources. ## Scaling AI-Assisted Security Research - Mythos Preview’s capabilities justify treating it as a different class of security tool rather than simply a better conventional scanner. - Scaling these systems will require: - Post-validation stages to filter speculative findings. - Sandboxed environments for compiling and testing proofs. - Human review of exploit chains and severity. - Explicit safety controls that do not depend solely on model refusals. - The main challenge is no longer only whether models can find vulnerabilities, but whether organizations can reliably validate, prioritize, and safely manage their output. Organizations should use advanced security models in controlled environments with layered safeguards and rigorous validation. Their ability to construct exploits is powerful, but their inconsistent safety behavior and noisy findings make unsupervised use inappropriate.

Read original(opens in new tab)
cloudflare3 min readCurated summary

Our billing pipeline was suddenly slow. The culprit was a hidden bottleneck in ClickHouse

Cloudflare’s migration to per-namespace retention in ClickHouse unexpectedly caused billing queries to slow dramatically, even though I/O, memory usage, rows scanned, and parts read appeared normal. The hidden bottleneck was query-planning lock contention: as the new `(namespace, day)` partitioning scheme multiplied the number of data parts, queries spent much of their time waiting for a mutex protecting the table’s active-part list. Investigation with flame graphs exposed the issue, leading Cloudflare to develop ClickHouse fixes. ## A Petabyte-Scale ClickHouse Platform - Cloudflare stores more than 100 PB across dozens of ClickHouse clusters. - Its Ready-Analytics system lets hundreds of internal teams share a massive table using: - A `namespace` to identify each dataset - A standard schema - A primary key of `(namespace, indexID, timestamp)` - By December 2024, the system contained more than 2 PiB and ingested millions of rows per second. ## The Limits of a Global Retention Policy - The table was partitioned by day, and a retention job dropped partitions older than 31 days. - This prevented teams from applying different retention periods: - Some needed years of data. - Others needed only a few days. - Teams requiring custom retention had to use more complicated, conventional table setups. ## Moving to Per-Namespace Partitions - Cloudflare considered: - Creating a separate table for every namespace. - Changing the partition key from `(day)` to `(namespace, day)`. - They chose the second option because it preserved the existing retention workflow while enabling namespace-level deletion. - The team expected more total parts but assumed query performance would remain stable because queries already filtered by namespace. - Migration began in January 2025 using ClickHouse’s `Merge` table feature. ## Billing Queries Begin to Slow - By late March 2025, billing aggregation jobs were approaching their daily deadlines. - Standard performance indicators looked healthy: - I/O and memory were normal. - Queries scanned no more rows or parts than before. - Query latency correlated strongly with the growing total number of parts in the cluster, revealing that merely having more parts could hurt performance. ## Finding the Hidden Lock Bottleneck - Cloudflare used ClickHouse’s `trace_log` to generate flame graphs for leaf `SELECT` queries. - CPU traces showed that roughly 45% of sampled CPU time was spent in `filterPartsByPartition`, which filters parts during query planning. - Reordering pruning heuristics produced only a 5% improvement. - “Real” traces, which include waiting and inactive threads, exposed the real issue: - More than half of query time was spent waiting on a mutex protecting the table’s active-part list. - Every query-planning thread had to contend for the `MergeTreeData` lock. - The migration increased the number of parts enough to make this previously unnoticed planning bottleneck dominant. The main lesson is that ClickHouse performance can degrade during query planning even when execution metrics look normal. When partitioning changes substantially increase part counts, teams should monitor planning time and lock contention—not just data scanned, I/O, or memory—and use real-time flame graphs to identify waits hidden by CPU-only profiling.

Read original(opens in new tab)
cloudflare3 min readCurated summary

Browser Run: now running on Cloudflare Containers, it’s faster and more scalable

Browser Run was rebuilt on Cloudflare Containers to improve speed, reliability, and scale. The migration increased capacity to 60 browser launches per minute and 120 concurrent browsers—four times the previous limit—while cutting Quick Action response times by more than 50%. The main architectural changes were global Container deployment, regional pools of pre-warmed browsers, and replacing eventually consistent KV state with transactional D1 and batched Queue updates. ## Browser Run’s Role - Provides programmatic access to headless browsers on Cloudflare’s global network. - Supports: - End-to-end testing - Suspicious URL investigation - PDF rendering - Screenshots and content extraction - Web interaction for AI agents - The goal is to offer secure, responsible browser automation at massive scale. ## Why the Previous Infrastructure Was Limiting - Browser Run originally shared infrastructure with Browser Isolation (BISO). - BISO’s larger container images caused slower startup and development cycles. - Browser Run lacked optimal global distribution, affecting latency and resilience. - BISO’s long-running sessions conflicted with Browser Run’s short, bursty workloads. - These differences created scaling and availability bottlenecks. ## Gradual Migration to Containers - A Worker initially routed a small number of requests to Container-based browsers while others continued using BISO. - This dual-running setup allowed the team to: - Compare performance - Find implementation bugs - Validate stability - Rollout stages included: - Quick Actions - Free-account Workers browser binding connections - Pay-as-you-go accounts - Contract customers - Customers did not need to change code or redeploy Workers. ## Regional Pools for Lower Latency - Durable Object-enabled Containers can create the Durable Object near the request while starting the Container elsewhere. - This is acceptable for one-off commands but inefficient for WebSocket workflows involving many messages. - The team introduced regional pools of pre-warmed, Durable Object-backed browsers. - Requests are assigned to a nearby Durable Object–Container pair, reducing latency between: - The user and Durable Object - The Durable Object and browser Container - The design requires global browser-state observability so capacity can be allocated and reassigned as demand changes. ## Replacing KV with D1 and Queues - Workers KV was initially used to track browser availability. - Its eventual consistency and cache TTL—around 30 seconds or longer—caused race conditions: - A browser could appear available when another request had already claimed it. - Delayed state updates led to over-allocation and limited responsiveness to traffic spikes. - Browser state was moved to D1, whose SQLite transactions provide atomic assignment. - A browser is exclusively assigned to one user, preventing simultaneous claims through transactional updates. Example acquisition logic updates selected candidates to `picked` and returns their data in one operation: ```sql WITH candidate_pool AS (...) UPDATE containers SET status = 'picked' WHERE sessionId IN ( SELECT sessionId FROM candidate_pool ORDER BY RANDOM() LIMIT ?5 ) RETURNING data; ``` ## Batching State Updates - D1 shards are maintained by location. - Thousands of containers report their state every five seconds, which could overload the database if each update were written individually. - Queue-based batching groups 100 updates into a single write. - This increases theoretical capacity from roughly 5,000 containers per location to as many as 500,000. - The team reports a P95 batch-write latency of 0.1 ms. - Queue consumers use: - Maximum batch size: 100 - Maximum batch timeout: 1 second - Maximum retries: 1 The migration is live, requires no customer changes, and gives Browser Run more room to handle demand from AI agents and other high-volume browser automation workloads.

Read original(opens in new tab)
cloudflare3 min readCurated summary

When "idle" isn't idle: how a Linux kernel optimization became a QUIC bug

CUBIC, the default congestion controller in Linux and quiche, can become permanently stuck at its minimum congestion window after an early congestion collapse. Cloudflare found the bug in a QUIC test where packet loss stopped completely, yet CUBIC continued oscillating between recovery and congestion avoidance instead of increasing its sending rate. The problem was traced to a Linux TCP optimization for idle or app-limited connections, and ultimately fixed with an elegant near-one-line change. ## How CUBIC manages traffic - CUBIC controls the sender’s congestion window (`cwnd`), limiting how many bytes can be in flight. - It increases `cwnd` when acknowledgments arrive without loss and reduces it when loss suggests the network is overloaded. - As quiche’s default congestion controller, CUBIC affects a substantial amount of QUIC traffic. - Recovery from the minimum congestion window is an important but relatively under-tested part of congestion control. ## The failing test - The test downloaded a 10 MB file over HTTP/3 between local quiche client and server. - Network conditions included: - 10 ms RTT - 30% random packet loss during the first two seconds - No packet loss after two seconds - A 10-second timeout - The expected result was for CUBIC to reduce its window during loss, then steadily recover once the network became reliable. - Instead, approximately 60% of repeated 100-run test batches failed to finish in time. ## CUBIC becomes stuck at its minimum - After packet loss stopped at two seconds, bytes in flight remained flat rather than increasing. - CUBIC’s congestion window stayed at its minimum of 2,700 bytes—roughly two full-sized packets. - The controller repeatedly switched between recovery and congestion avoidance: - 999 transitions over about 6.7 seconds - One transition approximately every 14 ms - The oscillation closely matched the connection’s RTT, indicating that each ACK round was triggering the behavior. - Because the test was a download, client ACKs caused the server’s bytes in flight to fall to zero; the server then sent another two-packet burst, repeatedly provoking the faulty state transition. - Reno passed the same test 100% of the time, confirming that the issue was specific to CUBIC rather than the test setup. ## The connection to Linux TCP - The investigation focused on behavior when `bytes_in_flight == 0`, effectively an idle or app-limited condition. - A 2017 Linux kernel change addressed a TCP CUBIC issue after application idle periods. - Before the change, CUBIC’s epoch could remain unchanged for a long time while the application was idle. - When sending resumed, the elapsed time used by CUBIC could be extremely large, producing an excessively aggressive growth slope and dangerous congestion-window inflation. - The kernel optimization was intended to align CUBIC with the app-limited exclusion described in RFC 9438 §4.2-12. - Porting this logic to QUIC exposed an unintended interaction: repeated short periods with no bytes in flight could be interpreted incorrectly, causing CUBIC to cycle between states and remain at its minimum window. The practical lesson is that congestion-control implementations must test not only steady-state throughput and ordinary loss recovery, but also recovery from the minimum window and repeated app-limited or idle periods. In this case, a small adjustment to the idle-state handling broke the cycle and allowed CUBIC to recover normally.

Read original(opens in new tab)
cloudflare3 min readCurated summary

Building For The Future

Cloudflare announced plans to eliminate more than 1,100 jobs as it restructures for an “agentic AI era.” The company says the decision reflects a fundamental redesign of internal processes, teams, and roles—not individual performance or a conventional cost-cutting exercise. Leaders argue that acting decisively now will reduce prolonged uncertainty and create a faster, more innovative organization. ## Restructuring Around Agentic AI - Cloudflare’s internal AI usage has increased by more than 600% in three months. - Employees across engineering, HR, finance, marketing, and other departments run thousands of AI-agent sessions daily. - Because Cloudflare uses AI extensively itself, leadership says the company must redesign how work is organized to capture its benefits. - The restructuring covers internal processes, teams, and roles across the company. ## A Company-Wide Workforce Reduction - More than 1,100 employees globally will leave Cloudflare. - The company emphasizes that departures are not judgments about employees’ talent or performance. - Founders Matthew Prince and Michelle Zatlyn are communicating the decision directly rather than routing notices through managers. - Every employee is receiving an email explaining how the changes affect them. ## Severance and Treatment of Departing Employees - Departing employees will receive the equivalent of their full base salary through the end of 2026. - U.S. healthcare support will continue through the end of 2026. - Equity will continue vesting through August 15. - Employees who had not reached their one-year vesting cliffs will receive prorated vesting through August. - Cloudflare frames these benefits as an effort to treat departing employees with empathy and exceed typical industry standards. ## Why Cloudflare Chose Decisive Action - Leadership argues that smaller, repeated layoffs or a prolonged reorganization would create continuing emotional uncertainty. - Completing the changes at once is intended to provide clarity to departing employees and stability for those who remain. - Cloudflare believes its original cloud-native structure helped it surpass older companies with slower systems and processes. - As the company has grown, it says it must avoid relying on organizational structures that worked in the past. ## Looking Ahead - Cloudflare expects the reshaped organization to operate more quickly and innovate more effectively. - The founders planned to discuss the announcement during the company’s earnings call and an all-hands meeting. - They presented the restructuring as necessary to continue advancing Cloudflare’s mission of building a better Internet. The practical conclusion is that Cloudflare is making a large, one-time organizational reset to align its workforce with AI-driven operations, while offering unusually extensive severance intended to reduce the disruption for affected employees.

Read original(opens in new tab)
cloudflare3 min readCurated summary

How Cloudflare responded to the “Copy Fail” Linux vulnerability

Cloudflare assessed the “Copy Fail” Linux privilege-escalation vulnerability (CVE-2026-31431) immediately after its disclosure on April 29, 2026. Its existing kernel update process meant the relevant fixes were already deployed across most infrastructure, while behavioral detections could identify exploitation attempts within minutes. Cloudflare reported no environmental impact, customer data exposure, or service disruption. ## Cloudflare’s Linux Kernel Update Process - Cloudflare runs custom Linux kernels based on community Long-Term Support releases across datacenters in 330 cities. - Automated jobs build updated kernels approximately weekly from upstream security and stability fixes. - New builds are tested in staging before global deployment. - The Edge Reboot Release pipeline rolls updates through edge infrastructure on a four-week cycle. - Control-plane systems generally use the newest kernel, with reboots scheduled based on workload requirements. - By the time vulnerabilities are publicly disclosed, fixes are typically already present in stable LTS releases and deployed by Cloudflare. - At disclosure, most systems used Linux 6.12 LTS, while some were transitioning to 6.18 LTS. ## How Copy Fail Worked - The vulnerability affected the Linux kernel’s `AF_ALG` interface, which lets unprivileged processes access cryptographic operations through the `algif_aead` module. - Attackers could combine: - `sendmsg()` or `splice()` to submit data - `recvmsg()` to execute the cryptographic operation - Page-cache references to redirect writes into files - An older in-place optimization allowed the AEAD implementation to write beyond the intended output boundary. - The `authencesn` wrapper performed a controllable four-byte out-of-bounds write. - By using `splice()`, an attacker could target pages belonging to any readable file and control: - The file being modified - The write offset - The four bytes written ## Privilege Escalation Through `/usr/bin/su` - The public exploit targeted `/usr/bin/su`, a setuid-root binary commonly present on Linux systems. - The attacker populated the binary’s contents in the page cache and connected those cached pages to a crypto scatterlist. - Shellcode was supplied through AAD bytes in `sendmsg()`. - `splice()` parameters controlled the target offset in the binary. - Although `recvmsg()` returned `-EBADMSG`, the out-of-bounds write had already modified the shared page cache. - Executing `/usr/bin/su` then loaded the modified cached pages, causing the injected code to run with root privileges. ## Upstream Fix and Cloudflare’s Response - The upstream fix, commit `a664bf3d603d`, reverted the 2017 in-place optimization responsible for the flaw. - After disclosure, Cloudflare’s security and kernel engineering teams worked in parallel to: - Identify vulnerable kernel versions - Assess infrastructure exposure - Review the exploit technique - Validate behavioral detections - Existing monitoring could detect the exploit pattern within minutes. - Cloudflare concluded that its infrastructure was not impacted and that no customer data or services were affected. Cloudflare’s experience demonstrates the value of maintaining patched LTS kernels, automating frequent kernel builds and staged rollouts, and combining preventative patching with behavioral exploit detection.

Read original(opens in new tab)
cloudflare3 min readCurated summary

When DNSSEC goes wrong: how we responded to the .de TLD outage

On May 5, 2026, DENIC published invalid DNSSEC signatures for the `.de` zone, causing validating resolvers to return `SERVFAIL` for affected domains. Cloudflare’s 1.1.1.1 mitigated much of the impact by serving expired cached records and temporarily bypassing DNSSEC validation for `.de`. The incident demonstrates both the value of DNSSEC and the operational risks of a failure at a top-level domain. ## How DNSSEC Protects DNS - DNSSEC adds cryptographic authentication to DNS records through `RRSIG` signatures. - It protects integrity and authenticity, but does not encrypt DNS traffic. - DNSSEC creates a chain of trust from the root zone to child domains: - The root delegates trust to `.de`. - `.de` delegates trust to domains such as `example.de`. - A failure anywhere in the chain causes validation to fail below that point. - Zones generally use: - A Zone Signing Key (ZSK) for signing records. - A Key Signing Key (KSK) for signing the ZSK. - Errors during key rotation can produce signatures that resolvers cannot validate, forcing them to return `SERVFAIL`. ## Impact of the `.de` Outage - Around 19:30 UTC, DENIC began publishing invalid DNSSEC signatures for `.de`. - Any DNSSEC-validating resolver, including Cloudflare’s 1.1.1.1, had to reject the responses. - The failure spread across domains under `.de`, potentially affecting millions of German websites and services. - `SERVFAIL` responses increased gradually as cached records expired and resolvers requested fresh, invalidly signed data. - Query volume also rose because clients commonly retry failed DNS queries multiple times. ## How “Serve Stale” Reduced the Damage - Recursive resolvers normally serve cached records only until their TTL expires. - Cloudflare’s 1.1.1.1 implements RFC 8767, allowing it to serve expired records when authoritative resolution fails. - Cached `.de` records from before the incident continued resolving successfully after their TTLs expired. - This kept the overall `NOERROR` rate relatively stable, even though fresh lookups increasingly failed. - Without stale serving, successful responses would have declined steadily from the start of the outage. ## Temporarily Bypassing DNSSEC with an NTA - RFC 7646 defines Negative Trust Anchors (NTAs), which allow resolvers to treat a broken signed zone as temporarily unsigned. - NTAs are intended for situations such as a TLD operator publishing invalid signatures. - Bypassing validation can restore availability because the failure originates in the parent zone, not necessarily in individual domains. - The tradeoff is reduced security: while the exception is active, `.de` domains are exposed to DNS spoofing or other attacks that DNSSEC would normally prevent. ## Cloudflare’s Mitigation - Cloudflare’s Big Pineapple resolver did not yet have a native NTA implementation. - Instead, Cloudflare used an existing override mechanism to mark `.de` as an insecure zone. - This caused `.de` queries to be resolved without DNSSEC validation, functionally providing the same result as an NTA. - Combined with stale serving, this reduced the outage’s effect while DENIC worked to correct the zone. Cloudflare’s response illustrates a practical incident strategy: preserve cached answers where possible, then use a narrowly scoped DNSSEC exception when a parent zone is demonstrably broken. Such overrides should be temporary and carefully monitored because they restore availability at the cost of DNSSEC protection.

Read original(opens in new tab)
cloudflare3 min readCurated summary

Code Orange: Fail Small is complete. The result is a stronger Cloudflare network

Cloudflare’s “Code Orange: Fail Small” initiative rebuilt key parts of its infrastructure to prevent repeats of the November 18 and December 5, 2025 global outages. The work focused on safer configuration rollouts, smaller failure impact, stronger emergency access, and improved incident communication. Cloudflare concludes that the network is now more resilient, though reliability remains an ongoing effort. ## Safer Configuration Changes - Configuration changes affecting customer traffic are now deployed progressively rather than instantly across the network. - Health monitoring can detect problems early and automatically roll back faulty changes. - Cloudflare introduced **Snapstone**, a unified system that: - Packages configuration changes. - Releases them gradually. - Monitors system health in real time. - Automatically rolls back unsafe deployments. - Snapstone supports different configuration types, including data files and global control flags. - New high-risk configuration pipelines have been identified and brought under the health-mediated deployment process. ## Reducing the Impact of Failure - Product teams reviewed failure modes and removed unnecessary runtime dependencies. - Systems now use the last known good configuration where possible, following a **“fail stale”** strategy. - Where stale configuration is unavailable, teams choose between: - **Fail open:** Continue serving traffic with reduced protection or functionality. - **Fail close:** Stop processing when that is safer than continuing. - The Bot Management outage scenario would now be detected during an early rollout stage, affecting only a small amount of traffic before rollback. - Services are increasingly segmented into independent systems serving different customer cohorts. - For example, the Workers runtime deploys first to less-critical segments, such as free customers, before reaching more critical traffic. - This approach limits the blast radius of faulty deployments and adjusts rollout speed based on customer criticality. - Cloudflare plans to extend cohort-based deployment to more systems. ## Revised Break-Glass and Incident Procedures - Cloudflare audited tools needed for visibility, debugging, and emergency production changes. - It created backup authorization paths for **18 key services**, along with emergency scripts and proxies. - These pathways are designed to remain usable if Cloudflare’s own Zero Trust infrastructure is affected by an outage. - More than 200 engineers participated in an organization-wide emergency drill on April 7, 2026. - Repeated exercises are intended to ensure engineers can use emergency access procedures effectively under pressure. - Cloudflare also began improving how technical incident observations are converted into clear customer communications. Cloudflare’s changes make configuration rollouts safer, reduce failure blast radius, and improve emergency response. The practical recommendation is to treat these safeguards as ongoing operational practices rather than a one-time project, continually testing them and extending them to additional systems.

Read original(opens in new tab)
cloudflare3 min readCurated summary

Introducing Dynamic Workflows: durable execution that follows the tenant

Dynamic Workflows extends Cloudflare’s durable execution system to multi-tenant and dynamically generated applications. While Dynamic Workers provide isolated runtime compute, Durable Object Facets provide tenant-specific storage, and Artifacts provide versioned source control, Dynamic Workflows lets each tenant supply its own long-running workflow code. The result is durable execution that can resume the correct tenant’s workflow after failures, hibernation, or delays of days. ## The Gap Between Durable and Dynamic Execution - Cloudflare Workflows turns a `run(event, step)` function into a durable program. - Workflow steps can: - Survive isolate recycling and failures - Sleep for hours or days - Wait for external events - Resume from the exact point where execution stopped - Workflows V2 supports up to 50,000 concurrent instances and 300 new instances per second per account. - Traditional Workflows assume the workflow class is included in the deployment and statically configured in `wrangler.jsonc`. - That model breaks for: - Multi-tenant SaaS platforms - AI-generated tenant applications - Repository-specific CI/CD pipelines - Agents that create their own durable plans - In these systems, workflow code varies by tenant, agent, repository, or request, so a single statically bound class is insufficient. ## Dynamic Workflows - `@cloudflare/dynamic-workflows` is a roughly 300-line TypeScript library. - It introduces a Worker Loader that: - Loads each tenant’s code dynamically - Routes workflow creation to the appropriate tenant - Ensures later workflow execution returns to that tenant’s code - The Loader creates a dynamic Worker with: - A tenant-specific module - A `TenantWorkflow` entrypoint - A wrapped `WORKFLOWS` binding - The dynamic entrypoint is registered as the workflow class in `wrangler.jsonc`. - Tenant code remains ordinary Cloudflare Workflows code and does not need to know it is being dynamically dispatched. ## Tenant Workflow Behavior - Tenants can use the normal Workflow APIs, including: - `env.WORKFLOWS.create(...)` - Workflow IDs and `.status()` - `.pause()` - Retries and durable steps - `step.sleep('24 hours')` - `step.waitForEvent()` - A tenant can define a standard `WorkflowEntrypoint` with a `run(event, step)` method. - The library’s primary responsibility is preserving the association between a workflow instance and the tenant implementation when the workflow resumes later. ## Three-Layer Execution Model - Dynamic Workflows consists of three layers: - The Cloudflare Workflows engine - The platform’s Worker Loader - The tenant’s dynamically loaded Worker code - A request first enters the Loader, which identifies the tenant and routes execution to its dynamic code. - The workflow engine then persists the workflow state and later invokes `run(event, step)`. - The Loader resolves the correct tenant implementation when execution resumes, even after delays or failures. Dynamic Workflows provides the missing durable-execution counterpart to Cloudflare’s dynamic compute, storage, and source-control primitives. It is particularly suited to platforms where customers or agents generate workflow code at runtime while still requiring standard durable guarantees.

Read original(opens in new tab)