Cloudflare/rust

11 posts

cloudflare

Secure all your internal vibe-coded applications — in one click (opens in new tab)

AI-driven development makes it easy for employees to deploy applications, but also increases the risk of unintentionally exposing company data. Cloudflare’s new Access integration for Workers makes applications private by default at the Worker or account level, regardless of how they are reached. It also exposes authenticated user identity directly in Worker code and supports private-by-default internal deployment platforms. ## Worker-Level Access Protection - Access authentication is enforced before requests reach application code. - Protection applies across custom domains, routes, `workers.dev` subdomains, and preview URLs. - Policies can cover: - Preview deployments only - Every hostname associated with a Worker - Attaching policies to the Worker eliminates the need to update Access settings whenever a new domain is added. - Existing identity providers, email addresses, domains, groups, and service tokens can control access. ## Account-Wide Private Defaults - An account-level policy automatically protects all current and future Workers. - Organizations can protect preview traffic, production traffic, or both. - Public Workers can explicitly bypass the account-wide policy. - For individual applications, Worker policies provide targeted protection. - When multiple policies apply, precedence is: - Hostname policies - Worker policies - Account policies ## Accessing User Identity in Worker Code - Authenticated requests expose identity through `ctx.access`. - `ctx.access.getIdentity()` returns information such as: - Email address - Name - Groups - Developers no longer need to parse, validate, and extract claims from Access JWTs manually. - Applications can use this identity for personalization, authorization, and per-user logging. - Code should handle requests without Access metadata, for example by returning a `403` response. ## Local Development and Testing - `wrangler dev` can simulate authenticated users locally. - An `access.dev` block in `wrangler.jsonc` defines a test audience and identity: ```json { "access": { "dev": { "aud": "my-app", "identity": { "email": "admin@company.com" } } } } ``` - Developers can change the configured email to test different user experiences without repeatedly deploying and authenticating through Access. ## Private Internal Deployment Platforms - Workers for Platforms can host many applications inside a namespace. - Traffic is routed through a shared dispatch Worker. - Protecting the dispatch Worker with Access makes every application deployed through it private by default. - Cloudflare provides an open-source example of an internal drag-and-drop deployment platform using this model. ## Infrastructure Behind the Feature - The capability relies on FL2, Cloudflare’s Rust-based modular proxy. - Workers routing had to be separated from execution so Cloudflare could determine the destination Worker before applying Access. - This routing change would have been more difficult in the older NGINX- and Lua-based FL1 architecture. Cloudflare’s approach shifts application security from an optional developer-configured step to an organizational default. Teams deploying internal or experimental Workers should use account-level or dispatch-level Access policies, while using Worker-level policies and local identity simulation for application-specific control and testing.

cloudflare

Introducing Kitesurf: The agent-first browser that runs in V8 isolates on Cloudflare Workers (opens in new tab)

Cloudflare argues that AI agents need a browser optimized for machine tasks rather than human browsing. Chromium provides far more functionality than agents require while consuming too much memory and compute, limiting accessibility and scalability. The company therefore built Kitesurf, a lightweight browser running entirely on Workers and designed for agentic workloads. ## Why Cloudflare Built a New Browser - Cloudflare had repeatedly considered building a browser but previously found the technical investment difficult to justify. - Recent advances in its Developer Platform changed the equation: - Mature WebAssembly support in Workers - Dynamic workers - SQLite-based Durable Objects - Worker-to-worker RPC and service bindings - Improved Node.js compatibility and higher platform limits - Growing demand for AI browser automation exposed Chromium’s limitations: - High CPU and memory consumption - Expensive dedicated browser instances - Poor scalability for large numbers of agents ## Designing for Agents Instead of Humans - Agents prioritize: - Low token counts - Large context windows - Scalability and performance - Low operating costs - Structured, machine-readable content - They do not need many human-oriented features, such as: - Tabs, themes, extensions, and device synchronization - Pixel-perfect rendering - Smooth 60-frame-per-second scrolling - AI browser security requires a different threat model, with prompt injection and tool safety treated as central concerns. - Kitesurf became the result: a browser available in beta through Cloudflare’s Browser Run product. ## From Prototype to Product - The project began with inspiration from Obscura, a lightweight Rust headless engine for AI automation. - Cloudflare used an AI agent to attempt a port to Workers. - The first prototype was weak, but a detailed plan and explicit success criteria allowed the agent to iterate effectively. - The promising proof of concept led the team to develop Kitesurf further. ## Testing as a Foundation - Cloudflare relied heavily on automated testing to accelerate development without sacrificing quality. - Web Platform Tests (WPT) provided standards-based criteria for implementing browser features. - Engineers curated feature assignments and sequencing so AI agents could work toward measurable goals. - Because WPT does not fully capture real-world website behavior, Cloudflare added: - Multistep Puppeteer integration tests - Comparisons against Chromium - Visual regression checks at every interaction step - This combination tested both standards conformance and practical rendering behavior. ## Rust and WebAssembly - Kitesurf uses Rust wherever possible and compiles directly to WebAssembly with `wasm-bindgen`. - This avoids the bulk and performance costs associated with Emscripten’s emulation layers and mocked dependencies. - The approach allows browser components to run closer to native performance inside Workers. ## Resilience Through Exception Handling - Since browsers must process unreliable and potentially hostile web content, failures must not terminate entire sessions. - Kitesurf follows a strict rule: - Errors degrade to a blank frame or missing element - Faults are caught at component boundaries - Safe empty defaults are used - Diagnostic information is logged - This makes individual rendering failures survivable rather than allowing malformed input to crash the browser. ## Isolation and Statelessness - Every page load is treated as untrusted input. - Sessions begin fresh, and components receive only the resources they require. - Workers provide isolation boundaries, but Kitesurf also enforces isolation within the application itself to prevent data leakage between pages. - Components are kept stateless wherever possible: - Failed components can simply be recreated - Work can be scaled horizontally and run in parallel - Burst-based workloads avoid the cost of maintaining idle instances - Recovery can consist of restarting a component and replaying a request Kitesurf’s central recommendation is to build browsers around the needs of their users—in this case, AI agents. By sacrificing human-focused features and emphasizing efficiency, structured output, isolation, resilience, and scale, Cloudflare aims to make browser automation practical for a much broader range of agentic applications.

cloudflare

How Cloudflare enforces engineering standards using AI (opens in new tab)

Cloudflare built the Codex to turn scattered engineering knowledge into governed, machine-readable standards that both engineers and AI agents can apply consistently. It now supports code reviews, technical design reviews, and incident reviews, with AI systems flagging nearly 230,000 violations and blocking about 16,000 merges. The central approach is to combine human-owned RFCs with structured extraction, staged enforcement, and context-aware agents. ## Why Cloudflare Built the Codex - Engineering guidance previously existed across formal documentation, repositories, chat, and individual experience. - Engineers struggled to determine whether guidance was current, authoritative, or relevant. - Growth made it difficult for anyone to know every standard or for reviewers to check every requirement. - The Codex provides a shared source of truth that can be retrieved and applied at the point of work. ## Governance and RFC Workflow - The Codex is divided into domains such as: - Architecture and control plane systems - Security and reliability - Programming languages including TypeScript and Rust - Each domain has an owner responsible for content quality and consistency. - Standards follow an RFC format using RFC 2119 terminology: - **SHOULD** for recommendations - **MUST** for mandatory requirements - Employees can propose RFCs through structured merge requests. - Proposals undergo increasingly broad review before domain-owner approval. - Approved RFCs are published to an internal Astro-powered site. - Enforcement is deliberately separated from approval: - Approved standards can generate findings. - Only enforced standards can block merges. - This gives teams time to adopt requirements and implement enforcement mechanisms. ## Structured Standards for Agents - Feeding all 60-plus RFCs directly into an LLM would consume too much context and reduce accuracy. - A dedicated agent extracts SHOULD and MUST statements into structured JSON. - Each statement includes: - A stable slug - RFC and domain metadata - Requirement level - Section and source link - Stable identifiers allow Cloudflare to track requirements across RFC revisions, systems, monitoring, and exception handling. - Cloudflare moved from concise Markdown extraction to JSON to enable filtering and progressive disclosure. - Future metadata may identify which SDLC stage applies, such as design, implementation, or runtime. ## AI Code Review - The AI code reviewer retrieves relevant statements first and loads complete RFCs only when more context is needed. - Approved-RFC findings are non-blocking recommendations. - Violations of MUST requirements in enforced RFCs can withhold approval or block a merge. - Since launch, the reviewer has: - Flagged nearly 230,000 violations - Withheld approval for almost 16,000 violations ## Faster Code Review Alternatives - Full AI reviews generally take several minutes because they use coordinators and multiple agents. - To reduce remediation delays, Cloudflare is also developing mechanically verifiable checks. - Language-specific Codex requirements can be distributed through custom linter configuration packages. - TypeScript was the first language to receive Codex linter support, alongside standardization on oxlint. The Codex’s practical value comes from connecting governed human standards to automated enforcement. Cloudflare’s staged RFC lifecycle, stable statement identifiers, and combination of AI review with fast linters provide a scalable way to preserve engineering knowledge while reducing review inconsistency.

cloudflare

How we found a bug in the hyper HTTP library (opens in new tab)

The Images binding’s migration to a local Unix-socket architecture exposed a rare race condition in Rust’s `hyper` HTTP library. Under slow-reader conditions, large image responses were truncated even though they returned `200 OK` and a full `Content-Length`, causing downstream processing or image decoding to fail. After six weeks of investigation, the issue was traced to premature socket shutdown and fixed with four lines of code. ## Images Bindings and the Request Path - Cloudflare’s Images service runs on Workers and uses `hyper` to manage HTTP connections. - The Images binding lets Workers send image data directly to the service, chain transformations, and receive the processed result as a stream. - The response path involved: - The Images service generating the complete encoded image. - `hyper` buffering the response. - Data moving through socket buffers managed by the kernel. - A client or intermediary reading the response. - If the reader was fast, `hyper` could flush the entire response and safely shut down the socket. - If the reader was slower, the socket’s outbound buffer filled, requiring `hyper` to pause and resume writing. ## Moving from FL to Local Unix Sockets - Initially, binding traffic passed through Cloudflare’s FL intermediary service. - In December 2025, the Images team replaced FL with an internal binding running on the same machine. - Unix sockets removed network and FL-processing overhead, including routing and DNS work. - The redesign improved performance and allowed the Images team to release binding changes independently. - The bug appeared within days of the rollout. ## Successful Responses with Truncated Bodies - The first report involved nested image-processing pipelines: - An inner Images binding composited large JPEG and PNG inputs from R2. - An outer URL-based pipeline resized, compressed, and transcoded the result. - The inner pipeline returned `200 OK` and a `Content-Length` for several megabytes, but delivered only a fraction of the body. - One response contained roughly 200 KB instead of the expected 3.3 MB. - The outer pipeline reported an end-of-file error because the body ended before the declared message length. - Depending on the image format, clients saw partially rendered images or completely broken images. ## Reproducing and Isolating the Race - Engineers recreated the nested setup, then removed layers until the failure occurred with the binding alone. - Batch testing produced failures reliably—for example, 19 of 25 requests in one run. - The amount of data received, approximately 200 KB, closely matched the production socket-buffer size. - This indicated that the failure was related to backpressure and socket-buffer exhaustion rather than the customer’s specific configuration. - Investigation eventually identified a race in `hyper` where the connection could be shut down before buffered response data had finished flushing. The incident demonstrates that HTTP success status codes do not guarantee complete response bodies when connection handling is incorrect. Systems streaming large payloads over sockets should test slow-reader and backpressure scenarios, and libraries should only close connections after all buffered data has been written.

cloudflare

Project Glasswing: what Mythos showed us (opens in new tab)

Project Glasswing found that Anthropic’s Mythos Preview represents a major advance in AI-assisted vulnerability research. Unlike conventional scanners, it can combine multiple low-level bugs into a credible exploit chain and generate working proofs by writing, compiling, and testing code iteratively. However, inconsistent refusals and a high rate of speculative findings mean capable models still require strong safeguards and human-led validation before large-scale deployment. ## Exploit Chain Construction - Mythos Preview can combine several seemingly minor vulnerabilities into a complete attack. - It can reason from primitives such as use-after-free bugs to arbitrary read/write access, control-flow hijacking, and ROP-based system takeover. - Earlier frontier models often identified individual bugs but failed to connect them into a working exploit. - This ability can elevate low-severity findings that might otherwise remain ignored in vulnerability backlogs. ## Automated Proof Generation - The model does more than describe suspected vulnerabilities: - Writes proof-of-concept code. - Compiles it in a scratch environment. - Executes it and checks whether the expected behavior occurs. - Revises its hypothesis when testing fails. - This feedback loop distinguishes plausible speculation from demonstrated exploitability. ## Inconsistent Model Refusals - Mythos Preview lacked the additional safeguards used in generally available models, but still developed emergent refusals around some offensive security tasks. - These refusals were inconsistent: - The same research task could succeed after an unrelated environmental change. - The model might confirm serious memory bugs but refuse to create an exploit. - Rephrasing the request or repeating it could produce a different result. - Organic model guardrails are therefore not reliable enough to act as a complete safety boundary. - Future publicly available cyber-capable models will need additional, deliberate safeguards beyond their learned behavior. ## The Signal-to-Noise Problem - Vulnerability research still requires determining which findings are real, exploitable, and urgent. - AI tools increase the volume of speculative findings, making triage more difficult. - Two major factors affect noise levels: - **Programming language:** C and C++ expose developers to memory bugs such as buffer overflows and out-of-bounds access, while memory-safe languages such as Rust eliminate many of these classes at compile time. Memory-unsafe projects produced more false positives. - **Model bias:** Models tend to report possible vulnerabilities even when evidence is weak, using qualifications such as “possibly” or “could in theory.” - Exploratory over-reporting may help discover novel issues, but it is costly in a production triage queue because each speculative finding consumes analyst time and model resources. ## Scaling AI-Assisted Security Research - Mythos Preview’s capabilities justify treating it as a different class of security tool rather than simply a better conventional scanner. - Scaling these systems will require: - Post-validation stages to filter speculative findings. - Sandboxed environments for compiling and testing proofs. - Human review of exploit chains and severity. - Explicit safety controls that do not depend solely on model refusals. - The main challenge is no longer only whether models can find vulnerabilities, but whether organizations can reliably validate, prioritize, and safely manage their output. Organizations should use advanced security models in controlled environments with layered safeguards and rigorous validation. Their ability to construct exploits is powerful, but their inconsistent safety behavior and noisy findings make unsupervised use inappropriate.

cloudflare

Making Rust Workers reliable: panic and abort recovery in wasm‑bindgen (opens in new tab)

Rust Workers historically treated Rust panics and aborts as fatal WebAssembly failures, potentially poisoning a Worker instance and causing unrelated requests to fail. Cloudflare’s latest work upstreamed into `wasm-bindgen` adds comprehensive recovery: `panic=unwind` preserves application state after recoverable panics, while abort handling ensures Rust code cannot run again after an unrecoverable abort. ## Initial Recovery Mitigations - Early Rust Workers used a custom panic handler to track failures and reinitialize the entire application before serving later requests. - JavaScript bindings were wrapped with Proxy-based indirection so every Rust entry point passed through recovery logic. - Generated bindings were modified to reinitialize the WebAssembly module after failures. - This approach shipped by default in `workers-rs` 0.6 and prevented persistent failure modes, but reinitialization could discard in-memory state. ## Panic Unwinding with WebAssembly Exception Handling - WebAssembly’s `wasm32-unknown-unknown` target traditionally defaults to `panic=abort`, turning panics into traps and `WebAssembly.RuntimeError` exceptions. - With WebAssembly Exception Handling support, Rust can be compiled using: ```bash RUSTFLAGS='-Cpanic=unwind' cargo build -Zbuild-std ``` - Unwinding allows Rust destructors to run, preserving state and cleaning up resources instead of terminating the entire instance. - `std::panic::catch_unwind` can translate a Rust panic into a recoverable `Result`. ## Changes to wasm-bindgen - The Walrus WebAssembly parser was updated to understand `try`/`catch` exception-handling instructions. - The descriptor interpreter was updated to evaluate code containing exception blocks. - Generated exports now catch Rust panics at the Rust–JavaScript boundary and expose them as `PanicError` exceptions. - Async exports reject their JavaScript promises with `PanicError`. - Exported functions use `extern "C-unwind"` so unwinding is explicitly permitted across the boundary. - A `MaybeUnwindSafe` trait checks `UnwindSafe` requirements only when compiling with `panic=unwind`. - For closures that cannot safely unwind, `Closure::new_aborting` provides an explicit alternative that terminates on panic rather than risking invalid state. ## Results of `panic=unwind` - Panics in exported Rust functions are caught by `wasm-bindgen`. - JavaScript receives a `PanicError`. - Async calls reject their promises instead of poisoning the Worker. - Rust destructors execute correctly. - The WebAssembly instance remains valid and reusable. - Stateful applications, including Durable Objects, can recover without losing all in-memory state. ## Abort Recovery - `panic=unwind` cannot handle aborts such as out-of-memory failures because aborts do not unwind. - The remaining recovery mechanism prevents Rust code from being re-entered after an abort, avoiding repeated execution in a corrupted WebAssembly state. - Together, unwinding and abort recovery prevent one failed request from poisoning sibling or future requests. The recommended approach is to use the latest `wasm-bindgen` and Rust Workers releases, enabling `panic=unwind` where state preservation matters while using explicit aborting closures when unwind safety cannot be guaranteed.

cloudflare

Unweight: how we compressed an LLM 22% without sacrificing quality (opens in new tab)

Unweight is Cloudflare’s lossless compression system for LLM weights, reducing model size by 15–22% while preserving bit-exact outputs. It targets the memory-bandwidth bottleneck in GPU inference by compressing weights in HBM and decompressing them directly into fast on-chip memory before tensor-core computation. On Llama-3.1-8B, the approach saves roughly 3 GB of VRAM and enables more models to run per GPU. ## The GPU Memory Bottleneck - LLM inference is often limited by memory bandwidth rather than computation. - Each generated token requires reading the model’s weights from GPU high-bandwidth memory (HBM). - NVIDIA H100 tensor cores can process data far faster than HBM can supply it. - Smaller weights reduce the amount of data transferred across the memory bus. - Decompression must be carefully integrated: if it adds latency that cannot overlap with matrix multiplication, token generation becomes slower. ## Why Lossless Compression Matters - Quantization commonly converts 16-bit values into 8- or 4-bit integers. - Because quantization is lossy, it can change model behavior and response quality unpredictably. - Unweight instead preserves exact outputs and does not require specialized hardware. - Existing systems were unsuitable because they focused on CPU decompression, custom FPGA hardware, or consumer GPUs rather than Hopper-generation GPUs and production inference. ## Compressing BF16 Weights - BF16 values contain: - A sign bit - An 8-bit exponent - A 7-bit mantissa - Sign and mantissa values appear largely random and are difficult to compress. - Exponents are highly predictable: the 16 most common exponent values account for more than 99% of weights in a typical layer. - Unweight applies Huffman coding to exponent bytes while leaving sign and mantissa bits unchanged. - Rare exponents are handled by storing an entire row of 64 weights verbatim, avoiding per-element branching during decoding. ## Selective Compression of Model Layers - Unweight compresses the MLP gate, up, and down projection matrices. - These matrices represent roughly two-thirds of model parameters and generate substantial memory traffic during decoding. - Attention weights, embeddings, and layer norms remain uncompressed. - The exponent compression produces about 30% savings in the targeted streams and approximately 20% reduction in total MLP weight size. - Overall model-size reductions reach 15–22%. ## Direct GPU Decompression - Model weights normally reside in large but slower HBM and are staged into small, fast shared memory before computation. - Conventional approaches decompress full matrices back into HBM and then run standard matrix multiplication, creating additional memory traffic. - Unweight decompresses weights in shared memory and feeds them directly to tensor cores. - Different execution strategies are used depending on the weight matrix and batch size. - An autotuner selects the fastest strategy for each workload. ## Results and Availability - Tests on Llama-3.1-8B achieved: - Around 30% compression for MLP weights - 15–22% reduction in total model size - Approximately 3 GB of VRAM savings - The savings allow more models to fit on each GPU, potentially reducing inference cost and improving global deployment coverage. - Cloudflare is publishing a technical paper and open-sourcing the GPU kernels. Unweight demonstrates that lossless, inference-time compression can improve GPU utilization without changing model behavior. The practical recommendation is to compress the portions of a model that dominate memory traffic while integrating decoding directly into the GPU execution path.

cloudflare

Launching Cloudflare’s Gen 13 servers- trading cache for cores for 2x edge compute performance (opens in new tab)

Cloudflare’s Gen 13 servers use AMD EPYC 5th Gen Turin processors to provide up to twice as many cores as Gen 12. However, Turin’s much smaller per-core cache caused the legacy FL1 request-handling layer to suffer severe latency increases, despite higher throughput. Cloudflare found that tuning alone could not fully solve the problem, reinforcing the need for FL2, a Rust-based rewrite designed to scale with cores rather than depend heavily on cache. ## Turin’s Core-Heavy Architecture - Gen 13 Turin processors offer: - Up to 192 cores and 384 SMT threads, compared with Gen 12’s 96 cores. - Improved instructions per cycle through the Zen 5 architecture. - Up to 32% lower power consumption per core. - DDR5-6400 support for greater memory bandwidth. - The tradeoff is substantially less cache: - Gen 12 Genoa-X provides 12 MB of L3 cache per core through 3D V-Cache. - The 192-core Turin 9965 provides only 2 MB per core. - This architecture favors aggregate throughput but challenges workloads dependent on cache locality. ## FL1’s Cache and Latency Problems - FL1, based on NGINX and LuaJIT, was optimized for Gen 12’s large cache. - AMD uProf measurements showed: - Dramatically higher L3 cache miss rates on Turin. - More requests requiring slow DRAM access. - Increasing latency as CPU utilization and cache contention rose. - An L3 hit takes roughly 50 CPU cycles, while a DRAM fetch can take more than 350 cycles. - As a result, Gen 13’s additional cores delivered throughput gains but introduced unacceptable latency penalties. ## Throughput Gains at an Unacceptable Cost - With FL1, Gen 13 produced: - 10% more throughput on the 128-core Turin 9755. - 31% more on the 160-core Turin 9845. - 62% more on the 192-core Turin 9965. - The Turin 9965 offered the strongest total-cost-of-ownership benefits. - However, latency increased by more than 50% at high CPU utilization, which would negatively affect customer experience and violate performance requirements. ## Hardware and Resource Tuning - Cloudflare tested several mitigations with AMD: - Hardware prefetcher and Data Fabric Probe Filter adjustments produced only marginal improvements. - Adding FL1 workers increased throughput but took resources away from other services. - CPU pinning and isolation provided limited benefits. - AMD’s Platform Quality of Service (PQOS) was used to control cache and memory-bandwidth sharing across Turin’s Core Complex Dies. ## Cache Isolation with PQOS - Reserving part of a single CCD’s cache for FL1 produced less than 5% additional throughput. - Configurations assigning FL1 50–75% of each CCD’s cache also delivered less than 5% improvement and caused minor degradation elsewhere. - A socket-level approach was more successful: - Six of twelve CCDs, aligned with a NUMA domain, were dedicated to FL1. - This provided more than 15% incremental throughput while keeping latency acceptable. - These results showed that workload placement and cache locality could help, but they were not a complete substitute for software designed around Turin’s cache profile. Cloudflare’s broader solution was FL2, a Rust-based rewrite of its core request-handling layer. By reducing dependence on large per-core caches, FL2 enabled Gen 13’s higher core count to translate into scalable edge-compute performance without the latency penalties seen with FL1.

cloudflare

Inside Gen 13- how we built our most powerful server yet (opens in new tab)

Cloudflare’s Gen 13 server is a hardware redesign aligned with its Rust-based FL2 request-processing stack. By choosing a 192-core AMD EPYC Turin 9965, doubling memory, expanding storage and networking, and adding stronger security and accelerator support, Cloudflare targets up to twice the throughput of Gen 12 while remaining within latency limits. The design also improves efficiency, rack density, and operational simplicity. ## Gen 13 at a Glance - Uses a 2U, single-socket design. - Key specifications: - 192-core AMD EPYC 9965 processor - 768 GB of DDR5-6400 memory - Three 7.68 TB E1.S PCIe 5.0 NVMe drives - Dual 100 GbE OCP 3.0 networking - 1,300W Titanium-grade power supply - ASPEED AST2600 BMC and AST1060 hardware root of trust - Compared with Gen 12, Gen 13 provides: - Up to 2× throughput - Up to 50% better performance per watt - Up to 60% more throughput per rack at the same power budget - Twice the memory capacity, 1.5× the storage, and 4× the network bandwidth - PCIe encryption in addition to memory encryption - Better support for high-heat PCIe accelerators ## Choosing the CPU - Gen 12 used a 96-core AMD EPYC Genoa-X 9684X with: - 400W TDP - 1,152 MB of L3 cache - Cloudflare evaluated three Turin processors: - 9755: strongest per-core performance - 9845: lower socket power and fewer cores - 9965: highest core count and better efficiency - The Turin 9965 was selected with 192 cores and 384 threads, doubling Gen 12’s hardware threads. - Its L3 cache is much smaller—384 MB total, or 2 MB per core versus Gen 12’s 12 MB per core—but FL2 workloads depend less on large L3 caches than the previous FL1 stack. - FL2 scales nearly linearly with additional cores, allowing the 9965 to deliver up to 100% higher throughput. - Production testing showed the 9965 achieved the best aggregate requests per second and favorable performance per watt at its 500W TDP. - Higher compute density also means fewer servers to provision, patch, monitor, and operate. - Turin’s support for DDR5-6400, PCIe 5.0, and CXL 2.0 provides a longer upgrade and security-support runway. ## Memory Bandwidth and Capacity - Gen 13 doubles memory from 384 GB to 768 GB while retaining 4 GB per core. - All twelve memory channels are populated using one 64 GB DDR5-6400 ECC RDIMM per channel. - This configuration delivers approximately 614 GB/s of peak memory bandwidth per socket, a 33.3% increase over Gen 12. - Using identical DIMMs across all channels enables balanced interleaving, distributing memory accesses across the full memory subsystem. - The design is intended to prevent the 192-core processor from being starved of data during highly parallel workloads. Cloudflare’s central design choice was to match hardware to the characteristics of FL2 rather than preserve Gen 12’s cache-heavy strategy. For workloads that scale well across cores, the Turin 9965 and fully populated memory system offer higher throughput, better rack economics, and simpler fleet operations.

cloudflare

Shedding old code with ecdysis: graceful restarts for Rust services at Cloudflare (opens in new tab)

Cloudflare’s open-source Rust library **ecdysis** enables zero-downtime restarts for high-volume network services. It preserves listening sockets and existing connections while a new process initializes, avoiding refused connections and dropped requests. After five years of production use, Cloudflare uses it to safely deploy fixes, security patches, and new features across its global infrastructure. ## Why Conventional Restarts Fail - Stopping the old process before starting the new one creates a period when no process is listening. - New clients receive `ECONNREFUSED`; even a 100 ms gap can drop hundreds of connections at a busy location. - Existing connections—including file uploads, video streams, WebSockets, and gRPC streams—are terminated when the old process exits. - `SO_REUSEPORT` allows multiple processes to bind the same port, but can orphan connections: - The kernel assigns an incoming `SYN` to one listening socket. - If that process exits before calling `accept()`, the queued connection is terminated. - This makes simply overlapping two independently bound processes unsafe for graceful upgrades. ## The ecdysis Restart Model ecdysis uses a process-forking approach pioneered by NGINX: - The parent calls `fork()` to create a child. - The child replaces itself with the new executable using `execve()`. - The child inherits the listening socket file descriptors through a named pipe shared with the parent. - The parent continues serving traffic while the child initializes. - Once the child signals readiness, the parent closes its copy of the listening socket and drains existing connections. - Both processes may briefly accept connections during the transition, but this is intentional and avoids coverage gaps. ## Crash Safety and Upgrade Requirements - The old process can fully shut down after the replacement is ready. - The new process receives time to initialize before taking over. - If initialization fails—for example, because of invalid configuration—the child exits while the parent continues serving normally. - Upgrades are serialized so that only one runs at a time, preventing cascading failures. - The unchanged listening socket ensures that new connections are not refused during the handoff. ## Rust and System Integration - ecdysis provides native Tokio stream wrappers for asynchronous Rust services. - Synchronous services can use it without an async runtime. - With the `systemd_notify` feature enabled, it integrates with systemd lifecycle notifications. - Configuring a service with `Type=notify-reload` allows systemd to track graceful upgrades correctly. Cloudflare’s approach demonstrates that graceful restarts require coordination between processes rather than simply starting a second server. Services needing reliable zero-downtime upgrades can use ecdysis to preserve connections, tolerate failed deployments, and safely roll out new Rust binaries.

cloudflare

What came first- the CNAME or the A record (opens in new tab)

A memory-optimization change in Cloudflare’s 1.1.1.1 resolver accidentally reordered DNS records, placing CNAMEs after A/AAAA records. Although DNS record order is generally considered irrelevant, some clients—including glibc’s `getaddrinfo`—process answers sequentially and require CNAMEs to appear first. The resulting failures affected users globally until Cloudflare reverted the release. ## Incident Timeline - **December 2, 2025:** The record-reordering change was added. - **December 10:** It reached the testing environment. - **January 7, 2026:** Global deployment began. - **January 8, 17:40 UTC:** The change reached 90% of servers. - **18:19:** The incident was declared. - **18:27:** The release was reverted. - **19:55:** The revert completed and the impact ended. ## How CNAME Chains Are Resolved - A hostname may point through multiple aliases before reaching an A or AAAA record: - `www.example.com → cdn.example.com → server.cdn-provider.com → 198.51.100.1` - Each record has its own TTL and may expire independently. - If only part of a chain expires, 1.1.1.1 can reuse the cached portion and resolve only the missing records. - The resolver then combines the cached CNAME records with newly resolved address records. ## The Memory Optimization That Changed Ordering - Previously, the resolver created a new list: - Insert the existing CNAME chain first. - Append the newly resolved A/AAAA records afterward. - The optimization avoided allocations and copies by appending new CNAME records directly to the existing answer list. - This caused some responses to place address records before CNAME records. ## Why Some DNS Clients Failed - Many clients treat answer-section ordering as irrelevant, but some parse records sequentially. - These clients: - Start by looking for records matching the original queried name. - Update the expected name when they encounter a CNAME. - Accept the corresponding A or AAAA record only after that update. - With the expected order, the client sees the CNAME first and then accepts the address record. - With the address record first, it ignores the address because it does not yet match the expected name. After encountering the CNAME, there are no records left to process, so it reports an empty response. - The affected implementation included glibc’s `getaddrinfo`, widely used for DNS resolution on Linux. The incident demonstrates that even seemingly insignificant DNS response ordering can matter in practice. Resolver implementations should preserve CNAME-before-address ordering, and DNS clients should avoid assuming that record order is meaningful unless the protocol explicitly requires it.