cloudflareJul 8, 2026

Introducing Meerkat- an experiment in global consensus (opens in new tab)

Cloudflare is building Meerkat, an experimental global consensus service for coordinating control-plane state across more than 330 data centers. It aims to provide linearizable reads and writes while remaining available despite machine failures, network degradation, and data-center outages. Meerkat uses QuePaxa rather than Raft because QuePaxa allows all replicas to write and does not halt progress while waiting for failure timeouts. ## The Challenge of Global Control-Plane State - Cloudflare services need to read and modify shared state from locations around the world. - Examples include: - Placement information for resources such as AI model instances. - Leadership information identifying which machine may write to a database. - The system must combine: - Strong consistency, so readers do not observe conflicting or stale state. - High availability, even when machines, links, queues, or data centers fail. - Wide-area networks are unpredictable, making replica synchronization difficult. ## Why Consensus Is Needed - Consensus algorithms allow machines to agree on a single ordered sequence of operations, such as key-value-store reads and writes. - A typical consensus system can continue safely as long as a majority of replicas remain alive and connected. - This provides a foundation for applications such as: - Transactional key-value stores. - Distributed leases and locks. - Database leadership management. ## Limitations of Raft in Wide-Area Networks - Raft depends on a leader, and only the leader can accept writes. - If the leader crashes or becomes unreachable, the system may become unavailable until a timeout triggers leader election. - Timeout configuration is especially difficult across global networks with unpredictable latency. - A single failed machine or degraded network link can therefore affect availability. - Cloudflare reports having experienced incidents caused by unavailable leaders in consensus-based systems. ## Strong Consistency and Linearizability - Consistency determines how concurrent reads and writes may be ordered or observed. - Weak consistency can allow writes to be reordered. - Stronger models may preserve write ordering while still allowing reads to observe different states. - Linearizability is the strongest model described: - Operations appear to occur in real-time order. - Every read after a completed write observes that write. - Linearizability lets developers reason about distributed state similarly to local memory on a single-threaded machine. - Meerkat’s planned key-value store also provides serializability, which Cloudflare says will be covered separately. ## Fault-Tolerance Requirements Meerkat is intended to remain available and correct under several classes of failure: - The system should support reads and writes from any data center when: - A majority of machines are alive and can communicate. - A client can reach a machine connected to that majority. - In a system of `2f + 1` machines, the design tolerates `f` faults. - Single-machine failures and individual network-link degradations should not interrupt availability. - The system must remain correct during: - Machine crashes and restarts. - Network failures and delays. - Data-center outages. - Up-to-date machines must never disagree about committed state. - Like Raft, Meerkat does not attempt to tolerate Byzantine faults or actively malicious participants. ## Introducing Meerkat and QuePaxa - Meerkat is being developed by Cloudflare Research as an internal, experimental consensus service. - It is powered by QuePaxa, a consensus algorithm published by EPFL researchers in 2023. - Unlike Raft: - Any replica can perform writes. - Progress does not stop because a timeout expires or a leader becomes unavailable. - Applications will be layered on Meerkat’s consensus log, initially focusing on small control-plane data. - The first use cases include database leadership and other coordination state. - Cloudflare describes this as the first planned industrial deployment of QuePaxa at global scale. - Meerkat will remain internal while it is still under development.

key-value-storedistributed-systemsfault-tolerancequepaxa+4

cloudflareApr 17, 2026

Introducing Flagship: feature flags built for the age of AI (opens in new tab)

AI-generated code is moving toward autonomous production deployment, making safety and controlled rollout essential. The post argues that feature flags provide the guardrails: agents can deploy disabled code, test it with limited cohorts, monitor results, and roll back automatically. Cloudflare’s new Flagship service is designed for this workflow, evaluating flags at the edge through Workers, KV, and Durable Objects. ## Feature Flags for Autonomous Deployment - Agents can ship code behind an off flag without affecting users. - They can enable features for themselves or small test cohorts, observe metrics, and expand or disable rollouts. - Humans define boundaries while flags limit the blast radius. - This separates not only deployment from release, but also routine shipping decisions from constant human attention. ## Problems with Feature Flags on Workers - Hardcoded flags are initially convenient because Workers deploy quickly. - Over time, flags become fragmented across teams, with no central visibility or audit trail. - Troubleshooting may require searching version history with tools such as `git blame`. - Calling an external flag service adds a network request to every user request, potentially introducing significant latency. - This undermines the advantage of running applications close to users at the edge. ## Why Local Evaluation Is Difficult on Workers - Traditional local-evaluation SDKs download rules into a long-lived process. - Worker isolates may be created and evicted between requests, requiring repeated initialization. - Serverless environments therefore need a distribution system with edge-local reads and managed synchronization. - Flagship uses Cloudflare KV to provide this distribution without persistent connections or per-request external calls. ## How Flagship Works - Flagship is built on Workers, Durable Objects, and KV, without external databases or centralized evaluation servers. - Durable Objects provide a globally unique, SQLite-backed source of truth for flag configuration and changelogs. - Changes are synchronized to KV within seconds and replicated throughout Cloudflare’s network. - Evaluations read configuration from KV at the edge and execute targeting and rollout logic inside the Worker isolate. - Both flag data and evaluation logic remain close to the request. ## Worker Binding and Typed Evaluation - Workers connect Flagship through a `wrangler.jsonc` binding containing a binding name and `app_id`. - The binding supports typed methods including: - `getBooleanValue()` - `getStringValue()` - `getNumberValue()` - `getObjectValue()` - `*Details()` methods return the value, matched variant, and selection reason. - Evaluation errors return the supplied default value. - Type mismatches throw exceptions because they indicate application bugs rather than temporary service failures. ## OpenFeature Integration - Flagship is built on OpenFeature, the CNCF standard for feature-flag evaluation. - It supports Workers as well as Node.js, Bun, Deno, and browser environments. - The service is currently available in closed beta. Flagship is positioned as an edge-native feature-flag system for safely automating deployment and rollout. For Cloudflare Workers, its direct binding avoids network round-trips while providing centralized configuration, targeting, auditability, and controlled release mechanisms.

key-value-storedatabase-designcloudflare-workerssqlite+4

airbnbOct 9, 2025

From Static Rate Limiting to Adaptive Traffic Management in Airbnb’s Key-Value Store (opens in new tab)

Airbnb evolved Mussel’s QoS system from static, per-client QPS limits into adaptive traffic management designed to maximize goodput. The newer approach accounts for the actual cost of requests, prioritizes critical workloads under stress, and detects hot keys or attack traffic before they overwhelm storage. Together, resource-aware quotas and real-time load shedding provide stronger protection against traffic spikes, uneven workloads, and DDoS-like bursts. ## Why Static QPS Limits Fell Short - Mussel is a multi-tenant key-value store serving millions of point and range reads across Airbnb. - Its original Redis-backed limiter assigned each client a fixed requests-per-second quota. - Requests exceeding the quota received HTTP 429 responses. - This model worked when backend effort roughly matched request count. - As usage grew, it could not account for: - The difference between a cheap one-row lookup and a 100,000-row scan. - Hot keys accessed by many clients simultaneously. - Localized storage-shard overload that affected unrelated traffic. - Sudden events such as bot floods, DDoS attacks, or large uploads. ## Resource-Aware Rate Control - Mussel replaced raw request counting with request units (RU), which represent estimated backend work. - RU calculations incorporate: - Fixed per-request overhead. - Rows and payload bytes processed. - Request latency, which distinguishes cached operations from disk-heavy ones. - The system uses calibrated linear formulas for reads and writes, with weights based on compute, network, and disk-I/O measurements. - Dispatchers debit a local token bucket according to each request’s RU cost rather than charging every request equally. - Periodic RU refills preserve simple, static quotas while making them more proportional to actual resource consumption. - Requests are rejected with HTTP 419 when the RU bucket is exhausted. - Load shedding remains separate, allowing latency-based protection to react dynamically without changing the underlying quota-refill mechanism. ## Load Shedding Under Sudden Stress - RU rate limiting smooths normal traffic but may react too slowly to rapidly changing workloads. - Mussel adds a load-shedding layer based on: - Traffic criticality. - A real-time latency ratio. - A CoDel-inspired queue-management policy. - Each dispatcher compares long-term p95 latency with short-term p95 latency. - A ratio near 1.0 indicates stable performance; a drop toward 0.3 signals rapidly increasing latency. - When stress crosses the threshold: - The system raises the effective RU cost for a designated lower-priority client class. - That class’s token bucket drains faster, causing its traffic to back off. - If conditions worsen, the penalty expands to additional classes. - Critical workloads, such as customer support and trust-and-safety traffic, can remain responsive while less important traffic is reduced. - The latency estimate uses the constant-memory P² algorithm, avoiding raw sample storage and cross-node coordination. ## Hot-Key Detection and DDoS Protection - Client-level quotas cannot prevent overload when many clients request the same popular key. - Mussel therefore detects skewed access patterns in real time. - When duplicate requests target a hot key, the system can protect storage by: - Serving responses from cache. - Coalescing identical requests before they reach the backend. - This approach protects the underlying shard whether the traffic comes from legitimate popularity, automation, or a DDoS burst. Mussel’s experience suggests that mature multi-tenant services should move beyond fixed QPS limits. Combining resource-based accounting, priority-aware load shedding, and hot-key mitigation provides a more effective way to preserve reliability while maximizing useful work during unpredictable traffic conditions.

key-value-storek8sdistributed-systemsredis+4

airbnbSep 24, 2025

Building a Next-Generation Key-Value Store at Airbnb (opens in new tab)

Airbnb rebuilt Mussel, its key-value store for derived data, from a complex EC2-based system into a cloud-native NewSQL platform. Mussel v2 combines bulk ingestion, streaming writes, low-latency reads, flexible consistency, and automated operations while supporting more than 100 existing use cases. A gradual, reversible blue/green migration moved production workloads without data loss or customer-visible downtime. ## Why Airbnb Rebuilt Mussel - New use cases—including real-time fraud detection, personalization, and dynamic pricing—required both streaming updates and large-scale bulk ingestion. - Mussel v1 had become difficult to operate and scale: - Node changes required multi-step Chef scripts on EC2. - Static hash partitioning created hotspots and latency spikes. - Consistency options were limited. - Resource consumption and costs were difficult to track. - Mussel v2 provides Kubernetes-based automation, dynamic range sharding, configurable consistency, namespace tenancy, quotas, and usage dashboards. ## Mussel v2 Architecture ### Stateless Dispatcher - A horizontally scalable Kubernetes service translates client requests into backend queries and mutations. - It supports: - Dual writes and shadow reads during migration - Retries, rate limiting, and dynamic throttling - Service-mesh security and discovery - Point lookups, range queries, prefix queries, and low-latency stale reads - Each dataname maps to a logical table, simplifying access patterns. ### Kafka-Based Write Pipeline - Writes are first persisted to Kafka for durability. - The Replayer and Write Dispatcher apply them to the backend in order. - Kafka absorbs traffic bursts and supports consistency, migrations, bootstrapping, and upgrades. - Airbnb plans to eventually rely more directly on the distributed database for ingestion and replication to reduce latency and operational complexity. ### Bulk Loading - Mussel retains support for both: - **Merge** jobs, which add data to existing tables - **Replace** jobs, which swap in a new dataset - Existing Airflow onboarding workflows transform warehouse data into a standard format and upload it to S3. - A stateless controller coordinates ingestion, while Kubernetes StatefulSet workers load data in parallel. - Deduplication, delta merges, and insert-on-duplicate-key-ignore improve throughput and reduce unnecessary writes. ## Scalable Data Expiration - Mussel v1 depended on storage-engine compaction for TTL expiration, which became inefficient at scale. - V2 uses a topology-aware expiration service: - Namespaces are divided into range-based subtasks. - Multiple workers scan and delete expired records concurrently. - Scheduling limits interference with live queries. - Max-version enforcement and targeted deletes help manage write-heavy tables. - The result is faster, more visible, and more scalable retention management. ## Blue/Green Migration - The migration had to handle massive datasets, thousands of tables, and mission-critical traffic with zero data loss and no availability impact. - Because v1 lacked table-level snapshots and CDC, Airbnb built a custom migration pipeline. - Tables were selected and migrated individually according to usage and risk. ### Migration Stages - **Blue:** All production traffic continued serving from v1. - **Shadowing:** Bootstrapped v2 tables processed parallel reads and writes, but v1 still served responses. - **Reverse:** V2 served live traffic while v1 remained available as a fallback. - **Cutover:** After validation, traffic was permanently moved to v2 one dataname at a time. - Automatic circuit breakers and fallback logic enabled rapid rollback if v2 showed errors or replication lag. - Kafka’s replication stream maintained eventual consistency between the two systems throughout the transition. ## Practical Takeaway Mussel v2 demonstrates that large datastore rearchitectures can be made safe through incremental migration, durable event logs, shadow traffic, and reversible per-table cutovers. The key recommendation is to combine a more scalable backend with strong operational automation and migration tooling, rather than attempting a single disruptive replacement.

key-value-storek8sapache-kafkadatabase-migration+4