netflix·Sep 26, 2025Original article

Building a Resilient Data Platform with Write-Ahead Log at Netflix | by Netflix Technology Blog | Netflix TechBlog (opens in new tab)

Netflix has developed a distributed Write-Ahead Log (WAL) abstraction to address critical data challenges such as accidental corruption, system entropy, and the complexities of cross-region replication. By decoupling data mutation from immediate persistence and providing a unified API, this system ensures strong durability and eventual consistency across diverse storage engines. The WAL acts as a resilient buffer that powers high-leverage features like secondary indexing and delayed retry queues while maintaining the massive scale required for global operations. ### The Role of the WAL Abstraction * The system serves as a centralized mechanism to capture data changes and reliably deliver them to downstream consumers, mitigating the risk of data loss during administrative errors or database corruption. * It provides a simplified `WriteToLog` gRPC endpoint that abstracts underlying infrastructure, allowing developers to focus on data logic rather than the specifics of the storage layer. * By acting as a durable intermediary, it prevents permanent data loss during incidents where primary datastores fail or require schema changes that might otherwise lead to corruption. ### Flexible Personas and Namespaces * The architecture utilizes "namespaces" to define logical separation, allowing different services to configure specific storage backends like Kafka or SQS based on their needs. * The "Delayed Queues" persona leverages SQS to provide a scalable way to retry failed messages in real-time pipelines without sacrificing overall system throughput. * The system can be configured for "Cross-Region Replication," enabling high availability and disaster recovery for storage engines that do not natively support multi-region data transfer. ### Solving System Entropy and Consistency * The WAL addresses the "dual-write" problem, where updates to primary stores (such as Cassandra) and search indices (such as Elasticsearch) can diverge over time, leading to data inconsistency. * It facilitates reliable secondary indexing for NoSQL databases by managing updates to multiple partitions as a coordinated sequence of events. * The platform mitigates operational risks, such as Out-of-Memory (OOM) errors on Key-Value nodes caused by bulk deletes, by staging and throttling mutations through the log. Organizations operating at scale should adopt a WAL-centric architecture to simplify the management of heterogeneous data stores and enhance system resilience. By centralizing the mutation log, teams can implement complex features like Change Data Capture (CDC) and cross-region failover through a single, consistent interface rather than building bespoke solutions for every service.

CassandraApache KafkaDistributed SystemsData Replication+4

datadog·Aug 4, 2025·4 min readCurated summary

Evolving our real-time timeseries storage again: Built in Rust for performance at scale

Datadog built a sixth-generation real-time timeseries database in Rust to keep pace with rapidly growing metric volume, cardinality, and query complexity. The new engine is designed for high throughput and low latency, reportedly achieving 60× higher ingestion performance and 5× faster peak-scale queries. Its development reflects a long evolution from general-purpose databases toward a purpose-built system with tighter control over storage, I/O, and execution. ## Datadog’s Metrics Storage Architecture - The metrics platform includes ingestion, enrichment, real-time and long-term storage, querying, and alerting. - This post focuses on real-time storage, which is split into two independently deployed services: - **RTDB:** Stores raw metric tuples of `<timeseries_id, timestamp, value>`, performs aggregations, and serves recent data. - **Index database:** Stores metric identifiers and their tags as `<timeseries_id, tags>`. - A storage router distributes incoming metrics across RTDB nodes based on load. - The query service contacts the relevant RTDB and index nodes, retrieves results, and combines them. - Each RTDB node includes: - An ingestion subsystem - A storage engine - A durability snapshot module - A gRPC query layer - Throttlers for resource management - A shared control plane coordinating these components ## Generation 1: Cassandra - Cassandra provided strong write scalability and a familiar operational model. - It was influenced by systems such as OpenTSDB and HBase. - Its main weaknesses were: - Limited flexibility for real-time queries - Difficulty supporting complex alerting and analytical workloads - Inefficient retrieval of large datasets - These limitations prompted Datadog to move to Redis. ## Generation 2: Redis - Redis improved read performance and offered a flexible, easy-to-understand storage model. - Datadog avoided Redis’s built-in clustering for reliability reasons, requiring the team to operate many independent instances. - Important drawbacks included: - Single-threaded execution limiting snapshotting during live traffic - Severe but uncommon memory-management and threading failures - Serialization and cross-process communication overhead - Inefficient memory layout, disk I/O, and CPU usage at scale - Redis nevertheless provided valuable operational insight and clarified the need for a purpose-built engine with direct control over I/O and system resources. ## Generation 3: MDBM and Memory-Mapped I/O - MDBM provided a memory-mapped key-value store based on `mmap`. - The operating system’s page cache loaded database pages on demand, making disk-backed data behave similarly to in-memory structures. - This simplified storage interactions initially, but performance degraded as workloads intensified. - Memory-mapped I/O introduced subtle performance and correctness concerns, leading Datadog to conclude that explicit I/O management would scale better. ## Generation 4: A Go-Based B+ Tree - Datadog replaced MDBM with a custom B+ tree written in Go. - The engine supported a thread-per-core-oriented design, with Go’s scheduler providing a useful foundation. - This change significantly improved throughput and latency. - It also created a platform that could be optimized more aggressively for Datadog’s workload. ## Generation 5: DDSketch and RocksDB - Datadog introduced DDSketch to support distribution metrics and accurate percentile estimation. - The existing Go engine was optimized for scalar floating-point values and was difficult to extend for sketches. - RocksDB was therefore integrated to store DDSketch data, offering flexibility and strong performance. - Over time, maintaining separate storage technologies created pressure to build a unified engine capable of handling multiple metric types efficiently. ## The Move Toward a New Engine - The progression from Cassandra to Redis, MDBM, a custom Go B+ tree, and RocksDB shows a pattern of replacing general-purpose components as scale and workload diversity increased. - Each generation solved important problems but introduced new operational or architectural trade-offs. - Datadog ultimately needed a unified, purpose-built storage system with: - High-throughput ingestion - Low-latency queries - Better support for high-cardinality data - Efficient handling of different metric types - More direct control over concurrency, memory, and I/O - The sixth generation addresses these requirements through a Rust-based real-time timeseries database. Datadog’s experience suggests that general-purpose storage systems can be effective early on, but sustained growth eventually favors a specialized engine. The practical lesson is to optimize existing infrastructure first while developing a purpose-built replacement before scale and workload complexity make incremental fixes insufficient.

Read original(opens in new tab)

CassandraDatabase DesignRustRedis+4