Achieving relentless Kafka reliability at scale with the Streaming Platform
Datadog built a Streaming Platform to make Kafka resilient and dynamically manageable across hundreds of clusters, thousands of topics, and millions of partitions. By decoupling applications from physical Kafka resources, the platform can reroute traffic, rebalance workloads, and fail over in seconds without redeployments. Its design trades strict processing order for parallelism and durability, while adding mechanisms to prevent poisoned events and backlogs from blocking entire partitions. ## A Fluid Control Plane for Multi-Cluster Orchestration - Kafka infrastructure is treated as interchangeable commodity hardware rather than a fixed application dependency. - Workloads can shift between clusters, availability zones, and Kubernetes environments automatically. - The control plane continuously replaces unhealthy components and maintains uninterrupted data flow. ### Resilient Pipelines with Streams - Streams abstract away Kafka topics and clusters behind stable identifiers. - A single Stream can span multiple Kafka clusters, availability zones, and Kubernetes clusters. - Producers and consumers do not need to know the underlying Kafka topology. - Infrastructure can be reconfigured in real time without application changes or redeployments. ### Live Traffic Failovers - When a cluster becomes unhealthy, the platform creates a replacement topic and redirects new traffic to it. - Consumers drain the backlog from the original topic before transitioning to the replacement. - The same mechanism supports proactive traffic redistribution, cluster decommissioning, capacity changes, and partition adjustments. - Operations that traditionally take hours can be completed in seconds. ### Consumer Semantics for Scale and Durability - Datadog uses at-least-once delivery rather than strict processing order. - Events can be processed independently and in parallel. - Ordering is restored later in the event store. - This trade-off enables efficient processing of petabytes of data across distributed infrastructure. ### The Assigner Coordinator - Kafka’s default group coordinator depends on session timeouts, making failover detection take tens of seconds or minutes. - Datadog’s Assigner monitors cluster health, resource usage, and workload distribution continuously. - It reacts in seconds to failures, traffic spikes, and capacity changes. - Workloads are balanced using real metrics such as CPU load and available resources, allowing heterogeneous environments to be used efficiently. ## Preventing Head-of-Line Blocking Kafka’s strict per-partition ordering means a single unprocessable event can block all later events. Datadog addresses this reliability problem through independent Stream lanes and a more flexible commit log. ### Stream Lanes and Quality of Service - Streams contain separate lanes for different priority levels and traffic requirements. - High-priority real-time traffic can be isolated from slower bursts or late-arriving data. - A dedicated dead-letter queue lane receives poison pills that cannot be processed. - Consumers can commit progress after moving failed events to the DLQ, preventing one bad event from blocking the partition without losing data. ### Advanced Commit Logging - Kafka traditionally maintains one committed pointer per partition. - That model prevents consumers from advancing past older events that are delayed or still being processed. - Datadog uses Kafka’s commit metadata to record multiple offsets or offset ranges simultaneously. - This allows consumers to continue processing live traffic while older events are handled separately, reducing backlog-related blocking. ## Overall Design Philosophy - The Streaming Platform combines stable logical Streams, real-time orchestration, flexible consumer semantics, QoS isolation, and enhanced offset tracking. - Together, these components make Kafka more self-healing and suitable for Datadog’s extreme scale. - The approach prioritizes uninterrupted processing, rapid recovery, and operational flexibility over Kafka’s default assumptions of fixed topology and strict ordering. Datadog’s design demonstrates that Kafka at very large scale requires a control layer around the broker infrastructure. Abstracting resources, automating failovers, and allowing independent progress through partitions can provide substantially better reliability and throughput than relying solely on Kafka’s native coordination model.
Read original(opens in new tab)