Introducing Husky, Datadog's third-generation event store
Datadog built Husky, a new event-storage system, after its original log architecture struggled with multi-tenant reliability, rapid platform growth, and evolving product requirements. The post explains how Datadog moved from metrics-oriented storage to event storage, introduced custom sharding and routing, and eventually recognized the need for a more flexible system. Husky emerged from these lessons about isolation, scalability, and retaining high-cardinality event data. ## From Metrics to Logs - Metrics systems store pre-aggregated tuples such as `<timeseries_id, timestamp, float64>`. - Aggregation makes metrics extremely efficient: millions of events in a second can become one compact datapoint, often requiring less than two bytes with delta-of-delta encoding. - This model is poorly suited to logs because logs must preserve individual events and their full context. - Metrics typically favor long-lived, low-cardinality dimensions such as: - Datacenter - Service - Pod name - Short-lived, high-cardinality fields such as transaction IDs and packet IDs are usually pre-aggregated or omitted. - Logs instead need to support: - Multi-kilobyte events - High-cardinality values such as UUIDs and stack traces - Arbitrary aggregations performed at query time ## Limitations of the Initial Logs System - Datadog’s first Logs architecture initially worked well but became vulnerable in a multi-tenant environment. - A single unhealthy or overloaded node could degrade service for every tenant in the cluster. - Scaling overloaded clusters could worsen the situation because nodes began streaming data to one another while already handling excessive read and write workloads. - Diagnosing and mitigating these cascading failures was difficult. ## Separating Storage from Clustering Datadog’s second architecture retained the same single-node storage engine but moved clustering responsibilities into dedicated services. - Storage nodes no longer knew about one another and behaved like independent one-node clusters. - Failures were isolated to the tenants assigned to a particular shard instead of spreading across the entire cluster. - A Shard Router: - Read events from Kafka - Reorganized them into shard-based Kafka partitions - Dynamically assigned tenants to an appropriate number of shards based on their recent five-minute data volume - Each shard was consumed by two storage-node replicas for redundancy. - A custom query engine tracked tenant-to-shard assignments, queried the relevant replicas, merged partial aggregates, and produced final results. ## Growth of the Event Platform - The new architecture substantially improved reliability and reduced operational burden. - Datadog expanded the platform beyond Logs to support products including: - Network Performance Monitoring - Real User Monitoring - Continuous Profiler - These products generated structured, multi-kilobyte events with storage and indexing requirements similar to logs. - As usage grew, new problems appeared: - A tenant producing a sudden burst of events could degrade query performance for other tenants sharing its shard. - Product teams requested longer retention for important but infrequently queried data, while still requiring it to remain immediately queryable. - The existing architecture was increasingly difficult to adapt to these isolation, scalability, and retention requirements, motivating the development of Husky. Datadog’s progression shows that event storage cannot simply reuse metrics-oriented designs. Systems must preserve event-level context, isolate tenants from one another, and support changing retention and query requirements as products and workloads evolve.
Read original(opens in new tab)