Introducing Husky, Datadog's third-generation event store

Datadog built Husky, a new event-storage system, after its original log architecture struggled with multi-tenant reliability, rapid platform growth, and evolving product requirements. The post explains how Datadog moved from metrics-oriented storage to event storage, introduced custom sharding and routing, and eventually recognized the need for a more flexible system. Husky emerged from these lessons about isolation, scalability, and retaining high-cardinality event data.

From Metrics to Logs

• Metrics systems store pre-aggregated tuples such as <timeseries_id, timestamp, float64>.
• Aggregation makes metrics extremely efficient: millions of events in a second can become one compact datapoint, often requiring less than two bytes with delta-of-delta encoding.
• This model is poorly suited to logs because logs must preserve individual events and their full context.
• Metrics typically favor long-lived, low-cardinality dimensions such as:
  • Datacenter
  • Service
  • Pod name
• Short-lived, high-cardinality fields such as transaction IDs and packet IDs are usually pre-aggregated or omitted.
• Logs instead need to support:
  • Multi-kilobyte events
  • High-cardinality values such as UUIDs and stack traces
  • Arbitrary aggregations performed at query time

Limitations of the Initial Logs System

• Datadog’s first Logs architecture initially worked well but became vulnerable in a multi-tenant environment.
• A single unhealthy or overloaded node could degrade service for every tenant in the cluster.
• Scaling overloaded clusters could worsen the situation because nodes began streaming data to one another while already handling excessive read and write workloads.
• Diagnosing and mitigating these cascading failures was difficult.

Separating Storage from Clustering

Datadog’s second architecture retained the same single-node storage engine but moved clustering responsibilities into dedicated services.

• Storage nodes no longer knew about one another and behaved like independent one-node clusters.
• Failures were isolated to the tenants assigned to a particular shard instead of spreading across the entire cluster.
• A Shard Router:
  • Read events from Kafka
  • Reorganized them into shard-based Kafka partitions
  • Dynamically assigned tenants to an appropriate number of shards based on their recent five-minute data volume
• Each shard was consumed by two storage-node replicas for redundancy.
• A custom query engine tracked tenant-to-shard assignments, queried the relevant replicas, merged partial aggregates, and produced final results.

Growth of the Event Platform

• The new architecture substantially improved reliability and reduced operational burden.
• Datadog expanded the platform beyond Logs to support products including:
  • Network Performance Monitoring
  • Real User Monitoring
  • Continuous Profiler
• These products generated structured, multi-kilobyte events with storage and indexing requirements similar to logs.
• As usage grew, new problems appeared:
  • A tenant producing a sudden burst of events could degrade query performance for other tenants sharing its shard.
  • Product teams requested longer retention for important but infrequently queried data, while still requiring it to remain immediately queryable.
• The existing architecture was increasingly difficult to adapt to these isolation, scalability, and retention requirements, motivating the development of Husky.

Datadog’s progression shows that event storage cannot simply reuse metrics-oriented designs. Systems must preserve event-level context, isolate tenants from one another, and support changing retention and query requirements as products and workloads evolve.