Timeseries indexing at scale | Datadog (opens in new tab)

Datadog’s “Time Series Indexing at Scale” explains how an observability platform can index and query enormous numbers of time series without making tag-based searches prohibitively expensive. The central challenge is matching flexible combinations of metric names and tags while keeping ingestion, storage, and query latency predictable. The article presents indexing strategies and architectural trade-offs that allow Datadog to support high-cardinality telemetry at scale.

The Challenge of Time-Series Indexing

• A time series is identified not only by its metric name but also by its complete set of tags.
• Modern monitoring systems may contain billions of series generated by containers, hosts, services, and dynamic infrastructure.
• Queries often filter on multiple tags, requiring the system to efficiently find the intersection of several large sets of series.
• Indexing must support both:
  • Fast writes as new series appear
  • Low-latency reads for interactive dashboards and alerts
• High-cardinality tags make naïve database indexes expensive in both storage and query processing.

Inverted Indexes for Tags

• Datadog uses an inverted-index model that maps searchable terms—such as metric names and tag values—to the series containing them.
• A query can retrieve the posting list for each term and intersect those lists rather than scanning every time series.
• Common terms may correspond to very large lists, so the system must optimize how these lists are stored, compressed, and combined.
• The index separates metadata used to identify series from the time-series values stored for those series.

Distributed Indexing

• Index data is partitioned across machines so that no single node must hold or process the entire dataset.
• Sharding enables horizontal scaling as the number of metrics, tags, and customers grows.
• Query coordination gathers results from multiple shards and combines them into a single response.
• The design must balance:
  • Even distribution of index data
  • Avoidance of hot shards
  • Efficient fan-out during queries
  • Resilience when individual nodes fail

Managing Index Growth and Cardinality

• Dynamic environments continuously create and remove series, making index lifecycle management essential.
• Datadog must handle churn caused by short-lived containers, deployments, and changing tag values.
• Compression and compact data structures reduce the memory and storage required for posting lists.
• The system distinguishes between frequently queried data and less-used data to control resource consumption.
• Cardinality limits and indexing policies help prevent unusually large tag dimensions from overwhelming the system.

Query Performance and Trade-offs

• Indexing every possible attribute would improve search flexibility but increase write, storage, and maintenance costs.
• The platform therefore makes trade-offs between indexing coverage, freshness, and query speed.
• Query execution can combine index filtering with additional processing over the remaining candidate series.
• Caching and reuse of intermediate results can reduce repeated work for common queries.
• The architecture is designed to maintain predictable latency even as data volume and query complexity increase.

Operational Considerations

• Large-scale indexing requires monitoring the index itself, including shard balance, ingestion lag, memory usage, and query fan-out.
• Background processes must compact, expire, and rebalance index data without disrupting active queries.
• Fault tolerance is important because an index outage can affect dashboards and alerts even when the underlying metric data remains available.
• Separating indexing from time-series storage allows each subsystem to scale and evolve independently.

Datadog’s approach illustrates that scalable observability depends as much on metadata indexing as on storing metric values. Systems handling high-cardinality telemetry should use distributed inverted indexes, compact representations, careful lifecycle management, and explicit trade-offs between flexibility and operational cost.