datadogJan 31, 2023

Performance improvements in the Datadog Agent metrics pipeline | Datadog (opens in new tab)

Datadog engineers recently optimized the Datadog Agent's metric processing pipeline to achieve higher throughput and lower CPU overhead. By identifying that metric context generation—the process of creating unique keys for metrics—was a primary bottleneck, they implemented a series of algorithmic changes and Go runtime optimizations. These improvements allow the Agent to process significantly more metrics using the same computational resources. ### Identifying Bottlenecks via CPU Profiling * Developers utilized Go’s native profiling tools to capture CPU usage during high-volume metric ingestion via DogStatsD. * Flamegraph analysis revealed that the `addSample` and `trackContext` functions were the most CPU-intensive components of the pipeline. * The profiling data specifically pointed to tag sorting and deduplication as the underlying operations consuming the most processing time. ### The Challenges of Metric Context Generation * The Agent must generate a unique hash (context) for every metric received to address it within a hash table in RAM. * To ensure the same metric always generates the same key, the original algorithm required sorting all tags and ensuring their uniqueness. * The computational cost of sorting lists repeatedly for every incoming message created a performance ceiling for the entire metrics pipeline. ### Specialization and Runtime Optimization * **Algorithmic Specialization:** The team implemented specialized sorting logic that adjusts based on the number of tags, optimizing the "hot path" for the most common metric structures. * **Hashing Efficiency:** Micro-benchmarks identified Murmur3 as the most efficient hash implementation for balancing speed and collision resistance in this use case. * **Leveraging Go Runtime:** The team transitioned from 128-bit hashes to 64-bit metric contexts. This change allowed the Agent to utilize Go's internal `mapassign_fast64` and `mapaccess2_fast64` functions, which provide optimized map operations for 64-bit keys. ### Redesigning for Performance * The original design followed a rigid "hash metric name -> sort tags -> deduplicate tags -> iterative hash" workflow. * Recognizing that sorting was the primary architectural bottleneck, the team moved toward a new design intended to minimize or eliminate the overhead of traditional list sorting during context generation. To achieve similar performance gains in high-throughput Go applications, developers should profile their applications under realistic load and look for opportunities to leverage runtime-specific optimizations, such as using 64-bit map keys to trigger specialized compiler paths.

metrics-aggregationdatabase-designgocpu-profiling+4

datadogJan 31, 2023

Performance improvements in the Datadog Agent metrics pipeline (opens in new tab)

The Datadog Agent needed to process more metrics without increasing CPU usage. Profiling showed that generating unique metric contexts—especially sorting and deduplicating tags—was a major bottleneck. Datadog improved throughput through specialized sorting paths, faster hashing, and a more efficient context-storage design. ## Identifying the Bottleneck - Datadog uses Go’s CPU and memory profiling tools to optimize the Agent’s metrics pipeline. - Profiles were captured while Agents processed large volumes of DogStatsD metrics, ensuring the results reflected real workload pressure. - Flamegraphs showed that `addSample` and `trackContext` consumed the most CPU. - Sorting-related functions, including `util.SortUniqInPlace` and `sort`, were significant contributors to that cost. ## How Metric Contexts Work - Each received metric is assigned a metric context that uniquely identifies it in an in-memory hash table. - The context must incorporate: - The metric name - Tags included in the DogStatsD message - Container-generated tags - The context is computed as a hash, so it must be fast while minimizing collisions. - Tags must be consistently ordered so the same metric always produces the same context. - The original implementation sorted tags and removed duplicates, making sorting a recurring CPU expense. ## Specialized Sorting - Performance varied according to the number of tags attached to a metric. - Datadog introduced specialized sorting paths based on tag count. - This allowed common cases to use more efficient algorithms while retaining correct ordering and deduplication. ## Faster Hashing and Map Access - Micro-benchmarks compared hash functions according to speed and uniqueness. - Murmur3 performed best for Datadog’s requirements. - Datadog also changed metric contexts from 128-bit to 64-bit hashes. - A 64-bit hash still provided sufficient collision resistance for the use case and enabled Go runtime optimizations: - `runtime.mapassign_fast64` - `runtime.mapaccess2_fast64` - These optimized map operations improved both context storage and metric sampling performance. ## Redesigning the Algorithm - Sorting served two purposes: producing an ordered tag list and helping deduplicate tags. - Because sorting was the largest bottleneck, Datadog began exploring a design that could address these responsibilities more efficiently rather than relying on a single general-purpose sort. The practical lesson is to profile under realistic load, optimize the hottest paths, and combine targeted specialization, benchmark-driven implementation choices, and data-structure redesign to increase throughput without adding CPU capacity.

metrics-aggregationperformance-optimizationgocpu-profiling+3