How we built the Datadog heatmap to visualize distributions over time at arbitrary scale
Datadog uses DDSketch-powered distribution metrics and heatmaps to reveal performance patterns that percentile lines can hide. Heatmaps preserve the full shape of latency distributions across hosts and time, making distinct behavioral modes, seasonality, and outliers visible. The visualization is designed to remain scalable and readable even with hundreds of trillions of underlying datapoints. ## Why Unaggregated Distributions Matter - Line graphs reduce billions of events to a single value, such as p50, p99, or max. - Multiple percentile lines provide more context, but the selected percentiles remain arbitrary and can obscure important behavior. - Aggregated percentile changes may suggest that all requests are slowing when only one subset of traffic is changing. - Heatmaps expose separate “modes”—distinct groups of measurements with different behavior. - For example, periodic latency spikes may come from a low-latency benchmarking service rather than from a general degradation in the endpoint. - Filtering out an identified mode can reveal other patterns, such as daily seasonality in the remaining traffic. ## Building Heatmaps with DDSketch - DDSketch sacrifices a small amount of precision to represent extremely large numbers of observations efficiently. - Datadog sends histogram bins and counts to the frontend instead of transmitting every individual datapoint. - Limiting the number of bins keeps the payload size constant as traffic volume grows. - Counts use `float32`, supporting values up to approximately `3 × 10^38` per bin—far beyond practical monitoring volumes. - This allows heatmaps to represent massive datasets, including hundreds of trillions of datapoints. ## Preserving Resolution and Avoiding Aliasing - Heatmap requests contain time buckets, distribution bins, and counts. - Since bucket boundaries are shared across a request, Datadog stores those boundaries only once. - Boundaries must be explicit because distributions may use logarithmic rather than linear scales. - Time buckets need to align with the source data intervals. - Misaligned intervals create aliasing artifacts: for example, grouping 10-second data into 7-second buckets produces repeating count patterns such as `[1, 1, 2, 1, 1, 2, …]`. - Careful discretization preserves the resolution available in the original DDSketch data. ## Designing the Color Scale - The default palette begins with light blue, consistent with other single-series Datadog visualizations. - It transitions toward purple to match Datadog’s visual identity. - The scale avoids lingering on red, which can imply negative alerts, and ends in orange for the hottest values. - Color choices must communicate both the volume and structure of the distribution. ## Maintaining Dynamic Range - A few high-count bins can dominate a linear color scale, leaving most of the heatmap visually indistinguishable. - This is especially problematic for power-law distributions with a dense central mode and a long tail. - A linear scale may clearly show the main mode around 20 ms while hiding a smaller mode near 1 second. - Human brightness perception is nonlinear, approximately following a power law described by Stevens’ law. - Applying nonlinear color interpolation improves the visibility of meaningful differences across both dense regions and long tails. - This helps preserve distribution details that would otherwise be lost when the color range is dominated by outliers or highly concentrated buckets. Datadog’s heatmap approach combines DDSketch compression, aligned high-resolution buckets, and perceptually informed color scaling. For systems where averages or a handful of percentiles conceal important subpopulations, distribution heatmaps provide a more reliable way to investigate performance at scale.
Read original(opens in new tab)