Our journey taking Kubernetes state metrics to the next level
Datadog contributed major scalability improvements to kube-state-metrics (KSM), after discovering that its metric collection process struggled with very large Kubernetes clusters. Millions of metrics could require tens of megabytes and tens of seconds to process every 15 seconds, forcing Datadog to reduce collection frequency. Their redesign improved collection duration by 15x and enabled more granular monitoring at scale. ## Datadog’s Kubernetes Observability Role - The Datadog Containers team monitors Kubernetes infrastructure and ensures reliable collection of: - Logs - Traces - Custom metrics - Profiles - Security signals - KSM is central to Datadog products such as Kubernetes metrics integration and Orchestrator Explorer. ## How Kubernetes State Metrics Works - KSM uses Kubernetes informers to watch objects registered with the API server. - Enabled collectors monitor resources such as pods, nodes, deployments, and services. - It generates lifecycle and metadata metrics in text-based OpenMetrics format. - Users can restrict monitored resources through the `resources` flag. - The Datadog Agent’s KSM check: - Runs every 15 seconds. - Discovers KSM containers. - Crawls their `/metrics` endpoint. - Reconciles metric metadata and applies configured label joins. - Label joins allow metadata from one metric, such as a deployment label, to become a tag on other metrics for the same object. ## Scaling Challenges - Datadog found that KSM needed to be split across multiple deployments beyond a few hundred nodes and thousands of pods. - Their deployments divided collectors by resource type: - Pods - Nodes - Services, deployments, jobs, persistent volumes, and other resources - Metric volume varied substantially: - Endpoints, jobs, and deployments produced roughly five metrics per object. - Nodes produced around nine metrics each. - Pods produced around 40 metrics each. - Large clusters with thousands of nodes and tens of thousands of pods could generate millions of metrics per scrape. - Crawling the metrics endpoint could take tens of seconds and transfer tens of megabytes. - Datadog had to reduce check frequency, sacrificing metric granularity and user experience. ## KSM’s Original Architecture - KSM v1 relied on a central loop that created a Builder and managed resource stores. - Each store used informers to track a particular Kubernetes resource. - For example, an HPA store maintained the list-and-watch logic for HorizontalPodAutoscalers. - The Builder generated metrics from the tracked resources. - Datadog identified two limitations: - Too much data was emitted and processed at query time. - The Builder did not provide a suitable extension point for custom metric-generation logic. ## Contributing the Redesign Upstream - As the KSM community prepared version 2.0 in early 2020, Datadog saw an opportunity to address its scalability and extensibility problems in the upstream project. - Rather than maintaining a private solution, the team contributed its findings and improvements to the open-source community. - The resulting work reportedly reduced metric collection duration by 15x, making high-scale, more frequent collection practical. Datadog’s experience shows that upstream open-source collaboration can solve internal infrastructure bottlenecks while improving the project for the broader Kubernetes community.
Read original(opens in new tab)