techlist.io
KREN

Observability

106 posts

datadog1 min readCurated summary

Profiling improvements in Go 1.18 | Datadog

The provided text does not include the blog post itself. It contains Datadog’s navigation menu and a promotional link announcing its recognition as a Leader in Gartner’s 2026 Magic Quadrant for Observability Platforms, but no technical discussion or conclusions from the referenced article. ## Available content ### Datadog’s observability platform - Datadog promotes products for: - Infrastructure and Kubernetes monitoring - Application performance monitoring and profiling - Logs, databases, and data observability - Security and cloud protection - Real-user and synthetic monitoring - CI/CD and software delivery - Incident and service management - AI-powered investigation and automation ### Gartner recognition - The page links to Datadog’s announcement that it was named a Leader in the Gartner Magic Quadrant for Observability Platforms. - The excerpt does not provide Gartner’s evaluation criteria, Datadog’s strengths or weaknesses, or supporting evidence. Please provide the article body or a working text extract for an accurate technical summary.

Read original(opens in new tab)
ObservabilityKubernetesDatadogApplication Performance Monitoring+4
datadogOriginal article

Our journey taking Kubernetes state metrics to the next level | Datadog (opens in new tab)

Datadog’s container observability team significantly improved the performance of kube-state-metrics (KSM) by contributing core architectural enhancements to the upstream open-source project. Faced with scalability bottlenecks where metrics collection for large clusters took tens of seconds and generated massive data payloads, they revamped the underlying library to achieve a 15x improvement in processing duration. These contributions allowed for high-granularity monitoring at scale, ensuring that the Datadog Agent can efficiently handle millions of metrics across thousands of Kubernetes nodes. ### Challenges with KSM Scalability * KSM uses the informer pattern to expose cluster-level metadata via the Openmetrics format, but the volume of data grows exponentially with cluster size. * In high-scale environments, a single node generates approximately nine metrics, while a single pod can generate up to 40 metrics. * In clusters with thousands of nodes and tens of thousands of pods, the `/metrics` endpoint produced payloads weighing tens of megabytes. * The time required to crawl these metrics often exceeded 15 seconds, forcing administrators to reduce check frequency and sacrifice real-time data granularity. ### Limitations of Legacy Implementations * KSM v1 relied on a monolithic loop that instantiated a Builder to track resources via stores, but it lacked efficient hooks for metric generation. * The original Python-based Datadog Agent check struggled with the "data dump" approach of KSM, where all metrics were processed at once during query time. * To manage the load, Datadog was forced to split KSM into multiple deployments based on resource types (e.g., separate deployments for pods, nodes, and secondary resources like services or deployments). * This fragmentation made the infrastructure more complex to manage and did not solve the fundamental issue of inefficient metric serialization. ### Architectural Improvements in KSM v2.0 * Datadog collaborated with the upstream community during the development of KSM v2.0 to introduce a more extensible design. * The team focused on improving the Builder and metric generation hooks to prevent the system from dumping the entire dataset at query time. * By moving away from the restrictive v1 library structure, they enabled more efficient reconciliation of metric names and metadata joins. * The resulting 15x performance gain allows the Datadog Agent to reconcile labels and tags—such as joining deployment labels to specific metrics—without the significant latency overhead previously experienced. Contributing back to the open-source community proved more effective than maintaining internal forks for scaling Kubernetes infrastructure. Organizations running high-density clusters should prioritize upgrading to KSM v2.0 and optimizing their agent configurations to leverage these architectural improvements for better observability performance.

ObservabilityDatabase DesignKubernetesDatadog+3
datadog3 min readCurated summary

Our journey taking Kubernetes state metrics to the next level

Datadog contributed major scalability improvements to kube-state-metrics (KSM), after discovering that its metric collection process struggled with very large Kubernetes clusters. Millions of metrics could require tens of megabytes and tens of seconds to process every 15 seconds, forcing Datadog to reduce collection frequency. Their redesign improved collection duration by 15x and enabled more granular monitoring at scale. ## Datadog’s Kubernetes Observability Role - The Datadog Containers team monitors Kubernetes infrastructure and ensures reliable collection of: - Logs - Traces - Custom metrics - Profiles - Security signals - KSM is central to Datadog products such as Kubernetes metrics integration and Orchestrator Explorer. ## How Kubernetes State Metrics Works - KSM uses Kubernetes informers to watch objects registered with the API server. - Enabled collectors monitor resources such as pods, nodes, deployments, and services. - It generates lifecycle and metadata metrics in text-based OpenMetrics format. - Users can restrict monitored resources through the `resources` flag. - The Datadog Agent’s KSM check: - Runs every 15 seconds. - Discovers KSM containers. - Crawls their `/metrics` endpoint. - Reconciles metric metadata and applies configured label joins. - Label joins allow metadata from one metric, such as a deployment label, to become a tag on other metrics for the same object. ## Scaling Challenges - Datadog found that KSM needed to be split across multiple deployments beyond a few hundred nodes and thousands of pods. - Their deployments divided collectors by resource type: - Pods - Nodes - Services, deployments, jobs, persistent volumes, and other resources - Metric volume varied substantially: - Endpoints, jobs, and deployments produced roughly five metrics per object. - Nodes produced around nine metrics each. - Pods produced around 40 metrics each. - Large clusters with thousands of nodes and tens of thousands of pods could generate millions of metrics per scrape. - Crawling the metrics endpoint could take tens of seconds and transfer tens of megabytes. - Datadog had to reduce check frequency, sacrificing metric granularity and user experience. ## KSM’s Original Architecture - KSM v1 relied on a central loop that created a Builder and managed resource stores. - Each store used informers to track a particular Kubernetes resource. - For example, an HPA store maintained the list-and-watch logic for HorizontalPodAutoscalers. - The Builder generated metrics from the tracked resources. - Datadog identified two limitations: - Too much data was emitted and processed at query time. - The Builder did not provide a suitable extension point for custom metric-generation logic. ## Contributing the Redesign Upstream - As the KSM community prepared version 2.0 in early 2020, Datadog saw an opportunity to address its scalability and extensibility problems in the upstream project. - Rather than maintaining a private solution, the team contributed its findings and improvements to the open-source community. - The resulting work reportedly reduced metric collection duration by 15x, making high-scale, more frequent collection practical. Datadog’s experience shows that upstream open-source collaboration can solve internal infrastructure bottlenecks while improving the project for the broader Kubernetes community.

Read original(opens in new tab)
ObservabilityDatabase DesignKubernetesDatadog+4
datadog2 min readCurated summary

How Datadog's IT team automated monitoring third-party accounts | Datadog

The provided text does not contain the tech blog post itself. It consists primarily of Datadog’s navigation menu and a promotional banner announcing its “Leader” ranking in the 2026 Gartner Magic Quadrant for Observability Platforms, so the article’s argument and conclusion cannot be reliably summarized. ## Promotional Announcement - Datadog links to a Gartner report about observability platforms. - The banner presents Datadog as a Leader in the report. ## Datadog Product Categories - **Infrastructure:** infrastructure, container, network, serverless, GPU, storage, and cloud-cost monitoring. - **Applications:** APM, service monitoring, profiling, dynamic instrumentation, and agent observability. - **Data and Logs:** database monitoring, data quality, job monitoring, log management, sensitive-data scanning, and observability pipelines. - **Security:** code, cloud, vulnerability, compliance, SIEM, workload, and application protection. - **Digital Experience:** browser and mobile RUM, session replay, synthetic monitoring, product analytics, and error tracking. - **Software Delivery:** CI visibility, test optimization, code coverage, feature flags, and developer portals. - **Service Management:** event management, incident response, SLOs, workflow automation, and case management. - **AI:** AI agents, GPU monitoring, integrations, investigation tools, and MCP support. The actual blog content—apparently related to how Datadog’s IT team automated monitoring of third-party accounts—is missing. A summary would require the article body or a complete excerpt.

Read original(opens in new tab)
ObservabilityKubernetesDatadogApplication Performance Monitoring+4
datadog1 min readCurated summary

How we minimized the overhead of Kubernetes in our job system | Datadog

The supplied content does not include the blog post itself; it contains Datadog’s navigation menu and a link titled “Moving a Job System to Kubernetes.” As a result, there is not enough article text to accurately summarize its arguments, implementation details, or conclusions. ## Available Information - The linked post appears to concern migrating a job-processing system to Kubernetes. - The surrounding page lists Datadog products for: - Infrastructure and Kubernetes monitoring - Application performance monitoring - Logs, databases, and jobs - Security and software delivery - No technical discussion, architecture description, challenges, or results from the post is included. Please provide the article’s body or a working page extract for a detailed summary.

Read original(opens in new tab)
ObservabilityKubernetesDatadogInfrastructure Monitoring+1
datadog2 min readCurated summary

Engineering spotlight: Maël Nison | Datadog

Datadog announces that it has been named a Leader in the 2026 Gartner® Magic Quadrant™ for Observability Platforms. The announcement positions Datadog as a broad observability platform spanning infrastructure, applications, logs, security, digital experience, software delivery, service management, and AI. The provided content does not include Gartner’s detailed evaluation or the blog post’s supporting arguments. ## Recognition and Platform Scope - Datadog highlights its leadership placement in Gartner’s observability-platform research. - Its platform covers: - Infrastructure and container monitoring - Application performance monitoring and profiling - Database, data-stream, and jobs monitoring - Log management and observability pipelines - Cloud, application, workload, and code security - Browser and mobile real user monitoring - Synthetic monitoring, session replay, and error tracking - CI visibility, testing, code coverage, and feature flags - Incident response, service catalogs, SLOs, and workflow automation ## AI and Automation - Datadog presents AI as an integrated part of its platform through: - Bits AI agents and investigation tools - AI integrations and agent observability - GPU monitoring - MCP Server and agent-building capabilities - AI-assisted security and developer workflows - Additional automation features include Watchdog, fleet automation, workflow automation, and incident-management tools. ## Overall Positioning - The product catalog emphasizes a unified approach to monitoring technology environments rather than separate tools for infrastructure, applications, security, and user experience. - The platform also includes dashboards, alerts, notebooks, governance controls, access management, and mobile access. The announcement’s central message is that Datadog combines extensive observability coverage with security, delivery, service-management, and AI capabilities. Readers seeking the actual Gartner assessment should consult the linked Magic Quadrant resource, since the supplied text contains only the announcement and navigation information.

Read original(opens in new tab)
ObservabilityKubernetesDatadogCI/CD+4
datadog3 min readCurated summary

PHP 8: Observability baked right in

PHP’s observability mechanisms failed to keep pace with Zend Engine improvements in PHP 7 and PHP 8, especially the introduction of JIT. Existing hooks imposed significant runtime costs, created compatibility and stability problems, and limited tracers such as Datadog’s ability to evolve. PHP 8 addressed these issues by introducing a new observer API designed specifically for modern, lower-overhead runtime instrumentation. ## Observability Before PHP 8 ### The `zend_execute_ex` VM Hook - Extensions could override `zend_execute_ex` to intercept every PHP-defined function and method call. - This moved PHP calls onto the native C stack, whose limited size (`ulimit -s`) could cause stack overflows and process crashes. - Every userland call was intercepted, even when an extension only needed to observe a subset, adding overhead to call-heavy applications. - The compiler could no longer use optimized distinctions between userland and internal calls, such as `DO_UCALL` and `DO_ICALL`. - Extensions had to manually forward the hook to other extensions, creating “noisy neighbor” problems, unexpected behavior, and possible crashes. - The hook was incompatible with PHP 8’s JIT compiler. ### Custom Opcode Handlers - Extensions could replace handlers for function-call opcodes, avoiding the native-stack problem associated with `zend_execute_ex`. - These handlers still required careful forwarding to neighboring extensions, which was historically unreliable. - Handlers could mutate VM state—for example, preventing the original opcode from running—making reliable cooperation between multiple extensions impossible in some cases. - Generators could not be fully instrumented through custom opcode handlers. - Like `zend_execute_ex`, custom opcode handlers were incompatible with the PHP 8 JIT. ### Zend Extension Hooks - Zend Extensions had privileged access to engine-level function-call begin and end handlers. - This approach caused the compiler to emit `EXT_FCALL_BEGIN` and `EXT_FCALL_END` around every function call. - The additional opcodes introduced too much overhead for production-grade tracing. ### AST Injection Experiments - Researchers explored injecting observability nodes into the abstract syntax tree during compilation. - These nodes could invoke tracing functions before and after calls. - However, injecting instrumentation around every function call was expected to have overhead comparable to Zend Extension hooks. - No production-ready tracers using this approach were known at the time. ## The Need for a New Observer API - Existing hooks forced observability tools to interfere deeply with VM execution or compiler output. - Their limitations included excessive overhead, stack-safety risks, incomplete generator support, extension conflicts, and JIT incompatibility. - These constraints prevented tools such as the Datadog PHP tracer from taking full advantage of PHP 8. - In response, the authors and the PHP internals community developed and shipped the observer API in PHP 8, providing a foundation for more modern and efficient tracing, profiling, and debugging. PHP 8’s observer API was necessary because older instrumentation techniques were either unsafe, too slow for production, difficult to compose, or incompatible with the JIT. A runtime-level observability mechanism designed alongside the engine is a more sustainable approach than modifying VM hooks, opcodes, or compiled syntax from extensions.

Read original(opens in new tab)
ObservabilityDatabase DesignDatadogPhp+4
datadog1 min readCurated summary

PHP 8: Observability baked right in | Datadog

Datadog announces that Gartner named it a Leader in the 2026 Magic Quadrant for Observability Platforms. The supplied content contains the announcement headline and Datadog’s product navigation, but not the report’s evaluation details, methodology, or supporting arguments. ## Gartner Recognition - Datadog is positioned as a Leader in Gartner’s Magic Quadrant for Observability Platforms. - The linked resource appears to provide the full Gartner report or announcement. - No specific Gartner strengths, cautions, rankings, or comparison with other vendors are included in the provided text. ## Datadog’s Observability Portfolio The page navigation highlights Datadog’s broad platform, including: - Infrastructure monitoring, metrics, containers, Kubernetes, networks, serverless, and cloud costs - APM, service monitoring, profiling, and dynamic instrumentation - Database, data-stream, job, and quality monitoring - Log management, observability pipelines, and sensitive-data scanning - Real-user monitoring, session replay, synthetic monitoring, and error tracking - CI visibility, testing, code coverage, and software delivery tools - Incident response, service catalogs, SLOs, workflow automation, and event management - AI capabilities such as agent observability, Bits AI, GPU monitoring, and MCP integrations Overall, the material presents Datadog as a broad, integrated observability platform, but the actual Gartner analysis is not included. For a detailed assessment, consult the linked Gartner report directly.

Read original(opens in new tab)
ObservabilityKubernetesDatadogApplication Performance Monitoring+2
datadog2 min readCurated summary

Introducing Glommio, a thread-per-core crate for Rust and Linux | Datadog

Datadog has been recognized as a Leader in the 2026 Gartner® Magic Quadrant™ for Observability Platforms. The announcement positions Datadog as a broad observability provider spanning infrastructure, applications, logs, security, digital experience, software delivery, service management, and AI. The supplied content contains mostly site navigation rather than the article’s supporting details or Gartner’s evaluation rationale. ## Datadog’s Observability Scope - Infrastructure monitoring, metrics, containers, Kubernetes, networks, serverless systems, cloud costs, GPUs, and storage. - Application performance monitoring, service monitoring, profiling, dynamic instrumentation, and agent observability. - Log management, sensitive-data scanning, audit trails, and observability pipelines. - Database, data-streams, data-quality, and jobs monitoring. ## Broader Platform Capabilities - Security features including cloud security, SIEM, workload protection, code security, vulnerability management, and compliance. - Digital-experience tools such as real-user monitoring, session replay, synthetic monitoring, error tracking, and product analytics. - Software-delivery capabilities covering CI visibility, test optimization, continuous testing, code coverage, and feature flags. - Service-management tools for incidents, events, SLOs, workflows, case management, and software catalogs. - AI offerings including Bits AI agents, investigation tools, agent observability, GPU monitoring, and MCP integrations. Overall, the announcement emphasizes Datadog’s unified and expansive observability platform. A complete assessment of Gartner’s specific strengths, cautions, and evaluation criteria would require the full blog post or linked Gartner report.

Read original(opens in new tab)
ObservabilityKubernetesDatadogApplication Performance Monitoring+3
datadog2 min readCurated summary

How we wrote a Python profiler | Datadog

The post explains how Datadog built a low-overhead statistical profiler for Python. Rather than tracing every function call, the profiler periodically samples running threads and reconstructs their Python and native call stacks. The main challenge is collecting accurate stack data without pausing applications for too long or introducing unsafe behavior inside CPython. ### Why Traditional Profiling Is Expensive - Deterministic profilers instrument every function call and return. - This provides detailed data but can significantly slow production workloads. - A statistical profiler reduces overhead by sampling execution at regular intervals instead of observing every event. ### Sampling Python Threads - The profiler interrupts running threads to capture their current execution state. - Python’s signal-handling model complicates this because signals are generally processed by the main thread. - The implementation must coordinate native threads, operating-system signals, and the Python interpreter to sample worker threads reliably. - Sampling must avoid interfering with application locks or triggering unsafe operations in signal handlers. ### Reconstructing Call Stacks - A useful profile needs both Python-level frames and native stack information. - The profiler walks Python frames to identify functions, files, and line numbers. - It also handles time spent in native extensions and the interpreter itself. - Collected samples are aggregated into call stacks, allowing Datadog to show CPU usage and hotspots across the application. ### Balancing Accuracy and Overhead - Sampling frequency affects the trade-off between detail and runtime cost. - More frequent samples improve visibility into short-lived work but consume more resources. - The profiler is designed to operate continuously in production, so it prioritizes low overhead, safe memory handling, and resilience across Python versions and deployment environments. The central recommendation is to use statistical sampling for always-on production profiling. It provides actionable performance data with far less impact than call-by-call instrumentation, provided the implementation carefully accounts for CPython’s threading, signal, and native-extension behavior.

Read original(opens in new tab)
ObservabilityKubernetesDatadogApplication Performance Monitoring+3
datadog4 min readCurated summary

Computing accurate percentiles with DDSketch | Datadog

Datadog’s post explains why accurately computing percentiles is difficult when monitoring large-scale, distributed systems. Traditional approaches either require retaining every observation or sacrifice accuracy through fixed-size summaries, especially for long-tailed data such as request latency. DDSketch addresses this by providing mergeable percentile estimates with a guaranteed relative-error bound and memory usage that remains effectively constant. ## Why Percentiles Matter - Averages can hide slow requests and do not describe the tail of a distribution. - Percentiles such as p95, p99, and p99.9 are more useful for measuring latency and reliability. - Monitoring systems must calculate these values from enormous numbers of observations across many hosts and services. - Storing every measurement is too expensive, while calculating percentiles independently on each machine and averaging the results is mathematically incorrect. ## Limitations of Common Approaches - Exact percentile calculation requires sorting or retaining all values, which is impractical for high-volume metrics. - Histograms use predefined buckets, making their accuracy dependent on bucket boundaries. - Fixed-width buckets are inefficient for distributions spanning several orders of magnitude: - Small values may require fine-grained buckets. - Large values may require a huge number of buckets. - Many quantile sketches optimize for rank accuracy, but a small rank error can still produce a large value error in heavy-tailed distributions. - Summaries must also be mergeable so that data collected from multiple agents can be combined without losing their accuracy guarantees. ## DDSketch’s Logarithmic Mapping - DDSketch groups values into logarithmically spaced bins rather than equally sized intervals. - Values close together near zero receive finer absolute resolution, while larger values receive wider buckets. - Each value is mapped to a key based on its logarithm: - Positive and negative values are handled separately. - Zero and values sufficiently close to zero use a dedicated zero bucket. - A representative value is chosen for each bucket, typically using the bucket’s geometric center. - Because adjacent buckets have a fixed ratio, the estimated value is bounded by a predictable relative error rather than a fixed absolute error. ## Relative-Error Guarantees - DDSketch is configured with a target relative accuracy, such as 1%. - Its logarithmic base is selected so that the returned quantile is within that relative-error bound of the true value. - Relative error is particularly appropriate for latency data: - An error of a few milliseconds matters greatly for a 10 ms request. - The same absolute error is much less significant for a 10-second request. - The sketch preserves accuracy across a wide range of values without requiring a proportional increase in the number of buckets. ## Distributed Aggregation and Memory Use - DDSketches can be merged by adding the bucket counts from separate sketches. - This allows agents, hosts, containers, and regional services to aggregate measurements into a global percentile. - Merging does not require access to the original observations. - The sketch stores counts rather than individual values, substantially reducing memory and network costs. - Datadog also describes bounded-memory variants that collapse older or less significant bins when necessary, allowing sketches to maintain a fixed storage limit while retaining useful tail information. ## Practical Trade-offs - Higher accuracy requires more buckets and therefore more memory. - Lower accuracy reduces resource usage but produces wider estimates. - The choice of relative accuracy should reflect the metric’s operational needs rather than defaulting to the smallest possible error. - Implementations must account for negative values, zeros, very small values, and values outside the normal range. - Accurate percentile reporting depends not only on the sketch algorithm but also on correct aggregation and consistent configuration across producers. DDSketch is therefore a practical choice for observability systems that need scalable, mergeable, and predictable percentile calculations. Its logarithmic buckets and relative-error guarantees make it especially well suited to latency and other long-tailed measurements where fixed-width histograms or rank-based approximations can be misleading.

Read original(opens in new tab)
ObservabilityKubernetesDatadogApplication Performance Monitoring+3
datadog1 min readCurated summary

Building highly reliable data pipelines at Datadog | Datadog

The provided text does not include the blog post’s actual article content. It contains Datadog’s navigation menu and a link titled “Highly Reliable Data Pipelines,” so the post’s argument, architecture, and technical conclusions cannot be summarized reliably. ## Available Information - The page appears to be a Datadog engineering blog post about building highly reliable data pipelines. - The surrounding content is primarily Datadog product navigation. - It also promotes Datadog’s recognition as a Leader in the 2026 Gartner® Magic Quadrant™ for Observability Platforms. Please provide the article body or a complete page extract for a detailed technical summary.

Read original(opens in new tab)
ObservabilityKubernetesDatadogApplication Performance Monitoring+3
datadog2 min readCurated summary

Rethinking UX for AI-driven alerting | Datadog

Datadog’s page announces that the company was named a Leader in the 2026 Gartner Magic Quadrant for Observability Platforms. The supplied content, however, primarily contains site navigation rather than the referenced blog post, so it does not provide details about the article’s argument concerning AI-driven alerting. ## Gartner Recognition - Datadog highlights its recognition as a Leader in Gartner’s Magic Quadrant for Observability Platforms. - The announcement is presented as a promotional resource linked from the Datadog website. ## Datadog’s Product Portfolio The navigation emphasizes Datadog’s broad observability and security platform, including: - **Infrastructure:** infrastructure, container, network, serverless, GPU, storage, and cloud-cost monitoring. - **Applications:** APM, service monitoring, profiling, dynamic instrumentation, and agent observability. - **Data and logs:** database, data-stream, job, quality, log, sensitive-data, and pipeline monitoring. - **Security:** code, cloud, vulnerability, workload, application, API, and SIEM security tools. - **Digital experience:** browser and mobile RUM, session replay, synthetic monitoring, product analytics, and error tracking. - **Software delivery:** CI visibility, test optimization, code coverage, feature flags, and developer portals. - **Service management:** incident response, SLOs, event management, workflows, and case management. - **AI:** Bits AI agents, investigations, chat, security analysis, agent observability, and MCP integrations. The provided text does not include enough of the actual “Rethinking UX for AI-Driven Alerting” article to summarize its technical concepts or conclusions.

Read original(opens in new tab)
ObservabilityAI AgentsKubernetesDatadog+4
datadog1 min readCurated summary

Improving trust with Datadog Log Management | Datadog

The provided content does not include the blog post’s main article text. It contains Datadog navigation links and a promotional banner announcing Datadog as a Leader in the 2026 Gartner® Magic Quadrant™ for Observability Platforms, plus a link suggesting the article concerns improving trust with Datadog Log Management. ## Datadog’s Observability Platform - Datadog promotes a broad observability platform covering: - Infrastructure and cloud monitoring - Application performance monitoring - Logs and sensitive-data protection - Security monitoring - Digital experience monitoring - Software delivery and service management - AI-powered investigation and automation - The banner highlights Datadog’s recognition as a Gartner Magic Quadrant Leader. ## Log Management and Trust - The linked article appears to focus on improving trust through Datadog Log Management. - The supplied text does not provide details about the specific problems, technologies, or recommendations discussed in the post. A complete summary requires the article’s actual body text rather than the surrounding website navigation.

Read original(opens in new tab)
ObservabilityKubernetesDatadogApplication Performance Monitoring+3