Datadog

190 posts

www.datadoghq.com/blog/engineering

Filter by tag

Datadog110 Kubernetes85 Observability83 Application Performance Monitoring64 Database Design61 Log Management56 Infrastructure Monitoring52 Cloud Security38 Synthetic Monitoring16 PostgreSQL15 Python14 Apache Kafka13+505

datadogFeb 2, 20211 min readCurated summary

PHP 8: Observability baked right in | Datadog

Datadog announces that Gartner named it a Leader in the 2026 Magic Quadrant for Observability Platforms. The supplied content contains the announcement headline and Datadog’s product navigation, but not the report’s evaluation details, methodology, or supporting arguments. ## Gartner Recognition - Datadog is positioned as a Leader in Gartner’s Magic Quadrant for Observability Platforms. - The linked resource appears to provide the full Gartner report or announcement. - No specific Gartner strengths, cautions, rankings, or comparison with other vendors are included in the provided text. ## Datadog’s Observability Portfolio The page navigation highlights Datadog’s broad platform, including: - Infrastructure monitoring, metrics, containers, Kubernetes, networks, serverless, and cloud costs - APM, service monitoring, profiling, and dynamic instrumentation - Database, data-stream, job, and quality monitoring - Log management, observability pipelines, and sensitive-data scanning - Real-user monitoring, session replay, synthetic monitoring, and error tracking - CI visibility, testing, code coverage, and software delivery tools - Incident response, service catalogs, SLOs, workflow automation, and event management - AI capabilities such as agent observability, Bits AI, GPU monitoring, and MCP integrations Overall, the material presents Datadog as a broad, integrated observability platform, but the actual Gartner analysis is not included. For a detailed assessment, consult the linked Gartner report directly.

Kubernetes Datadog Observability Application Performance Monitoring+2

datadogNov 2, 20203 min readCurated summary

Introducing Glommio, a thread-per-core crate for Rust and Linux

Thread-per-core architecture can significantly improve performance and reduce cloud costs by avoiding lock contention and expensive context switches. However, adopting it directly can reduce developer productivity because it requires new programming patterns and careful data ownership. Datadog developed Glommio, a Rust framework intended to make thread-per-core applications easier to build and maintain. ## Why Traditional Threading Has Limits - Applications commonly use multiple threads to perform independent tasks in parallel. - Shared data requires locks, which introduce contention and waiting. - Thread context switches can cost around five microseconds—potentially more than modern storage I/O operations using technologies such as `io_uring`. - Asynchronous programming reduces blocking, but many runtimes still rely on thread pools or separate worker threads for operations such as file I/O. ## How Thread-per-Core Works - Each CPU core runs a single application thread, often pinned to that core. - Because the operating system does not move the thread between cores, ordinary thread context switches are eliminated. - Hardware interrupts and auxiliary tasks can still interrupt execution. - For maximum performance, operators may reserve certain CPUs for interrupts and system services rather than application work. ## Sharding Data Across Cores - Thread-per-core applications depend on sharding: each thread owns a distinct subset of the data or requests. - Examples include assigning Kafka partitions or database key ranges to individual threads. - Requests assigned to one thread execute there to completion unless the code explicitly yields. - This ownership model prevents multiple threads from handling the same request or data simultaneously. ## Eliminating Locks - Since one thread processes a shard at a time, operations on that shard are naturally serialized. - A conventional threaded cache requires locks because multiple threads may update the same data concurrently. - Sharding reduces contention by dividing a large cache into smaller sections, but locks may still be needed if the operating system switches between threads. - With thread-per-core, updates to keys in the same shard occur sequentially, so an update can complete without acquiring a lock. ## Glommio and Existing Precedents - Thread-per-core is not a new concept; the author previously worked with Seastar, a C++ framework used by ScyllaDB. - Datadog’s Glommio brings the model to Rust while aiming to make its programming challenges more manageable. - The framework is motivated by the need to preserve developer productivity while achieving the efficiency gains of thread-per-core systems. Thread-per-core is most suitable for highly parallel, high-throughput workloads with naturally shardable data. Its performance benefits depend on disciplined data ownership and cooperative execution, while frameworks such as Glommio can reduce the complexity of adopting the model.

Database Design Datadog Rust Sharding+4

datadogNov 2, 20202 min readCurated summary

Introducing Glommio, a thread-per-core crate for Rust and Linux | Datadog

Datadog has been recognized as a Leader in the 2026 Gartner® Magic Quadrant™ for Observability Platforms. The announcement positions Datadog as a broad observability provider spanning infrastructure, applications, logs, security, digital experience, software delivery, service management, and AI. The supplied content contains mostly site navigation rather than the article’s supporting details or Gartner’s evaluation rationale. ## Datadog’s Observability Scope - Infrastructure monitoring, metrics, containers, Kubernetes, networks, serverless systems, cloud costs, GPUs, and storage. - Application performance monitoring, service monitoring, profiling, dynamic instrumentation, and agent observability. - Log management, sensitive-data scanning, audit trails, and observability pipelines. - Database, data-streams, data-quality, and jobs monitoring. ## Broader Platform Capabilities - Security features including cloud security, SIEM, workload protection, code security, vulnerability management, and compliance. - Digital-experience tools such as real-user monitoring, session replay, synthetic monitoring, error tracking, and product analytics. - Software-delivery capabilities covering CI visibility, test optimization, continuous testing, code coverage, and feature flags. - Service-management tools for incidents, events, SLOs, workflows, case management, and software catalogs. - AI offerings including Bits AI agents, investigation tools, agent observability, GPU monitoring, and MCP integrations. Overall, the announcement emphasizes Datadog’s unified and expansive observability platform. A complete assessment of Gartner’s specific strengths, cautions, and evaluation criteria would require the full blog post or linked Gartner report.

Kubernetes Datadog Observability Application Performance Monitoring+3

datadogOct 7, 20202 min readCurated summary

How we wrote a Python profiler | Datadog

The post explains how Datadog built a low-overhead statistical profiler for Python. Rather than tracing every function call, the profiler periodically samples running threads and reconstructs their Python and native call stacks. The main challenge is collecting accurate stack data without pausing applications for too long or introducing unsafe behavior inside CPython. ### Why Traditional Profiling Is Expensive - Deterministic profilers instrument every function call and return. - This provides detailed data but can significantly slow production workloads. - A statistical profiler reduces overhead by sampling execution at regular intervals instead of observing every event. ### Sampling Python Threads - The profiler interrupts running threads to capture their current execution state. - Python’s signal-handling model complicates this because signals are generally processed by the main thread. - The implementation must coordinate native threads, operating-system signals, and the Python interpreter to sample worker threads reliably. - Sampling must avoid interfering with application locks or triggering unsafe operations in signal handlers. ### Reconstructing Call Stacks - A useful profile needs both Python-level frames and native stack information. - The profiler walks Python frames to identify functions, files, and line numbers. - It also handles time spent in native extensions and the interpreter itself. - Collected samples are aggregated into call stacks, allowing Datadog to show CPU usage and hotspots across the application. ### Balancing Accuracy and Overhead - Sampling frequency affects the trade-off between detail and runtime cost. - More frequent samples improve visibility into short-lived work but consume more resources. - The profiler is designed to operate continuously in production, so it prioritizes low overhead, safe memory handling, and resilience across Python versions and deployment environments. The central recommendation is to use statistical sampling for always-on production profiling. It provides actionable performance data with far less impact than call-by-call instrumentation, provided the implementation carefully accounts for CPython’s threading, signal, and native-extension behavior.

Kubernetes Datadog Observability Application Performance Monitoring+3

datadogOct 7, 20203 min readCurated summary

How we wrote a Python profiler

Datadog built a Python continuous profiler because Python lacked Java-style, always-on production profiling tools. The post argues that deterministic profilers such as `cProfile` impose too much overhead for continuous use, while statistical profiling can provide representative performance data with minimal disruption. Datadog’s profiler addresses this through modular collectors, recording, scheduling, and data export. ## Profiling Versus Tracing - **Profiling** measures resource consumption such as CPU time and memory allocation to reveal performance problems. - **Tracing** records individual operations—such as SQL queries or HTTP requests—within a request timeline. - Tracing explains request latency, but profiling provides deeper insight into code-level execution and operating-system resource usage. ## Limitations of Deterministic Python Profilers - Python’s `cProfile`, available since CPython 2.5, records every function call and the time spent in each call. - It can provide a complete execution flow, but its usefulness depends heavily on code structure: - A program built around a few large functions produces little actionable detail. - A program containing thousands of functions can incur two- or three-times runtime overhead. - This overhead makes deterministic profiling unsuitable for always-on production environments. ## Why Profile in Production? - Optimizing without profiling is essentially guessing; real workloads often differ from development environments. - Production systems vary from developer machines in hardware, concurrency, input data, and workload behavior. - Continuous profiling captures how an application actually consumes resources under authentic conditions. - These requirements lead to statistical rather than deterministic profiling. ## Statistical Profiling in Python - Statistical profilers sample program activity periodically instead of recording every function call. - Individual short-lived calls may be missed, but repeated sampling over hours produces a reliable picture of resource consumption. - Lower overhead allows the application to run closer to its normal, unprofiled behavior. - Datadog evaluated numerous open-source Python profilers but found limitations involving platform support, collected data, or presentation-focused designs. - The team therefore developed its own statistical profiler, incorporating ideas from the tools it studied. ## Datadog Python Profiler Design The profiler was designed around three constraints: - Keep runtime overhead as low as possible. - Make deployment simple. - Support common operating systems and environments. Its architecture, inspired by the JDK Flight Recorder, consists of: - **Collectors:** Gather data such as CPU usage and memory allocation. - **Recorder:** Stores events produced by collectors. - **Exporter:** Sends profiling data outside the application. - **Scheduler:** Invokes components at appropriate intervals, such as exporting data every 60 seconds. - **Profiler:** Provides the high-level interface used by applications. ## Stack Collection - The stack collector is the primary built-in collector. - It wakes 100 times per second and captures the execution stack of every Python thread. - For each thread, it gathers information including: - The currently executing function - CPU time consumed - Exceptions being handled - The collector monitors the time required to inspect the application so it can control and limit its own CPU overhead. A statistical profiler with low overhead is the appropriate foundation for continuous production profiling, giving teams evidence about real application behavior without substantially changing that behavior.

Datadog Python Java Profiling+4

datadogSep 23, 20201 min readCurated summary

The Old Datadog and the Sea | Datadog

The provided content does not include the tech blog post itself. It mainly contains Datadog’s navigation menu and a promotional link announcing its Gartner Magic Quadrant leadership, so there is not enough article content to summarize accurately. Please provide the blog post’s full text or the relevant sections.

Kubernetes Datadog Observability Application Performance Monitoring+3

datadogSep 23, 20203 min readCurated summary

The Old Datadog and the Sea

Wouter de Bie describes upgrading his 1992 Hunter Legend sailboat, *Second Wind*, into a connected, data-driven vessel. His goals were to unify onboard instruments, improve safety with vessel tracking, add accurate wind measurements, and collect data for later analysis. The central solution was an NMEA 2000 network, which allowed compatible devices to share data and power over a single backbone. ## Building a Unified Instrument Network - The boat originally had a depth sounder, speedometer, autopilot, chartplotter, and GPS, but the systems were not centrally connected. - The existing Raymarine equipment used SeatalkNG, which is compatible with NMEA 2000 despite having different cables and connectors. - The older speed and depth displays used NMEA 0183 rather than NMEA 2000. - Instead of adding protocol converters, Wouter replaced the older displays with new NMEA 2000-compatible instruments. - Once connected to the backbone, speed and depth data appeared on the chartplotter. ## Why NMEA 2000 Was Chosen - NMEA 2000 is similar to the CAN bus used in automobiles. - Devices connect to a shared backbone and broadcast small data messages for other equipment to consume. - The network operates at 250 kbit/s, which is sufficient for marine instruments. - Its cables carry both power and data, reducing wiring complexity. - Most marine electronics vendors support the standard. ## Improving Safety with AIS - Wouter added an Automatic Identification System (AIS) transponder to detect nearby vessels. - AIS broadcasts a vessel’s name, identifier, speed, heading, and coordinates over VHF radio. - The system only detects AIS-equipped vessels within line of sight, but offers a lower-cost alternative to radar. - After installing the transponder and routing its antenna cable to the stern, nearby ships appeared on the chartplotter. - The transponder’s built-in GPS also supplied positioning data to the NMEA 2000 network. ## Adding Wind Measurements - The original wind indicator was only an analog vane, requiring Wouter to look up at the mast and providing no wind-speed measurement. - He purchased a NMEA 2000 wind transducer and cockpit display. - The transducer measures wind angle with a vane and wind speed with a rotor, typically from the top of the mast. - Installation required routing a cable through the mast and boat. - A friend was hoisted up the mast on a calm evening to drill the mounting hole and install the sensor. The project demonstrates how replacing incompatible legacy instruments and connecting modern devices through NMEA 2000 can turn a boat into an integrated safety and analytics platform. Once all sensor data is centralized, it can support both better sailing decisions and longer-term analysis of performance and conditions.

Database Design Data Analysis Wireless Communication Nmea 2000+4

datadogSep 23, 20193 min readCurated summary

Computing accurate percentiles with DDSketch

DDSketch was created to compute accurate percentiles from massive, distributed monitoring datasets without storing every value. The post explains why averages and exact percentile calculations are impractical for latency data, and why existing quantile sketches produced too much noise—especially at high percentiles. DDSketch addresses these needs through compact, approximate, and mergeable data structures. ## Why Percentiles Matter for Monitoring - Application latency strongly affects user experience, sales, and revenue. - Averages can hide extreme values and fail to represent users with the worst experiences. - High percentiles, such as the 99th percentile, provide a more useful view of typical “bad” experiences while ignoring extreme outliers. ## Why Percentile Computation Is Difficult - Minima, maxima, sums, and counts can be calculated using constant memory. - Exact percentiles require retaining every value, sorting the collection, and selecting the value at the desired rank. - This approach is infeasible for streams containing millions of monitoring points. - Percentiles are also difficult to aggregate across distributed systems: - Partition-level maxima, minima, sums, and counts can be merged exactly. - Partition-level percentiles cannot determine the global percentile. - Sending all raw values across the network would eliminate the benefits of distributed aggregation. ## Quantile Sketches - Sketches compress data into smaller structures that preserve enough information to answer approximate queries. - They trade some accuracy for significantly lower memory usage. - Quantile sketches estimate percentiles without retaining every input value. - Mergeable sketches allow multiple independently generated sketches to be combined without introducing additional precision loss. - Mergeability supports distributed computation and pre-aggregation, such as processing data at individual hosts before sending it elsewhere. - Datadog initially used the Greenwald-Khanna (GK) sketch for percentile metrics and histogram generation. ## Accuracy Problems with Existing Sketches - Percentile graphs often contain noise and spikes, particularly at high percentiles. - Some variation reflects real changes in latency caused by load or network conditions. - GK approximation errors can add substantial artificial variation. - Comparisons with exact percentiles show that GK-generated errors can create spikes that do not exist in the underlying data. - Errors appear especially pronounced for the 99th percentile, reducing the reliability and usefulness of monitoring visualizations. ## Rank-Error Versus Relative-Error Guarantees - Sketch algorithms generally expose a parameter controlling the accuracy–memory trade-off. - Higher accuracy requires a larger sketch. - The post identifies the need for percentile approximations whose errors are appropriate for noisy, large-scale monitoring data, motivating the design of DDSketch. DDSketch is presented as a purpose-built solution for monitoring systems: it aims to provide accurate percentile estimates with compact storage and support for lossless merging across distributed data sources.

Data Aggregation Dds Sketch Quantile Sketch Distributed Computation+4

datadogSep 23, 20194 min readCurated summary

Computing accurate percentiles with DDSketch | Datadog

Datadog’s post explains why accurately computing percentiles is difficult when monitoring large-scale, distributed systems. Traditional approaches either require retaining every observation or sacrifice accuracy through fixed-size summaries, especially for long-tailed data such as request latency. DDSketch addresses this by providing mergeable percentile estimates with a guaranteed relative-error bound and memory usage that remains effectively constant. ## Why Percentiles Matter - Averages can hide slow requests and do not describe the tail of a distribution. - Percentiles such as p95, p99, and p99.9 are more useful for measuring latency and reliability. - Monitoring systems must calculate these values from enormous numbers of observations across many hosts and services. - Storing every measurement is too expensive, while calculating percentiles independently on each machine and averaging the results is mathematically incorrect. ## Limitations of Common Approaches - Exact percentile calculation requires sorting or retaining all values, which is impractical for high-volume metrics. - Histograms use predefined buckets, making their accuracy dependent on bucket boundaries. - Fixed-width buckets are inefficient for distributions spanning several orders of magnitude: - Small values may require fine-grained buckets. - Large values may require a huge number of buckets. - Many quantile sketches optimize for rank accuracy, but a small rank error can still produce a large value error in heavy-tailed distributions. - Summaries must also be mergeable so that data collected from multiple agents can be combined without losing their accuracy guarantees. ## DDSketch’s Logarithmic Mapping - DDSketch groups values into logarithmically spaced bins rather than equally sized intervals. - Values close together near zero receive finer absolute resolution, while larger values receive wider buckets. - Each value is mapped to a key based on its logarithm: - Positive and negative values are handled separately. - Zero and values sufficiently close to zero use a dedicated zero bucket. - A representative value is chosen for each bucket, typically using the bucket’s geometric center. - Because adjacent buckets have a fixed ratio, the estimated value is bounded by a predictable relative error rather than a fixed absolute error. ## Relative-Error Guarantees - DDSketch is configured with a target relative accuracy, such as 1%. - Its logarithmic base is selected so that the returned quantile is within that relative-error bound of the true value. - Relative error is particularly appropriate for latency data: - An error of a few milliseconds matters greatly for a 10 ms request. - The same absolute error is much less significant for a 10-second request. - The sketch preserves accuracy across a wide range of values without requiring a proportional increase in the number of buckets. ## Distributed Aggregation and Memory Use - DDSketches can be merged by adding the bucket counts from separate sketches. - This allows agents, hosts, containers, and regional services to aggregate measurements into a global percentile. - Merging does not require access to the original observations. - The sketch stores counts rather than individual values, substantially reducing memory and network costs. - Datadog also describes bounded-memory variants that collapse older or less significant bins when necessary, allowing sketches to maintain a fixed storage limit while retaining useful tail information. ## Practical Trade-offs - Higher accuracy requires more buckets and therefore more memory. - Lower accuracy reduces resource usage but produces wider estimates. - The choice of relative accuracy should reflect the metric’s operational needs rather than defaulting to the smallest possible error. - Implementations must account for negative values, zeros, very small values, and values outside the normal range. - Accurate percentile reporting depends not only on the sketch algorithm but also on correct aggregation and consistent configuration across producers. DDSketch is therefore a practical choice for observability systems that need scalable, mergeable, and predictable percentile calculations. Its logarithmic buckets and relative-error guarantees make it especially well suited to latency and other long-tailed measurements where fixed-width histograms or rank-based approximations can be misleading.

Kubernetes Datadog Observability Application Performance Monitoring+3

datadogJun 3, 20193 min readCurated summary

Secure publication of Datadog Agent integrations with TUF and in-toto

Datadog built a compromise-resilient CI/CD system to publish Agent integrations independently of full Agent releases. It combines in-toto for end-to-end supply-chain verification with TUF for secure key and metadata distribution. Together, these technologies ensure that users install only integrations derived from developer-approved source code, even if parts of the infrastructure are compromised. ## Independent Integration Publishing - Agent integrations were traditionally bundled into full Agent releases. - This delayed important integration updates and prevented users from trying new integrations immediately. - Datadog wanted automation to build and publish integrations on demand without fully trusting the automation itself. ## End-to-End Verification with in-toto - TLS and GPG package signatures help prevent man-in-the-middle attacks but do not protect against compromised build or publishing infrastructure. - in-toto defines the software supply chain as a fixed sequence of signed steps. - Each step records the inputs it received and the outputs it produced, allowing the Agent to verify that only authorized parties performed the required work. - The integration pipeline includes: - Developers signing Python and YAML source files. - CI/CD packaging the source into Python wheels without modifying existing wheels. - A signing step applying TUF signatures to the wheels. - The Datadog Agent verifying that the downloaded wheel matches the developer-signed source. ## Secure Distribution with TUF - in-toto does not itself provide a secure way to distribute, revoke, or replace verification keys. - TUF supplies signed, compromise-resilient metadata for: - The root of trust for wheels and supply-chain metadata. - The in-toto-defined workflow. - Public keys used to verify the workflow. - TUF protects against tampering, rollback attacks, and indefinite replay of outdated metadata. - Offline trust bootstrapping and protected developer keys are essential to the overall security model. ## Hardware-Protected Developer Signing - Developers use Yubikeys to generate and store GPG signing keys. - Private keys cannot be exported from the device, assuming correct firmware. - Signing requires both a secret PIN and physical interaction with the Yubikey. - A command-line tool integrates in-toto and GPG, preserving a convenient developer workflow while reducing key-compromise risk. ## Transparent Verification for Users - The Datadog Agent automatically invokes TUF and in-toto when downloading or updating integrations. - Users need no workflow changes under normal conditions. - If metadata, signatures, or supply-chain steps fail verification, installation is blocked and the Agent reports the failure. Datadog’s approach demonstrates that secure automated publishing requires layered controls: in-toto verifies how software was produced, while TUF securely manages the trust and distribution mechanisms needed to validate it.

Database Design Datadog CI/CD Python+4

datadogJun 3, 20193 min readCurated summary

Secure publication of Datadog Agent integrations with TUF and in-toto | Datadog

Datadog describes how it secures the publication and distribution of Datadog Agent integrations using The Update Framework (TUF) and in-toto. TUF protects clients from tampered, outdated, or incorrectly signed packages, while in-toto provides verifiable evidence about how each integration was built. Together, the systems create a chain of trust from source and build processes to the integration installed by an Agent. ## Why Agent integrations need stronger supply-chain security - Integrations are distributed software components that run inside the Datadog Agent. - A compromise of the source repository, build infrastructure, signing credentials, or distribution system could lead to malicious code reaching customers. - Authenticating only the final package is insufficient if attackers can: - Replace repository metadata - Replay an older vulnerable release - Roll back clients to compromised versions - Exploit a stolen signing key - Publish artifacts that were not produced by the approved build process ## TUF for secure package distribution - TUF separates repository responsibilities across cryptographic roles rather than relying on one signing key. - Metadata roles such as root, targets, snapshot, and timestamp help clients verify: - Which keys are trusted - Which integrations and versions are authorized - Whether metadata is current - Whether files have been modified - TUF protects against common repository attacks, including: - Key compromise through key rotation and delegation - Rollback attacks using version information - Freeze attacks using metadata expiration - Mix-and-match attacks involving inconsistent metadata - The Datadog Agent can therefore reject integrations that fail authenticity, integrity, freshness, or version checks. ## in-toto for build provenance - in-toto complements TUF by describing and verifying the steps used to produce an integration. - Build metadata can show that required steps—such as source retrieval, dependency installation, testing, packaging, and signing—were performed by authorized parties. - Attestations connect each build step to its inputs and outputs, making unauthorized substitutions easier to detect. - This ensures that a validly signed package was not merely signed, but was produced through the expected supply-chain process. ## Combining distribution security and provenance - TUF answers whether an integration is authorized and safe to download. - in-toto answers whether it was built according to the approved process. - The combined design creates layered verification: - TUF validates repository metadata and package integrity. - in-toto validates build identity, steps, and provenance. - The Agent enforces the resulting trust decisions before installation or update. - This approach limits the impact of a compromise in any single part of the publication pipeline. Datadog’s approach illustrates that software supply-chain security requires more than package signatures. Using TUF for resilient distribution metadata and in-toto for build provenance provides a stronger, defense-in-depth model for safely delivering Agent integrations.

Software Supply Chain Security Datadog Agent Tuf In Toto+1

datadogApr 2, 20193 min readCurated summary

Building highly reliable data pipelines at Datadog

Datadog’s approach to reliable data pipelines focuses on delivering correct data on time, even when individual jobs fail. Reliability therefore requires fault tolerance, monitoring, and fast recovery rather than eliminating every failure. The company achieves this through isolated, short-lived clusters and pipelines designed to limit the impact of failures. ## Reliability Means Timely, Correct Results - A reliable pipeline is one that consistently produces correct outputs within the required time window. - Occasional crashes do not necessarily make a pipeline unreliable if automatic recovery still delivers the data on schedule. - Pipelines should be designed with the expectation that failures will eventually occur. - Monitoring must detect unexpected failures early, while operational processes should support rapid recovery. ## Architecture for Batch Pipelines - Datadog streams and analyzes live data in real time but uses batch pipelines for features such as optimized long-term storage. - Historical data is stored in object storage. - Cloud Hadoop/Spark services launch and configure processing clusters. - Luigi workers manage tasks and workflows, while Spark workers compile code and submit jobs. - Jobs can be launched through a web interface, command line, or scheduler. ## One Cluster per Pipeline Instead of placing all workloads on one large Hadoop cluster, Datadog gives each pipeline its own cluster. - **Isolation:** Jobs do not compete for resources or interfere with one another, simplifying monitoring and diagnosis. - **Workload-specific hardware:** Clusters can use CPU-optimized or memory-optimized instances depending on the job. - **Elastic scaling:** Clusters can be expanded to catch up with delays or handle growing data volumes without waiting for a shared cluster. - **Safer upgrades:** Hadoop and Spark versions can be upgraded gradually across separate clusters. - Clusters are typically short-lived, averaging about three hours, although dozens may run simultaneously. ## Using Spot Instances to Encourage Fault Tolerance - AWS spot instances can reduce infrastructure costs by as much as 80%, but their nodes may be terminated whenever capacity or demand changes. - Rather than avoiding this failure mode, Datadog designs pipelines to tolerate disappearing clusters. - Long-running jobs are risky because failures discard more work and make recovery slower. - Pipelines are split into smaller jobs: - **Vertically:** Separate transformations into multiple stages, persisting intermediate results in S3. - **Horizontally:** Partition input data so multiple jobs process different portions concurrently. ## Breaking Up the Rollup Pipeline - Datadog’s rollup pipeline generates aggregated time-series data for historical metrics queries. - A single job would take more than 14 hours, making failures costly and difficult to recover from. - The pipeline is divided into two stages: - Aggregate high-resolution data and checkpoint it to S3 as Parquet files. - Convert the intermediate data into a custom format optimized for queries. - As these jobs grew, they were partitioned further using Kafka’s partitioning scheme. - Kafka partitions are grouped into shards, allowing Datadog to: - Adjust how much data each job processes. - Run more or fewer jobs as needed. - Isolate unusually large or sensitive shards. - This decomposition adds overhead because launching jobs and checkpointing to S3 take extra time, but it substantially limits the work lost during failures. ## Practical Recommendation Design pipelines around failure rather than assuming uninterrupted execution. Use isolated, scalable clusters, short jobs, intermediate checkpoints, and partitioned processing so that failures affect only a small portion of the workload and recovery remains fast.

Database Design PostgreSQL AWS Monitoring+4

datadogApr 2, 20191 min readCurated summary

Building highly reliable data pipelines at Datadog | Datadog

The provided text does not include the blog post’s actual article content. It contains Datadog’s navigation menu and a link titled “Highly Reliable Data Pipelines,” so the post’s argument, architecture, and technical conclusions cannot be summarized reliably. ## Available Information - The page appears to be a Datadog engineering blog post about building highly reliable data pipelines. - The surrounding content is primarily Datadog product navigation. - It also promotes Datadog’s recognition as a Leader in the 2026 Gartner® Magic Quadrant™ for Observability Platforms. Please provide the article body or a complete page extract for a detailed technical summary.

Kubernetes Datadog Observability Application Performance Monitoring+3

datadogJan 22, 20193 min readCurated summary

Rethinking UX for AI-driven alerting

Alerting UX is shifting from manually configured static thresholds toward statistical systems that understand trends, seasonality, and group behavior. Forecasting, anomaly detection, and outlier detection reduce maintenance and nuisance alerts, while algorithmic feeds can discover problems users never explicitly configured. The post argues that supervised feeds—trained by user feedback—could become the most significant evolution in monitoring. ## Traditional Alerting UX Most alerts are built from four dimensions: - **Scope:** The hosts, containers, services, or other targets being monitored. - **Metric:** The value tracked, such as free disk space. - **Thresholds:** Static warning or critical values that trigger alerts. - **Time:** A duration or time window during which the condition must occur. Static threshold alerts remain common, with many taking the form: “If free disk space equals zero, alert.” ## Problems with Static Thresholds - Static thresholds do not adapt to system growth, changing workloads, or temporary events such as holiday traffic. - They require regular review and maintenance to remain useful. - Warning thresholds often serve as manual “heads-ups” so engineers can inspect a graph and estimate whether intervention is necessary. - Large numbers of warning alerts create false positives and alert fatigue. - Monitoring systems must be explicitly told which scopes and metrics to watch, leading to duplicated configuration and ongoing maintenance. ## Algorithmic Alerting Statistical alerting introduces three primary methods: - **Forecasting** - Uses historical data to predict when a metric will cross a threshold. - Changes “alert when disk reaches zero” into “alert if disk will reach zero within 24 hours.” - Lets teams specify how much remediation time they need. - Can eliminate separate warning thresholds because the forecast provides advance notice. - **Anomaly detection** - Predicts what should be happening now based on historical behavior. - Considers configurable confidence intervals and seasonality, such as daily or weekly patterns. - Alerts when current behavior deviates significantly from the expected range. - **Outlier detection** - Compares members of a group that should behave similarly. - Flags an individual server or service whose behavior differs from its peers. - Does not depend on historical behavior. These methods make thresholds and time behavior more flexible, but they still require users to define the metrics and scopes in advance. ## Algorithmic Feeds Algorithmic feeds apply similar statistical techniques without requiring detailed alert configuration. - They can monitor systems without predefined individual scopes or metrics. - They are especially useful for unpredictable anomalies and outliers. - Examples include Slack Highlights and Datadog Watchdog. - Feeds shift monitoring from **opt-in alerting**—where users specify what to watch—to discovering noteworthy activity automatically. - The post presents this shift as potentially the largest change in alerting UX, while noting that algorithmic feeds are still immature. ## Supervised Algorithmic Feeds Once a monitoring system generates a stream of events, user feedback can help train it to surface more relevant information. - The model is compared to social media feeds, where actions such as “likes” guide future recommendations. - This suggests a future in which engineers can teach monitoring systems which anomalies and events matter to them. - The provided excerpt ends while introducing this concept, so it does not describe the specific feedback mechanisms or implementation details. Monitoring is likely moving toward adaptive systems that combine statistical detection with user-guided prioritization. Teams should use forecasting and anomaly-based alerts where appropriate, while treating algorithmic feeds as a complementary way to discover issues outside manually configured monitoring.

Database Design Datadog Cloud Infrastructure Anomaly Detection+4

datadogJan 22, 20192 min readCurated summary

Rethinking UX for AI-driven alerting | Datadog

Datadog’s page announces that the company was named a Leader in the 2026 Gartner Magic Quadrant for Observability Platforms. The supplied content, however, primarily contains site navigation rather than the referenced blog post, so it does not provide details about the article’s argument concerning AI-driven alerting. ## Gartner Recognition - Datadog highlights its recognition as a Leader in Gartner’s Magic Quadrant for Observability Platforms. - The announcement is presented as a promotional resource linked from the Datadog website. ## Datadog’s Product Portfolio The navigation emphasizes Datadog’s broad observability and security platform, including: - **Infrastructure:** infrastructure, container, network, serverless, GPU, storage, and cloud-cost monitoring. - **Applications:** APM, service monitoring, profiling, dynamic instrumentation, and agent observability. - **Data and logs:** database, data-stream, job, quality, log, sensitive-data, and pipeline monitoring. - **Security:** code, cloud, vulnerability, workload, application, API, and SIEM security tools. - **Digital experience:** browser and mobile RUM, session replay, synthetic monitoring, product analytics, and error tracking. - **Software delivery:** CI visibility, test optimization, code coverage, feature flags, and developer portals. - **Service management:** incident response, SLOs, event management, workflows, and case management. - **AI:** Bits AI agents, investigations, chat, security analysis, agent observability, and MCP integrations. The provided text does not include enough of the actual “Rethinking UX for AI-Driven Alerting” article to summarize its technical concepts or conclusions.

AI Agents Kubernetes Datadog Observability+4