Datadog

190 posts

www.datadoghq.com/blog/engineering

Filter by tag

datadog3 min readCurated summary

Redux-Doghouse: Creating reusable React-Redux components through scoping

Redux-Doghouse is a Redux library for creating scoped actions and reducers, allowing reusable React/Redux components to coexist without responding to one another’s actions. It preserves Redux’s ability to coordinate state across an application while ensuring that component-local actions affect only the instance that dispatched them. Datadog developed it to support reusable Query Editors within larger editors such as dashboards and Expression Editors. ## The Problem with Reusable Redux Components - Redux reducers respond to actions based on their `TYPE`. - If multiple instances of the same component share a Redux store, an action such as `MY_ACTION` can update every instance. - This is useful for application-wide events, but incorrect when an action should affect only one component instance. - Refactoring each component to use unique action types would undermine its reusability and independence. ## Scoped Actions and Reducers - Redux-Doghouse adds a unique scope to each component instance’s actions. - Reducers are wrapped so that a scoped action is routed only to the matching component instance. - A component can therefore continue using generic action types such as `MY_ACTION` while remaining isolated from sibling instances. - Higher-level components can still observe and respond to those actions, preserving Redux’s cross-component coordination. - The parent can extend a child component’s behavior without requiring the child to know about its parent. ## Datadog’s Query Editor Use Case - Datadog’s dashboards contain Query Editor components for editing individual metrics. - Query Editors were rebuilt as miniature React/Redux applications so they could be reused in: - Dashboard graph editors - Monitor editors - Notebook editors - Other application contexts - The Expression Editor needed to render an arbitrary number of Query Editors and combine their queries with expressions such as `a + b / c`. ## Coordinating Child Editors The Expression Editor needed to: - Validate that expressions reference existing query labels, rejecting inputs such as `a + d` when query `d` does not exist. - Enforce compatible `group by` values across queries: - Queries may share a value such as `host`. - Some queries may have no grouping. - Non-empty groupings such as `host` and `device` cannot be mixed. - Ensure that a `SET_GROUP` action from Query Editor A affects only A, not Query Editor B. - Allow the Expression Editor itself to observe `SET_GROUP` and enforce rules across all queries. - Keep Query Editors independent so they remain usable outside an Expression Editor. ## How Doghouse Solves It - The parent assigns each Query Editor a scope, such as `A`, `B`, or `C`. - Actions dispatched by each editor receive metadata identifying that scope. - The parent wraps each editor’s reducers and routes actions only to the reducer with the matching scope. - The Expression Editor can still listen to the same actions at a higher level and apply cross-editor validation or coordination. Redux-Doghouse is most useful when reusable Redux components need isolated local behavior while still participating in a shared application state. It lets teams organize actions and reducers by component, rather than forcing all Redux logic to be structured around entire views.

Read original(opens in new tab)
datadog2 min readCurated summary

Redux-Doghouse: Creating reusable React-Redux components through scoping | Datadog

Redux Doghouse presents a way to make React/Redux components genuinely reusable by giving each component instance its own scope. The approach addresses collisions in Redux action types, state, and selectors when the same component appears multiple times. Its conclusion is that scoped Redux logic can preserve the benefits of a centralized store while allowing components to behave like isolated, reusable units. ## The Reuse Problem in React and Redux - Redux state is global by default, while reusable components often need instance-specific state. - Reusing a connected component can cause: - Action types from one instance affecting another - Selectors reading the wrong state - Reducers sharing or overwriting unrelated data - Boilerplate for manually generating unique identifiers - Components therefore become tightly coupled to the shape and location of the application’s Redux store. ## Scoping Redux State - Redux Doghouse introduces scopes that associate actions, reducers, and selectors with a particular component instance. - Each instance receives an isolated section of state, even when multiple copies of the same component are rendered. - Actions are scoped so dispatching an event in one component does not unintentionally update another instance. - Selectors resolve data within the current scope rather than relying on hard-coded global paths. ## Reusable React Components - Components can package their Redux behavior alongside their React UI. - The parent application supplies or creates a scope when mounting the component. - The same component can then be embedded in different parts of an application without duplicating Redux wiring. - This approach keeps implementation details—state shape, action handling, and selectors—inside the reusable component. ## Benefits and Trade-offs - Scoping reduces naming conflicts and makes component behavior easier to reason about. - It supports modular development by separating local component state from application-wide state. - Shared global data can still remain in ordinary Redux state where appropriate. - Developers must manage scope identity and lifecycle carefully, particularly when components are dynamically mounted or removed. Redux Doghouse is best viewed as an architectural pattern for combining Redux’s centralized data flow with component-level isolation. Teams building reusable or multiply-instantiated React components can use scoped state and behavior to avoid collisions without abandoning Redux.

Read original(opens in new tab)
datadog2 min readCurated summary

Cheering on coworkers: Building culture with Datadog dashboards

Christian’s colleagues built a Datadog dashboard to remotely track his progress in a six-day, 850 km ultramarathon. They scraped live race data from the event website, converted it into Datadog metrics, and visualized his distance, ranking, and elapsed time alongside video and other dashboard elements. At publication, Christian was leading by more than 47 km with 44 hours remaining. ## Extracting Race Data - The event website regularly published runners’ statistics and race progress in plain HTML. - A Python crawler using `Requests` retrieved the webpage. - `BeautifulSoup` parsed the HTML to extract: - Current ranking - Total distance run - Elapsed time - Other race information ## Sending Metrics to Datadog - The team used the Datadog Python client and StatsD to emit metrics through the Datadog Agent. - For each runner, the script sent gauge metrics for: - `runner.distance` - `runner.ranking` - `runner.elapsed_time` - Metrics were tagged with each runner’s name, enabling individual tracking and comparisons. ## Building the Dashboard - The collected metrics were combined into a Datadog dashboard. - The dashboard included: - Live race statistics - A live video feed - Animated GIFs for entertainment - Visualizations of meaningful progress metrics - Screens displaying the dashboard were placed in the company’s New York and Paris offices so colleagues could follow and encourage Christian throughout the race. The project demonstrates how a lightweight web scraper, StatsD metrics, and a monitoring dashboard can turn publicly available data into a live, engaging team experience.

Read original(opens in new tab)
datadogOriginal article

Cheering on coworkers: Building culture with Datadog dashboards | Datadog (opens in new tab)

Datadog engineers developed a real-time tracking dashboard to monitor a colleague’s progress during an 850km, six-day ultra-marathon challenge. By scraping public race statistics and piping the data into their monitoring platform, the team created a centralized visualization tool to provide remote support and office-wide engagement. ### Data Extraction and Parsing The team needed to harvest race data that was only available as plain HTML on the event’s official website. * A crawler was built using the Python `Requests` library to automate the retrieval of the webpage's source code. * The team utilized `BeautifulSoup` to parse the HTML and isolate specific data points, such as the runner's current ranking and total distance covered. ### Ingesting Metrics with StatsD Once the data was structured, it was converted into telemetry using the Datadog agent and the `statsd` Python library. * The script utilized `dog.gauge` to emit three primary metrics: `runner.distance`, `runner.ranking`, and `runner.elapsed_time`. * Each metric was assigned a "name" tag corresponding to the runner, allowing the team to filter data and compare participants within the Datadog interface. * The data was updated periodically to ensure the dashboard reflected the most current race standings. ### Dashboard Visualization and Results The final phase involved synthesizing the metrics into a high-visibility dashboard displayed in the company’s New York and Paris offices. * The dashboard combined technical performance graphs with multimedia elements, including live video feeds and GIFs, to create an interactive cheering station. * The system successfully tracked the athlete's 47km lead in real-time, providing the team with immediate updates on his physical progress and elapsed time over the 144-hour event. This project demonstrates how standard observability tools can be repurposed for creative "life-graphing" applications. By combining simple web scraping with metric ingestion, engineers can quickly build custom monitoring solutions for any public data source.

datadog3 min readCurated summary

Restroom hacks

Datadog built an office bathroom-availability monitor to reduce contention without compromising privacy or existing door functionality. Raspberry Pi 2 devices, GPIO-connected sensors, and simple Unix tools provided a low-maintenance way to report whether bathrooms were occupied. The project showed that the hardest parts were adapting to varied real-world hardware, mounting sensors cleanly, and dealing with unreliable Wi-Fi—not writing software. ## Project Goals - Avoid intrusive monitoring: - No cameras or sensors that could feel invasive. - Provide reliable occupancy information with minimal false positives and negatives. - Use door-lock status where possible as the occupancy signal. - Avoid interfering with existing locks and doors. - Keep devices secure, professional-looking, easy to maintain, and remotely updateable. - Treat the project as a fun hardware experiment. ## Adapting to Different Bathrooms - Bathrooms differed significantly in: - Lock styles, including push-button handles and rotary stall locks. - Number of rooms or stalls. - Availability and location of power outlets. - Wi-Fi quality, especially near concrete walls and older electrical equipment. - These variations required different sensor designs rather than one universal installation. ## Raspberry Pi and Sensor Hardware - Raspberry Pi 2 Model Bs served as the project’s controllers because they: - Ran Linux. - Supported Wi-Fi and SSH administration. - Were compact enough to conceal. - The team used several sensor types: - Magnetic reed switches for detecting door position. - Pin switches for detecting sliding stall-lock positions. - Photoresistors were purchased as a possible way to detect darkness but were not needed in the MVP. - For push-button locks, reed switches detected whether the door was open or closed. Although this could theoretically misreport a closed but unoccupied bathroom, it worked reliably in practice. - Stall-lock sensors were hidden inside hollow metal panels. Automotive-style pin switches were mounted using simple carved wooden blocks that contacted the sliding lock without obstructing it. - Wiring was concealed in wiremolding, with Raspberry Pis placed inside outlet boxes where possible. ## GPIO and Unix-Based Monitoring - Raspberry Pi GPIO pins were accessed through files in `/sys/class/gpio/`. - A Python script read sensor values and translated them into bathroom availability. - Configuration handled differences between normally open and normally closed sensors. - The service was exposed through `tcpserver` and managed with `daemontools`. - A basic command-line client could query status with Netcat, for example: ```sh nc 11.bathrooms.datadog-internal.com 50 ``` ## Making Availability Easy to Use - Employees could check status from the command line. - Some added bathroom availability to TextBar. - Datadog dashboards displayed bathroom status throughout the New York office. - The implementation required very little code; most effort went into sensor selection, physical installation, and network troubleshooting. The project demonstrates that inexpensive, hackable hardware combined with simple Linux tools can solve a practical office problem. For similar systems, prioritize non-intrusive sensors, flexible installation designs, and secure remote management; the resulting software can remain remarkably small.

Read original(opens in new tab)
datadog1 min readCurated summary

Restroom hacks | Datadog

Datadog announces that Gartner named it a Leader in the 2026 Magic Quadrant for Observability Platforms. The supplied content primarily consists of Datadog’s navigation menu, so it does not provide the report’s evaluation criteria, supporting evidence, or detailed conclusions. ## Datadog’s Gartner Recognition - Datadog is presented as a Leader in Gartner’s Magic Quadrant for Observability Platforms. - The announcement links to a Datadog resource about the 2026 report. - No specific Gartner strengths, cautions, ranking details, or competitor comparisons are included in the provided text. ## Datadog’s Product Scope The navigation indicates that Datadog offers capabilities across: - Infrastructure and cloud monitoring - Application performance monitoring and profiling - Logs, metrics, databases, and data observability - Security monitoring and vulnerability management - Digital experience monitoring, synthetic testing, and session replay - CI/CD and software delivery observability - Incident response, service management, and workflow automation - AI-agent observability and AI-assisted investigation The material provided is insufficient for a deeper summary of the blog post’s argument or supporting technical details.

Read original(opens in new tab)
datadog1 min readCurated summary

Consul at Datadog | Datadog

The provided content does not include the blog post itself. It contains a Datadog promotional banner announcing its recognition as a Leader in Gartner’s Magic Quadrant for Observability Platforms, followed by the website’s navigation menu. ## Datadog’s Observability Offering - Infrastructure monitoring, metrics, containers, Kubernetes, networks, serverless systems, and cloud costs - Application performance monitoring, profiling, dynamic instrumentation, and agent observability - Logs, database monitoring, data pipelines, and data quality tools - Security, digital experience monitoring, software delivery, incident management, and AI features ## Promotional Announcement - Datadog promotes its recognition as a Leader in the 2026 Gartner Magic Quadrant for Observability Platforms. - The provided excerpt does not include Gartner’s evaluation, Datadog’s supporting arguments, or the article’s technical discussion. The actual blog text is needed for a meaningful technical summary.

Read original(opens in new tab)
datadog3 min readCurated summary

Consul at Datadog

Consul has become a critical part of Datadog’s production infrastructure for distributing configuration and discovering services. After 18 months of use, the main lesson is that Consul requires careful capacity planning, controlled access, efficient query patterns, and continuous monitoring. The recommendations aim to keep clusters stable while supporting frequent configuration updates and high-volume service discovery. ## Consul Server Capacity and CPU Consul servers use Raft consensus to elect a leader and coordinate the cluster. - Followers trigger a leadership transition if they cannot hear from the leader for 500 milliseconds. - Frequent leadership transitions usually indicate insufficient CPU capacity. - Datadog’s approximate sizing guidance: - `m3.large`: about 300 agent nodes - `c3.xlarge`: about 500 agent nodes - `c3.2xlarge`: about 800 agent nodes - If transitions occur hourly or more often, increase server CPU capacity until they happen no more than daily. - Standard monitoring may miss brief 500-millisecond CPU spikes, so reducing CPU pressure is important even when dashboards look normal. ## Auditable Configuration Changes Consul’s key-value store is useful for distributing configuration throughout a cluster. - Configuration can be retrieved through HTTP or delivered through Consul watches. - Direct edits without an audit trail make it difficult to determine who changed a value and when. - `git2consul` distributes configuration from a Git repository, providing version control and accountability. - Datadog uses it for cluster-wide configuration updates roughly every 60 seconds, dozens of times per day. ## Access Control with ACLs Consul ACLs prevent unauthorized processes from modifying or deleting key-value data. - Tokens should be limited to the data and operations each process requires. - Scoped permissions reduce the impact of accidental changes. - ACLs provide an important safety boundary between services and configuration areas. ## Watches Instead of Excessive Polling Consul can handle substantial traffic, but it should not be queried hundreds of thousands of times per second like Redis or Memcached. - Watches notify clients when key-value data changes. - This reduces unnecessary polling and distributes updates efficiently. - Watches can sometimes trigger unexpectedly or too frequently. - Tools such as `sifter` can help protect systems from excessive watch activity. ## Using dnsmasq for Service Discovery Applications using Consul’s DNS interface can reduce load by placing `dnsmasq` between clients and Consul. - Use short DNS TTLs; Datadog commonly uses 10 seconds. - Query `dnsmasq` rather than Consul directly so repeated answers can be cached locally. - At very high request volumes, cache Consul services in an additional hosts file loaded by `dnsmasq`. - This setup served over 100,000 DNS requests per second while sending only about 400 requests per second directly to Consul. - `goshe` can collect `dnsmasq` statistics for monitoring. ## Monitoring Cluster Health Monitoring is essential for operating Consul reliably. - `consul.consul.leader.reconcile.count` should remain stable and indicate that a leader exists. - `consul.serf.events.consul_new_leader` shows leadership transitions; frequent events suggest instability. - `consul.raft.leader.lastContact` measures how recently nodes contacted the leader. - `consul.consul.dns.domain_query.count` reveals how many DNS requests are reaching Consul directly. - Also monitor CPU and network usage on Consul servers. Consul works best when server nodes have sufficient CPU, configuration changes are managed through version control, ACLs restrict access, watches replace aggressive polling, and `dnsmasq` absorbs service-discovery traffic. Continuous monitoring of leadership, Raft connectivity, DNS load, CPU, and networking helps identify failures before they affect production.

Read original(opens in new tab)
datadog2 min readCurated summary

Releasing czlib and zstd Go bindings | Datadog

Datadog announces that it has been named a Leader in the 2026 Gartner® Magic Quadrant™ for Observability Platforms. The surrounding content presents Datadog as a broad platform spanning infrastructure, applications, data, logs, security, digital experience, software delivery, service management, and AI. The provided excerpt does not include Gartner’s detailed evaluation or the reasoning behind the placement. ## Gartner Recognition - Datadog highlights its designation as a Leader in the Gartner Magic Quadrant for Observability Platforms. - The announcement links to a Gartner-related resource for additional details. ## Broad Observability Platform - **Infrastructure:** Infrastructure and container monitoring, metrics, Kubernetes autoscaling, network monitoring, serverless, cloud costs, storage, GPUs, and Cloudcraft. - **Applications:** Application Performance Monitoring, service monitoring, continuous profiling, dynamic instrumentation, and agent observability. - **Data and logs:** Database and data-stream monitoring, data quality, jobs monitoring, log management, sensitive-data scanning, audit trails, and observability pipelines. - **Security:** Code, cloud, workload, application, API, vulnerability, compliance, SIEM, and secret-scanning capabilities. - **Digital experience:** Browser and mobile RUM, product analytics, session replay, synthetic monitoring, mobile testing, and error tracking. - **Software delivery and service management:** CI visibility, test optimization, code coverage, feature flags, internal developer portals, incident response, SLOs, workflow automation, and case management. - **AI:** GPU monitoring, AI integrations, agent observability, AI-powered investigation and chat, agent builders, MCP support, and related developer tools. ## Platform Capabilities - Datadog also emphasizes shared platform features such as dashboards, notebooks, alerts, Watchdog, access control, governance, fleet automation, and mobile access. - The product organization suggests an integrated approach to monitoring systems, applications, user experiences, security, delivery pipelines, and AI workloads. Datadog’s positioning is that observability increasingly requires a unified platform rather than isolated monitoring tools. Readers seeking the specific strengths, limitations, and evaluation criteria behind the Gartner designation would need to consult the linked Gartner resource.

Read original(opens in new tab)
datadog2 min readCurated summary

Releasing czlib and zstd Go bindings

Datadog released Go bindings for two C compression libraries, czlib and zstd, to improve compression performance in production data pipelines. czlib addresses the standard library’s slower pure-Go zlib implementation, while zstd offers faster decompression, competitive compression, and modern features such as dictionaries and compression levels. The post concludes that performance depends heavily on payload size and interface, so applications should benchmark representative data. ## czlib: Faster zlib-Compatible Compression - czlib began as a fork of Vitess’s `cgzip` package. - Datadog modified it to use zlib wrapping instead of gzip headers, matching the format used by its primary data pipeline. - It provides: - Non-streaming compression and decompression - Streaming interfaces - Batch-oriented interfaces - The authors recommend benchmarking with realistic messages using: - `PAYLOAD=path_to_message go test -run=NONE -bench .` ## Benchmark Results - For a 2 KB plaintext message: - czlib compression: 44.42 MB/s - Streaming czlib compression: 34.11 MB/s - Standard `compress/zlib`: 9.27 MB/s - czlib decompression: 255.62 MB/s - Standard-library decompression: 66.72 MB/s - For a 1.7 MB message: - czlib and standard zlib had similar compression speeds, around 24 MB/s. - czlib decompression reached roughly 257 MB/s. - Standard-library decompression reached about 121 MB/s. - The results show that czlib’s advantage is especially significant for smaller messages and decompression workloads. ## zstd: A Modern Alternative - zstd, or Zstandard, was developed by Yann Collet, who also created lz4. - At the time of publication, its format had recently been finalized and version 1.0 was pending. - Compared with zlib at compression level 6, zstd: - Compresses slightly faster - Produces a slightly better compression ratio - Decompresses substantially faster - Its features include: - Streaming compression - Configurable compression levels - Precomputed dictionaries - Fixed-length batch compression similar to the lz4 interface ## Go Binding Design - The zstd binding intentionally mirrors the zlib API. - It is designed to be a functional drop-in replacement, aside from a few zstd functions that do not return errors. - Dictionary-building support is available in the upstream repositories. - The binding exposes both advanced streaming functionality and efficient batch compression. Use czlib when compatibility with zlib is required but the standard implementation is too slow; consider zstd for new systems that benefit from faster decompression and dictionary support. Always benchmark with data and interfaces representative of the target workload.

Read original(opens in new tab)