Datadog

110 posts

datadog3 min readCurated summary

Secure (and usable) multi-AWS account IAM setup | Datadog

The post presents a defense-in-depth approach to securing AWS accounts with IAM. Its central recommendation is to minimize long-lived credentials and broad permissions by combining strong authentication, role-based access, least privilege, and continuous auditing. Secure account governance is treated as an ongoing operational process rather than a one-time configuration. ## Protect the Root User - Use the root user only for tasks that cannot be performed through IAM. - Enable multi-factor authentication (MFA), preferably with a hardware security key. - Avoid creating root access keys. - Store root credentials securely and monitor for any root-user activity. ## Use Federated, Role-Based Access - Prefer AWS IAM Identity Center or an external identity provider for human access. - Grant users access through groups and roles instead of individual permissions. - Use short-lived role credentials rather than permanent IAM user access keys. - Require separate roles for administrative, development, production, and read-only work. ## Apply Least Privilege - Start with narrowly scoped permissions and expand them only when necessary. - Restrict actions by resource, account, region, and relevant condition keys. - Avoid wildcard permissions such as `Action: "*"` and `Resource: "*"`. - Use IAM Access Analyzer and CloudTrail activity to identify unused or excessive permissions. - Add permission boundaries or organization-level Service Control Policies when teams need guardrails around delegated administration. ## Secure Workloads and Automation - Assign IAM roles directly to EC2 instances, Lambda functions, containers, and other workloads. - Do not embed access keys in source code, configuration files, or deployment artifacts. - Store unavoidable secrets in services such as AWS Secrets Manager or Systems Manager Parameter Store. - Rotate and revoke credentials promptly when they are exposed or no longer required. ## Monitor and Audit IAM - Enable CloudTrail across accounts and regions, with logs protected from modification. - Alert on suspicious activity, including root-user use, policy changes, disabled logging, and unusual access-key behavior. - Regularly review users, groups, roles, policies, and unused credentials. - Use AWS Config, Security Hub, or equivalent controls to check compliance with account-security requirements. ## Centralize Governance - Manage multiple AWS accounts through AWS Organizations. - Keep production and sensitive workloads isolated from development accounts. - Apply Service Control Policies to prevent high-risk actions, even for administrators. - Establish a controlled emergency or “break-glass” access process with strong monitoring. The practical recommendation is to combine MFA, centralized identity, temporary role credentials, narrowly scoped permissions, and continuous auditing. No individual IAM setting is sufficient on its own; security comes from layering preventive controls with detection and response.

Read original(opens in new tab)
datadog1 min readCurated summary

Robust statistical distances for machine learning | Datadog

The supplied text does not include the blog post itself; it is largely Datadog’s navigation menu. The only identifiable article is **“Robust Statistical Distances for Machine Learning,”** so a detailed, source-grounded summary is not possible without the article body. ## Article Focus - The post appears to address statistical distances used to compare probability distributions in machine-learning systems. - Its focus is likely making these comparisons more **robust to outliers, noisy observations, and distribution shifts**. - Such distances can support tasks including anomaly detection, model monitoring, data-drift detection, and evaluating generated data. ## Why Robustness Matters - Conventional distance measures may be disproportionately influenced by extreme values. - Outliers can make two otherwise similar datasets appear substantially different. - A robust distance should distinguish meaningful distribution changes from isolated or corrupted observations. ## Practical Implication The article’s central recommendation is presumably to choose statistical-distance methods based not only on mathematical properties, but also on their resistance to noise and outliers. Please provide the actual article text for a complete, section-by-section summary with the specific techniques and conclusions.

Read original(opens in new tab)
datadog1 min readCurated summary

Scaling support with Vagrant and Terraform | Datadog

The provided content does not include the blog post itself. It consists primarily of Datadog’s navigation menu and a promotional banner announcing its Gartner recognition, while the linked page suggests an article about scaling support with Vagrant and Terraform. ### Visible Content - Datadog announces that it was named a **Leader in the Gartner Magic Quadrant for Observability Platforms**. - The page navigation lists products across: - Infrastructure and application monitoring - Logs, security, and digital experience - CI/CD, service management, and AI - The URL references an engineering post titled **“Scaling Support With Vagrant and Terraform,”** but no article text is present. A meaningful technical summary requires the article body or a complete extract of the post.

Read original(opens in new tab)
datadog2 min readCurated summary

Improving cloud security visibility with ChatOps | Datadog

Datadog announces that it was named a Leader in the 2026 Gartner® Magic Quadrant™ for Observability Platforms. The provided content does not include the blog post’s body or Gartner’s evaluation details; it mainly contains Datadog’s navigation menu and product links. ## Announcement - Datadog highlights its recognition as a Leader in Gartner’s 2026 Observability Platforms Magic Quadrant. - The linked resource appears to be a Gartner-related announcement rather than a technical deep dive. ## Datadog’s Product Scope The navigation reflects a broad observability and operations platform covering: - Infrastructure monitoring, metrics, containers, Kubernetes, networks, serverless systems, and cloud costs - Application performance monitoring, profiling, dynamic instrumentation, and agent observability - Database, data-stream, jobs, and quality monitoring - Log management, sensitive-data scanning, audit trails, and observability pipelines - Security capabilities including cloud security, SIEM, workload protection, code security, and vulnerability management - Digital experience tools such as real-user monitoring, session replay, synthetic monitoring, and error tracking - Software delivery, CI visibility, testing, feature flags, and code coverage - Incident response, service catalogs, SLOs, workflow automation, and case management - AI agents, GPU monitoring, AI integrations, and investigation tools The supplied excerpt does not provide enough information to summarize Gartner’s criteria, Datadog’s strengths or weaknesses, or the report’s comparative findings.

Read original(opens in new tab)
datadog3 min readCurated summary

Piecewise regression: When one line simply isn’t enough | Datadog

Piecewise regression offers a practical way to model time series whose trends change over time, something a single straight-line regression cannot represent well. The technique divides data into segments and fits a separate regression line to each one, allowing systems such as observability platforms to detect trend shifts more accurately. Its usefulness depends on selecting meaningful breakpoints without overfitting noise. ## Why a Single Regression Line Falls Short - Ordinary linear regression assumes one constant relationship between time and the measured value. - Real-world operational metrics often contain: - Sudden changes in growth rate - Traffic or usage shifts - Deployment-related behavior changes - Periods of increase followed by stabilization or decline - A single line averages these different behaviors, producing inaccurate trend estimates and potentially misleading forecasts. ## How Piecewise Regression Works - The time series is divided into multiple regions by one or more breakpoints. - Each region receives its own regression equation, such as: - Before the breakpoint: one intercept and slope - After the breakpoint: a different intercept and slope - The fitting process searches for the breakpoint that minimizes the combined prediction error across all segments. - Models may require the lines to connect at the breakpoint, preventing unrealistic discontinuities, or allow independent segments when abrupt jumps are meaningful. ## Finding Useful Breakpoints - Candidate breakpoints are evaluated by comparing the residual error produced by different segmentations. - A breakpoint is valuable when it significantly improves the fit rather than merely explaining random fluctuations. - More segments can capture complex behavior, but they also increase the risk of overfitting. - Practical implementations therefore need safeguards such as minimum segment sizes, limits on the number of breakpoints, and validation against noisy data. ## Applications in Observability - Piecewise models can improve the interpretation of infrastructure and application metrics. - They are particularly useful for identifying: - Changes in request volume - Altered resource-consumption patterns - Performance regressions - Long-term growth phases - Recovery or stabilization after an incident - By distinguishing genuine trend changes from normal variation, the method can support better anomaly detection and forecasting. ## Limitations and Tradeoffs - Noisy or sparse data can make breakpoint selection unstable. - A model with too many segments may describe historical noise instead of general behavior. - Sudden outliers can distort regression parameters unless they are handled separately. - Piecewise regression captures trend changes, but it does not automatically explain their causes; engineers still need deployment, traffic, and infrastructure context. Piecewise regression is therefore best treated as a lightweight, interpretable tool for detecting changes in metric behavior. It provides more realistic trend modeling than a single regression line while remaining simpler and easier to operate than highly complex forecasting models.

Read original(opens in new tab)
datadog3 min readCurated summary

Hackathon project: Viewing Datadog metrics in Minecraft

Datadog engineers used a two-day hackathon to display real-time Datadog metrics inside Minecraft. They connected Minecraft’s Python API with Datadog’s metrics API, then built configurable, live-updating graphs and monitor indicators in the game world. The project demonstrated that even an unconventional visualization environment can be practical to prototype with familiar tools. ## Controlling Minecraft with Python - The team used Raspberry Juice and a Minecraft Pi Edition server to expose Minecraft controls. - They ran the setup on laptops for better performance and faster development. - The `py3minepi` library enabled Python code to create, remove, and query blocks. - Creating a block required only a connection to the server and a call such as `mc.setBlock(...)`. ## Retrieving Datadog Metrics - The Datadog Python library provided access to the Metrics API. - The prototype authenticated with an API key and application key. - It queried recent data, such as average system CPU idle time over the previous five minutes. - The Minecraft and Datadog components were then combined so metric values could be rendered as blocks and structures. - Monitor status indicators changed between green and red depending on whether an alert was active. ## YAML-Based Dashboard Configuration - The team moved dashboard definitions out of Python code into YAML files. - Configuration specified: - Graph position, size, and orientation - Visual properties such as colors, transparency, and borders - Datadog queries and time ranges - Monitor IDs and display locations - This allowed complete dashboards containing multiple graphs and monitor indicators to be updated in real time. ## Handling Minecraft’s Persistence - Minecraft blocks remain in the world after being created, while metric graphs change constantly. - Early experiments left behind random cubes that made the world difficult to navigate. - The team implemented “vacuum” functions to remove everything generated by the visualization code before redrawing it. ## Rendering and Performance Challenges - Without browser technologies such as JavaScript and CSS, graphs had to be reduced to rows of data and represented with Minecraft blocks. - Large graphs could overwhelm the data pipeline. - Caching was added to reduce bandwidth usage and avoid repeatedly requesting the same data. - The performance concerns mirrored Datadog’s everyday engineering work, where caching and efficient data handling are essential. ## Hackathon Experience - The first four hours focused on configuring the environment and connecting the systems. - The remaining time was spent experimenting with building, viewing, destroying, and rebuilding metric displays. - The project’s main value was creative exploration rather than production monitoring. The prototype shows how quickly APIs can be combined to create unusual monitoring interfaces. While Minecraft is not intended to replace conventional dashboards, the project is a playful demonstration of real-time data visualization and rapid experimentation.

Read original(opens in new tab)
datadog1 min readCurated summary

Hackathon project: Viewing Datadog metrics in Minecraft | Datadog

The provided content does not include the blog post itself. It contains Datadog’s navigation menu and a link titled “Hackathon Project: Viewing Datadog Metrics in Minecraft,” but no article text describing the project, implementation, or conclusions. ## Available Information - The linked post appears to cover a hackathon project involving: - Datadog metrics - Minecraft - A likely visualization or integration between the two - The surrounding content is primarily Datadog’s product navigation, covering: - Infrastructure and application monitoring - Logs, security, and digital experience - Software delivery and service management - AI products and platform capabilities ## Conclusion The article body is missing from the supplied material, so a reliable technical summary cannot be produced. Please provide the post’s full text or relevant excerpt.

Read original(opens in new tab)
datadog2 min readCurated summary

Protobuf parsing in Python

Protocol Buffers provides a compact, efficient binary format for structured data, making it suitable for APIs and inter-machine communication. The post introduces Protobuf through a Python metrics example and explains how to serialize and deserialize messages. It also shows how to stream multiple messages by prefixing each with its length, since Protobuf messages are not inherently self-delimiting. ## Protocol Buffers Basics - A `.proto` file defines the structure of a message. - The example `Metric` message contains: - A name - A type - A floating-point value - Repeated string tags - The `protoc` compiler generates language-specific code, such as Python’s `metric_pb2.py`. - Python can serialize a message with `SerializeToString()` and restore it with `ParseFromString()`. ## Streaming Multiple Messages - A single Protobuf message can be parsed directly, but consecutive messages need delimiters. - Protobuf does not automatically indicate where one message ends and the next begins. - The recommended approach is to prepend each serialized message with its byte length. - The length is encoded as a Varint, which uses fewer bytes for smaller integers. - This mirrors Java’s `writeDelimitedTo` and `parseDelimitedFrom` behavior and is also how the kube-state-metrics API chains messages. ## Varints and Python Implementation - Python’s Protobuf library does not provide public convenience methods for delimited messages. - The implementation uses internal helpers: - `_VarintBytes` to encode message lengths - `_DecodeVarint32` to read them - Serialization writes the length followed by the message bytes. - Deserialization reads the length, extracts the corresponding byte range, and parses it as a `Metric`. - The example loads the entire stream into memory, though a production implementation could process data incrementally. For APIs that exchange sequences of structured records, length-prefixed Protobuf messages offer an efficient and interoperable alternative to plain-text formats. Teams should account for message framing explicitly and use generated code plus appropriate streaming logic when handling multiple messages.

Read original(opens in new tab)
datadog2 min readCurated summary

Being a solutions engineer at Datadog

Solutions Engineers at Datadog combine customer support, technical investigation, and product improvement. They troubleshoot issues across configuration, dashboards, metrics, and alerting while collaborating with customers and engineers. The role also offers structured opportunities to build engineering skills and shape a career toward engineering, product management, or sales engineering. ## Role and Responsibilities - Serve as the technical point of contact for Datadog customers. - Handle tickets involving: - Agent and integration configuration - Dashboard and metric visualization - Alerting issues and bugs - Investigate customer needs, inspect source code, fix bugs when necessary, and coordinate with engineering teams. - Provide short-term workarounds and identify potential product enhancements. - Communicate with customers through chat and technical calls. ## Learning and Collaboration - New Solutions Engineers rapidly develop skills in technologies such as Linux, Ruby, and SQL. - The role requires multitasking while continuously learning new technical concepts. - Team members share knowledge through Datadog documentation and collaborative projects. - Customer feedback provides direct insight into how the product could evolve. ## Engineering Experience and Career Growth - During two-week “embedding” sprints, Solutions Engineers join engineering teams and work on projects alongside developers. - Embedding provides hands-on experience with: - Datadog’s internal technologies - Engineering and design challenges - Architectural decisions - Cross-team collaboration - Once fully trained, employees can pursue side projects such as improving demo environments or automating internal processes. - Career paths can be tailored toward engineering, product management, or sales engineering. Datadog presents the Solutions Engineer role as a strong fit for people who enjoy customer interaction, technical problem-solving, collaboration, and continuous learning. The company was hiring for these positions in New York, Paris, and remote locations.

Read original(opens in new tab)
datadog2 min readCurated summary

Being a solutions engineer at Datadog | Datadog

Datadog’s page announces that the company has been named a Leader in the 2026 Gartner® Magic Quadrant™ for Observability Platforms. The provided content primarily consists of navigation links rather than the blog article itself, so it does not include Gartner’s evaluation details or Datadog’s specific strengths. It does, however, show the broad range of products Datadog positions as part of its observability platform. ## Gartner Recognition - Datadog highlights its designation as a Leader in Gartner’s 2026 Magic Quadrant for Observability Platforms. - No supporting analysis, ranking details, or Gartner assessment criteria are included in the provided text. ## Datadog’s Platform Scope - **Infrastructure:** Infrastructure and container monitoring, metrics, Kubernetes autoscaling, network monitoring, serverless, cloud cost, GPU, and storage management. - **Applications and data:** Application Performance Monitoring, profiling, dynamic instrumentation, database monitoring, data-stream monitoring, and data quality. - **Logs and security:** Log management, observability pipelines, sensitive-data scanning, cloud security, SIEM, workload protection, and code security. - **Digital experience:** Browser and mobile RUM, session replay, synthetic monitoring, product analytics, experiments, and error tracking. - **Software delivery and service management:** CI visibility, test optimization, code coverage, feature flags, incident response, SLOs, workflow automation, and internal developer portals. - **AI capabilities:** Agent observability, AI integrations, GPU monitoring, Bits AI agents, investigation tools, and an MCP server. Datadog’s positioning is therefore centered on providing a unified platform spanning infrastructure, applications, logs, security, user experience, software delivery, and AI. A complete assessment would require the missing body of the article or Gartner report details.

Read original(opens in new tab)
datadog2 min readCurated summary

The trouble with mounting

Datadog found that some agents stopped reporting all metrics because they became stuck in an unkillable state during disk checks. The root cause was `os.statvfs`, whose glibc implementation can hang while inspecting NFS mounts configured with hard-mount behavior. Since agents run in unpredictable customer environments, Datadog isolated the call in a separate thread and allowed the main process to continue after a timeout. ## Detecting the Hang - Customers reported gaps across every metric, indicating that the agent—not an individual check—had stopped functioning. - Logs showed the agent sometimes hung without producing an error. - A watchdog failed to terminate it because the process was stuck in an unkillable system call. - Developer-mode timing data identified `os.statvfs` as the consistently slow operation. ## How NFS Causes Unkillable Processes - `os.statvfs` calls the Linux `statvfs` function through CPython and glibc. - `statvfs` can hang when examining a remote directory mounted through NFS. - NFS hard mounts retry indefinitely and do not time out system calls. - Soft mounts eventually return an error, while the `intr` option allows interruption of the calling process. - Hard mounts may be appropriate when reads and writes must eventually succeed, but they are risky with unreliable NFS connections because they are the default in many configurations. ## The `/proc/mounts` Complication - Glibc’s `statvfs` implementation checks each directory listed in `/proc/mounts` until it finds the requested mount. - Consequently, a disconnected NFS mount can block `statvfs` even when the agent is checking a different filesystem. - This made changing NFS mount options impractical as a universal fix because Datadog cannot control customers’ system configurations. ## Datadog’s Workaround - The agent now runs `statvfs` on a separate thread. - If the call exceeds a timeout, the main agent thread continues operating. - This approach avoids total metric loss across heterogeneous environments. - The trade-off is a modest increase in memory usage on systems with hard-mounted NFS volumes. The practical lesson is to treat filesystem statistics as potentially blocking operations, especially in environments with NFS. Isolating such calls behind timeouts provides more reliable monitoring than assuming system calls will always return promptly.

Read original(opens in new tab)
datadog2 min readCurated summary

The trouble with mounting | Datadog

The post explains why filesystem mounting becomes surprisingly complex in containerized Linux environments. Datadog’s Agent needs to inspect host filesystems from inside a container, but mount namespaces, bind mounts, and propagation rules can make the host’s view incomplete or inconsistent. The conclusion is that reliable mounting requires understanding namespace boundaries and deliberately configuring mount propagation rather than treating mounts as ordinary directory mappings. ## Linux Mount Namespaces - Each process can have its own mount namespace, isolating its view of mounted filesystems. - A container therefore sees a filesystem tree that may differ substantially from the host’s tree. - Bind-mounting a directory into a container does not necessarily expose mounts created beneath that directory. - This is particularly problematic for paths such as `/proc`, `/sys`, `/var/lib/docker`, and other locations containing nested mounts. ## The Problem with Bind Mounts - A bind mount initially exposes only the directory tree visible at the time it is created. - Later mounts beneath the source directory may not appear in the container. - Recursive bind mounts can copy nested mounts, but they introduce their own behavior and compatibility concerns. - Mounts can also be shared, private, or “slave,” determining whether mount and unmount events propagate between namespaces. ## Mount Propagation - Linux mount propagation controls how changes in one namespace are reflected in another. - Shared mounts propagate events in both directions, while slave mounts receive changes without sending them back. - Private mounts isolate changes completely. - Choosing the wrong propagation mode can cause the Agent to miss newly mounted filesystems or, worse, allow container-side changes to affect the host. ## Datadog’s Engineering Challenge - Datadog needs host-level visibility while keeping the monitoring container isolated and safe. - The Agent must account for mounts that appear after startup, including dynamically created container and volume mounts. - Correct behavior depends on both the container runtime configuration and the host’s existing mount topology. - A robust implementation must inspect mount metadata and handle namespace and propagation details explicitly. The practical recommendation is to treat mounting as a namespace and event-propagation problem, not merely a path-sharing mechanism. Systems that need host visibility should use carefully selected recursive mounts and propagation modes, validate the resulting mount tree, and test behavior when mounts are created or removed dynamically.

Read original(opens in new tab)
datadog2 min readCurated summary

Engineering spotlight: Marie-Laure Bardonnet

Datadog’s Notebooks feature demonstrates that interns can make substantial contributions when given meaningful ownership and effective mentorship. Marie-Laure Bardonnet progressed from small bug fixes to prototyping the feature, gaining hands-on experience with React, Redux, and Redux Saga. Her experience ultimately led to continued part-time work and a full-time role at Datadog. ## From Bug Fixes to Feature Ownership - Marie-Laure initially handled small issues, such as fixing a dashboard favorite-star interaction. - Gradually more complex tasks helped her learn Datadog’s codebase and application architecture. - Rather than limiting her to routine maintenance, Datadog assigned her a major project earlier than expected. ## Building Datadog Notebooks - Notebooks let users save graphs from a specific point in time alongside text and other contextual information. - The feature is designed to preserve and share organizational knowledge, helping teams respond more quickly. - Marie-Laure built most of the prototype during her seven-month internship. - The project introduced her to: - React for the frontend - Redux for state management - Redux Saga for handling side effects ## The Role of Mentorship - Team lead Ivan DiLernia gave Marie-Laure substantial autonomy while remaining available for difficult architectural decisions. - He encouraged her to investigate ideas independently, then collaborated with her when problems required deeper discussion. - Marie-Laure identified this balance between independence and guidance as one of the most valuable parts of the internship. ## Lasting Impact - The internship changed how Marie-Laure viewed her academic coursework, helping her distinguish practical engineering skills from more theoretical material. - After returning to France, she continued working part-time on Notebooks and other web-platform projects. - She later completed her studies and accepted a full-time position at Datadog. Datadog’s experience suggests that internships are most effective when they combine gradual onboarding, meaningful technical ownership, and thoughtful mentorship rather than restricting interns to low-impact tasks.

Read original(opens in new tab)
datadog1 min readCurated summary

Engineering spotlight: Marie-Laure Bardonnet | Datadog

The provided text contains Datadog’s navigation and promotional banner announcing its recognition as a Leader in the 2026 Gartner® Magic Quadrant™ for Observability Platforms. It does not include the actual blog post about Marie-Laure Bardonnet, so its argument, technical details, and conclusion cannot be reliably summarized. ## Available Content - Datadog promotes its observability platform and Gartner recognition. - The navigation lists products covering: - Infrastructure and application monitoring - Logs, databases, and data observability - Security and cloud protection - Digital experience monitoring - CI/CD and software delivery - Incident and service management - AI-powered observability and investigation - The page URL suggests the article is an “Engineering Spotlight” featuring Marie-Laure Bardonnet. Please provide the article body or a complete extraction of the page for a substantive summary.

Read original(opens in new tab)
datadog2 min readCurated summary

Redux-Doghouse: Creating reusable React-Redux components through scoping | Datadog

Redux Doghouse presents a way to make React/Redux components genuinely reusable by giving each component instance its own scope. The approach addresses collisions in Redux action types, state, and selectors when the same component appears multiple times. Its conclusion is that scoped Redux logic can preserve the benefits of a centralized store while allowing components to behave like isolated, reusable units. ## The Reuse Problem in React and Redux - Redux state is global by default, while reusable components often need instance-specific state. - Reusing a connected component can cause: - Action types from one instance affecting another - Selectors reading the wrong state - Reducers sharing or overwriting unrelated data - Boilerplate for manually generating unique identifiers - Components therefore become tightly coupled to the shape and location of the application’s Redux store. ## Scoping Redux State - Redux Doghouse introduces scopes that associate actions, reducers, and selectors with a particular component instance. - Each instance receives an isolated section of state, even when multiple copies of the same component are rendered. - Actions are scoped so dispatching an event in one component does not unintentionally update another instance. - Selectors resolve data within the current scope rather than relying on hard-coded global paths. ## Reusable React Components - Components can package their Redux behavior alongside their React UI. - The parent application supplies or creates a scope when mounting the component. - The same component can then be embedded in different parts of an application without duplicating Redux wiring. - This approach keeps implementation details—state shape, action handling, and selectors—inside the reusable component. ## Benefits and Trade-offs - Scoping reduces naming conflicts and makes component behavior easier to reason about. - It supports modular development by separating local component state from application-wide state. - Shared global data can still remain in ordinary Redux state where appropriate. - Developers must manage scope identity and lifecycle carefully, particularly when components are dynamically mounted or removed. Redux Doghouse is best viewed as an architectural pattern for combining Redux’s centralized data flow with component-level isolation. Teams building reusable or multiply-instantiated React components can use scoped state and behavior to avoid collisions without abandoning Redux.

Read original(opens in new tab)