datadogOct 21, 2025

Detecting malicious pull requests at scale with LLMs | Datadog (opens in new tab)

Malicious pull requests can turn routine code review and CI workflows into supply-chain attack vectors. The post explains how attackers abuse automated builds—especially when workflows expose repository secrets or elevated GitHub permissions—and recommends treating all pull-request code as untrusted. Strong isolation, least privilege, careful workflow design, and monitoring are essential to prevent credential theft and unauthorized access. ## How Malicious Pull Requests Work - Attackers submit seemingly harmless changes that alter: - GitHub Actions workflows - Build or test scripts - Dependency configuration - Developer tooling - The malicious code executes automatically when CI runs the pull request. - Its goal may be to: - Exfiltrate repository or cloud credentials - Modify artifacts - Access internal systems - Establish persistence in the development pipeline ## Why CI Workflows Are Vulnerable - Pull-request jobs often execute attacker-controlled code through tests, package installation, or build commands. - Using privileged workflow events such as `pull_request_target` can expose secrets while checking out untrusted contributor code. - Broad `GITHUB_TOKEN` permissions increase the impact of a compromised job. - Secrets may leak through logs, environment variables, artifacts, or outbound network requests. ## Defensive Engineering Practices - Treat code from forks and external contributors as untrusted. - Avoid making secrets available to pull-request jobs. - Use minimal `GITHUB_TOKEN` permissions and separate privileged workflows from validation workflows. - Pin third-party GitHub Actions and dependencies to trusted commits or versions. - Require explicit approval before running workflows from untrusted contributors. - Isolate CI jobs with ephemeral runners, restricted network access, and limited filesystem permissions. - Review changes to workflow files with heightened scrutiny. ## Detection and Response - Monitor workflow behavior for unexpected network connections, credential access, or modified build outputs. - Audit repository and CI permissions regularly. - Use short-lived credentials and OIDC-based cloud access instead of long-lived static secrets. - Preserve workflow logs and artifacts to support investigation. - Revoke credentials immediately if a pull request or CI job is suspected of compromise. The practical recommendation is to design CI as though every pull request could be hostile: validate untrusted code in a restricted environment, keep secrets and write permissions out of those jobs, and require deliberate promotion into trusted workflows.

synthetic-monitoringk8sdatadogobservability+4

datadogOct 1, 2025

Inside Husky’s query engine: Real-time access to 100 trillion events | Datadog (opens in new tab)

The provided content does not include the blog post itself. It contains Datadog’s navigation menu and a promotional link announcing its recognition as a Leader in Gartner’s Magic Quadrant for Observability Platforms, but no substantive discussion of the linked “Husky Query Architecture” article. ## Available Content ### Datadog’s Observability Platform - Datadog promotes products covering: - Infrastructure and container monitoring - Application performance monitoring - Logs and database monitoring - Security - Digital experience monitoring - CI/CD and software delivery - Incident and service management - AI and agent observability - The navigation emphasizes Datadog’s broad, integrated platform approach. ### Gartner Recognition - The page links to Datadog’s announcement that it was named a Leader in the 2026 Gartner Magic Quadrant for Observability Platforms. - The supplied text does not include the evaluation criteria, cited strengths, limitations, or Gartner’s comparative analysis. No reliable summary of the Husky query architecture can be produced without the article’s body text.

synthetic-monitoringk8sobservabilitylog-management+4

datadogJun 23, 2025

How we built reliable log delivery to thousands of unpredictable endpoints | Datadog (opens in new tab)

Datadog’s “Reliable Log Delivery” post explains how log-collection systems can avoid losing data when networks, destinations, or agents fail. Its central recommendation is to combine acknowledgments, buffering, retries, and controlled backpressure to provide at-least-once delivery without allowing outages to overwhelm the collector. ## Why Reliable Delivery Matters - Logs are often needed during incidents, precisely when infrastructure and networks may be unstable. - Temporary destination failures can cause data loss if collectors only keep logs in memory. - Retrying without limits can create duplicate logs, unbounded memory usage, or cascading failures. ## Buffering and Persistence - Collectors should buffer logs while downstream services are unavailable. - In-memory buffers provide speed but cannot survive process crashes or host restarts. - Disk-backed queues improve durability by preserving unsent logs across transient failures. - Storage limits are necessary so a prolonged outage does not fill the host’s disk. ## Acknowledgments and Retries - A log should be removed from the queue only after the destination confirms successful receipt. - Failed or unacknowledged deliveries are retried, allowing temporary network and service failures to recover automatically. - At-least-once delivery is the practical reliability target, meaning duplicates may occur and downstream systems should handle them safely. - Retry policies should use delays and backoff rather than continuously retrying at full speed. ## Backpressure and Operational Trade-offs - When downstream systems slow down, collectors must apply backpressure instead of accepting unlimited data. - Backpressure can limit memory consumption and protect the rest of the host. - Teams must define what happens when buffers reach capacity, such as dropping the oldest data, rejecting new logs, or prioritizing important streams. - Reliability also requires monitoring queue size, delivery latency, retry rates, and dropped records. A dependable logging pipeline is not built from retries alone. It requires durable buffering, explicit delivery acknowledgments, bounded resources, and clear failure behavior; organizations should choose retention and overflow policies according to the operational value of their logs.

synthetic-monitoringk8sdatadogobservability+4

datadogJun 3, 2025

Detecting faulty deployments: Our journey from unlabeled data to supervised learning | Datadog (opens in new tab)

The supplied content does not include the blog post itself; it contains Datadog’s navigation menu and a link titled “Detecting Faulty Deployments.” As a result, there is not enough information to accurately summarize the article’s arguments, implementation details, or conclusions. ## Available context - The linked article appears to concern identifying deployments that introduce faults or regressions. - Datadog’s platform covers related capabilities such as: - Application Performance Monitoring - Metrics and infrastructure monitoring - Logs and error tracking - CI Visibility and software delivery monitoring - Service-level objectives and incident response - The page also promotes Datadog’s recognition as a Leader in the Gartner Magic Quadrant for Observability Platforms. ## Missing information - The article’s detection methodology - Metrics, queries, or deployment signals used - Alerting, rollback, or remediation procedures - Technical examples and conclusions Please provide the article text or a page extract containing the post body for an accurate summary.

synthetic-monitoringk8sdatadogobservability+4

datadogJun 28, 2024

Timeseries indexing at scale | Datadog (opens in new tab)

Datadog’s “Time Series Indexing at Scale” explains how an observability platform can index and query enormous numbers of time series without making tag-based searches prohibitively expensive. The central challenge is matching flexible combinations of metric names and tags while keeping ingestion, storage, and query latency predictable. The article presents indexing strategies and architectural trade-offs that allow Datadog to support high-cardinality telemetry at scale. ## The Challenge of Time-Series Indexing - A time series is identified not only by its metric name but also by its complete set of tags. - Modern monitoring systems may contain billions of series generated by containers, hosts, services, and dynamic infrastructure. - Queries often filter on multiple tags, requiring the system to efficiently find the intersection of several large sets of series. - Indexing must support both: - Fast writes as new series appear - Low-latency reads for interactive dashboards and alerts - High-cardinality tags make naïve database indexes expensive in both storage and query processing. ## Inverted Indexes for Tags - Datadog uses an inverted-index model that maps searchable terms—such as metric names and tag values—to the series containing them. - A query can retrieve the posting list for each term and intersect those lists rather than scanning every time series. - Common terms may correspond to very large lists, so the system must optimize how these lists are stored, compressed, and combined. - The index separates metadata used to identify series from the time-series values stored for those series. ## Distributed Indexing - Index data is partitioned across machines so that no single node must hold or process the entire dataset. - Sharding enables horizontal scaling as the number of metrics, tags, and customers grows. - Query coordination gathers results from multiple shards and combines them into a single response. - The design must balance: - Even distribution of index data - Avoidance of hot shards - Efficient fan-out during queries - Resilience when individual nodes fail ## Managing Index Growth and Cardinality - Dynamic environments continuously create and remove series, making index lifecycle management essential. - Datadog must handle churn caused by short-lived containers, deployments, and changing tag values. - Compression and compact data structures reduce the memory and storage required for posting lists. - The system distinguishes between frequently queried data and less-used data to control resource consumption. - Cardinality limits and indexing policies help prevent unusually large tag dimensions from overwhelming the system. ## Query Performance and Trade-offs - Indexing every possible attribute would improve search flexibility but increase write, storage, and maintenance costs. - The platform therefore makes trade-offs between indexing coverage, freshness, and query speed. - Query execution can combine index filtering with additional processing over the remaining candidate series. - Caching and reuse of intermediate results can reduce repeated work for common queries. - The architecture is designed to maintain predictable latency even as data volume and query complexity increase. ## Operational Considerations - Large-scale indexing requires monitoring the index itself, including shard balance, ingestion lag, memory usage, and query fan-out. - Background processes must compact, expire, and rebalance index data without disrupting active queries. - Fault tolerance is important because an index outage can affect dashboards and alerts even when the underlying metric data remains available. - Separating indexing from time-series storage allows each subsystem to scale and evolve independently. Datadog’s approach illustrates that scalable observability depends as much on metadata indexing as on storing metric values. Systems handling high-cardinality telemetry should use distributed inverted indexes, compact representations, careful lifecycle management, and explicit trade-offs between flexibility and operational cost.

synthetic-monitoringk8sobservabilitylog-management+4

datadogApr 18, 2024

How we brought Datadog's data visualization to iOS: A focus on performance | Datadog (opens in new tab)

The provided content does not include the blog post’s actual article text. It contains Datadog’s navigation menu, product links, and a reference to an engineering post about bringing Datadog data visualization to iOS performance, so its technical argument and conclusion cannot be reliably summarized. ## Available Content - Datadog promotes its recognition as a Leader in the Gartner Magic Quadrant for Observability Platforms. - The navigation lists products across: - Infrastructure and application monitoring - Logs, databases, and data observability - Security - Digital experience monitoring - Software delivery - Service management - AI and platform capabilities - The referenced engineering URL suggests a post focused on implementing Datadog data visualization for iOS performance monitoring, but no implementation details are provided. Please provide the article body or a readable extraction of the post for a substantive technical summary.

synthetic-monitoringk8sdatadogobservability+4

datadogDec 21, 2023

Scaling Self-Serve Analytics: The Tools Empowering 5,000 Employees | Datadog (opens in new tab)

The provided content does not include the blog post itself. It contains Datadog’s navigation menu and a promotional banner announcing its position as a Leader in the 2026 Gartner® Magic Quadrant™ for Observability Platforms, but no technical discussion from the article. ## Available Content - The page appears to be a Datadog engineering blog post related to a CrunchConf talk on self-serve analytics. - The supplied text primarily lists Datadog products across: - Infrastructure and application monitoring - Data and log management - Security - Digital experience - Software delivery - Service management - AI capabilities - A separate banner promotes Datadog’s recognition by Gartner. ## Missing Article Details - No article title, introduction, body sections, technical examples, architecture, or conclusions are included. - The available text is insufficient to accurately summarize the post’s arguments or implementation details. Please provide the blog post’s main text or a complete extraction of the page for a substantive summary.

synthetic-monitoringk8sdatadogobservability+4

datadogJun 30, 2023

How we migrated our acceptance tests to use Synthetic Monitoring (opens in new tab)

Datadog’s Frontend Developer Experience team migrated 565 flaky, maintenance-heavy Puppeteer acceptance tests to Synthetic Monitoring. The change replaced manually scripted browser interactions with recorded tests that could run reliably from CI through a dedicated CLI. The year-long migration improved maintainability and built engineer confidence through documentation, gradual adoption, and non-blocking rollout tooling. ## Why the Existing Acceptance Tests Were Failing - Tests ran in Node.js on a custom Puppeteer-based runner. - End-to-end tests were flaky because they depended on browsers, virtual graphics, dedicated machines, navigation, and application timing. - Even simple actions required extensive scripting: - Confirming an element existed - Checking that it was enabled - Performing the interaction - Handling compatibility with Puppeteer - Custom UI elements, such as dropdowns, made reliable automation substantially harder. - Product changes frequently required updates to both tests and the testing infrastructure. - The six CI jobs took up to 14 minutes, with total machine time reaching 35 minutes per commit. ## Synthetic Monitoring as the Replacement - The team adopted Datadog’s own Synthetic Monitoring product to record page interactions rather than manually script them. - They created `synthetics-ci`, a CLI that: - Finds files named `*.synthetics.json` - Accepts configuration overrides - Triggers Synthetic tests - Polls for result statuses - Prints human-readable output - Because the tool represented a broader pattern for using Datadog from CI/CD, it was generalized into `datadog-ci`. ## Scope of the Migration At the start of the migration in June 2021, the frontend repository had: - 300 engineers working in one repository - Approximately 90 new pull requests and 1,120 commits every day - Six acceptance-test CI jobs - 35 minutes of machine time per commit - 84 relevant files - 565 tests - About 100,000 lines of test and infrastructure code The scale of the repository and its rapid development activity meant the migration needed to be gradual and carefully coordinated. ## Building Trust and Adoption - The team wrote documentation covering: - How to write effective Synthetic tests - Which behaviors were worth testing - Which testing patterns to avoid - They demonstrated the system in company-wide and frontend-focused meetings. - Teams learned how to: - Record tests through the UI - Use scheduled tests in CI - Replace existing acceptance tests - Reduce ongoing maintenance - The team worked directly with groups that owned the largest test collections. - Jira tickets tracked the migration of every acceptance test and assigned ownership to the appropriate team. ## Gradual CI Integration - A non-blocking CI job allowed teams to introduce Synthetic tests without risking the entire pipeline. - Failures appeared as pull-request comments rather than blocking merges. - This gave engineers time to understand and trust the new system. - Once tests ran reliably, the team made the pipeline blocking. - The old acceptance-testing platform could then be retired progressively instead of being removed all at once. The migration took roughly one year and succeeded through a combination of better tooling, clear communication, incremental rollout, and shared ownership across frontend teams. For large organizations replacing a critical testing system, introducing the new workflow safely before enforcing it can make adoption far less disruptive.

synthetic-monitoringdatadogci-cdnode.js+3

datadogJun 30, 2023

How we migrated our acceptance tests to use Synthetic Monitoring | Datadog (opens in new tab)

Datadog’s Frontend Developer Experience team migrated their massive codebase from a fragile, custom Puppeteer-based acceptance testing framework to Datadog Synthetic Monitoring to address persistent flakiness and high maintenance overhead. By leveraging a record-and-play approach and integrating it into their CI/CD pipelines via the `datadog-ci` tool, they successfully reduced developer friction and improved testing reliability for over 300 engineers. This transition demonstrates how replacing manual browser scripting with specialized monitoring tools can significantly streamline high-scale frontend workflows. ### Limitations of Puppeteer-Based Testing * Custom runners built on Puppeteer suffered from inherent flakiness because they relied on a complex chain of virtual graphic engines, browser manipulation, and network stability that frequently failed unexpectedly. * Writing tests was unintuitive, requiring engineers to manually script interaction details—such as verifying if a button is present and enabled before clicking—which became exponentially more complex for custom elements like dropdowns. * The testing infrastructure was slow and expensive, with CI jobs taking up to 35 minutes of machine time per commit to cover the application's 565 tests and 100,000 lines of test code. * Maintenance was a constant burden; every product update required a corresponding manual update to the scripts, making the process as labor-intensive as writing new features. ### Adopting Synthetic Monitoring and Tooling * The team moved to Synthetic Monitoring, which allows engineers to record browser interactions directly rather than writing code, significantly lowering the barrier to entry for creating tests. * To integrate these tests into the development lifecycle, the team developed `datadog-ci`, a CLI tool designed to trigger tests and poll result statuses directly from the CI environment. * The new system uses a specific file format (`.synthetics.json`) to identify tests within the codebase, allowing for configuration overrides and human-readable output in the build logs. * This transition turned an internal need into a product improvement, as the `datadog-ci` tool was generalized to help all Datadog users execute commands from within their CI/CD scripts. ### Strategies for High-Scale Migration and Adoption * The team utilized comprehensive documentation and internal "frontend gatherings" to educate 300 engineers on how to record tests and why the new system required less maintenance. * To build developer trust, the team initially implemented the new tests as non-blocking CI jobs, surfacing failures as PR comments rather than breaking builds. * Migration was treated as a distributed effort, with 565 individual tests tracked via Jira and assigned to their respective product teams to ensure ownership and a steady pace. * By progressively sunsetting the old platform as tests were migrated, the team managed a year-long transition without disrupting the daily output of 160 authors pushing 90 new PRs every day. To successfully migrate large-scale testing infrastructures, organizations should prioritize developer trust by introducing new tools through non-blocking pipelines and providing comprehensive documentation. Transitioning from manual browser scripting to automated recording tools not only reduces technical debt but also empowers engineers to maintain high-quality codebases without the burden of managing complex testing infrastructure.

synthetic-monitoringdatadognode.jsfrontend-development+2

datadogMay 12, 2022

How Datadog's IT team automated account inactivity and SaaS spend management | Datadog (opens in new tab)

Datadog’s IT team built an automated system to identify inactive user accounts and reduce unnecessary SaaS spending. The approach replaces manual audits with data-driven workflows that detect inactivity, notify users or owners, and reclaim unused licenses while preserving access controls and accountability. ## Automating Account Inactivity Detection - The system monitors account activity across SaaS applications. - It identifies users who have not logged in or used assigned tools for a defined period. - Automated notifications give users or managers an opportunity to confirm continued business need. - Accounts can then be suspended, deprovisioned, or escalated for review. ## Managing SaaS Spend - Inactivity data is used to find unused or underused licenses. - IT can reclaim seats instead of continuing to pay for unused subscriptions. - Usage information supports more accurate renewal and purchasing decisions. - Centralized automation reduces the manual effort required to audit many applications. ## Governance and Operational Benefits - Standardized workflows make account reviews more consistent across tools. - Automated approvals and escalation paths provide visibility into decisions. - The process helps balance cost reduction with security and user access requirements. - IT teams gain a repeatable way to manage the growing complexity of SaaS environments. Organizations with substantial SaaS usage can apply the same model: centralize activity data, define inactivity policies, automate notifications and approvals, and connect the results to license reclamation and access-management workflows.

synthetic-monitoringk8sdatadogobservability+4

datadogFeb 28, 2022

Profiling improvements in Go 1.18 | Datadog (opens in new tab)

The provided text does not include the blog post itself. It contains Datadog’s navigation menu and a promotional link announcing its recognition as a Leader in Gartner’s 2026 Magic Quadrant for Observability Platforms, but no technical discussion or conclusions from the referenced article. ## Available content ### Datadog’s observability platform - Datadog promotes products for: - Infrastructure and Kubernetes monitoring - Application performance monitoring and profiling - Logs, databases, and data observability - Security and cloud protection - Real-user and synthetic monitoring - CI/CD and software delivery - Incident and service management - AI-powered investigation and automation ### Gartner recognition - The page links to Datadog’s announcement that it was named a Leader in the Gartner Magic Quadrant for Observability Platforms. - The excerpt does not provide Gartner’s evaluation criteria, Datadog’s strengths or weaknesses, or supporting evidence. Please provide the article body or a working text extract for an accurate technical summary.

synthetic-monitoringk8sdatadogobservability+4

datadogApr 30, 2021

How Datadog's IT team automated monitoring third-party accounts | Datadog (opens in new tab)

The provided text does not contain the tech blog post itself. It consists primarily of Datadog’s navigation menu and a promotional banner announcing its “Leader” ranking in the 2026 Gartner Magic Quadrant for Observability Platforms, so the article’s argument and conclusion cannot be reliably summarized. ## Promotional Announcement - Datadog links to a Gartner report about observability platforms. - The banner presents Datadog as a Leader in the report. ## Datadog Product Categories - **Infrastructure:** infrastructure, container, network, serverless, GPU, storage, and cloud-cost monitoring. - **Applications:** APM, service monitoring, profiling, dynamic instrumentation, and agent observability. - **Data and Logs:** database monitoring, data quality, job monitoring, log management, sensitive-data scanning, and observability pipelines. - **Security:** code, cloud, vulnerability, compliance, SIEM, workload, and application protection. - **Digital Experience:** browser and mobile RUM, session replay, synthetic monitoring, product analytics, and error tracking. - **Software Delivery:** CI visibility, test optimization, code coverage, feature flags, and developer portals. - **Service Management:** event management, incident response, SLOs, workflow automation, and case management. - **AI:** AI agents, GPU monitoring, integrations, investigation tools, and MCP support. The actual blog content—apparently related to how Datadog’s IT team automated monitoring of third-party accounts—is missing. A summary would require the article body or a complete excerpt.

synthetic-monitoringk8sdatadogobservability+4

datadogSep 6, 2017

Robust statistical distances for machine learning | Datadog (opens in new tab)

The supplied text does not include the blog post itself; it is largely Datadog’s navigation menu. The only identifiable article is **“Robust Statistical Distances for Machine Learning,”** so a detailed, source-grounded summary is not possible without the article body. ## Article Focus - The post appears to address statistical distances used to compare probability distributions in machine-learning systems. - Its focus is likely making these comparisons more **robust to outliers, noisy observations, and distribution shifts**. - Such distances can support tasks including anomaly detection, model monitoring, data-drift detection, and evaluating generated data. ## Why Robustness Matters - Conventional distance measures may be disproportionately influenced by extreme values. - Outliers can make two otherwise similar datasets appear substantially different. - A robust distance should distinguish meaningful distribution changes from isolated or corrupted observations. ## Practical Implication The article’s central recommendation is presumably to choose statistical-distance methods based not only on mathematical properties, but also on their resistance to noise and outliers. Please provide the actual article text for a complete, section-by-section summary with the specific techniques and conclusions.

synthetic-monitoringk8sdatadogobservability+4

datadogJul 11, 2017

Piecewise regression: When one line simply isn’t enough | Datadog (opens in new tab)

Piecewise regression offers a practical way to model time series whose trends change over time, something a single straight-line regression cannot represent well. The technique divides data into segments and fits a separate regression line to each one, allowing systems such as observability platforms to detect trend shifts more accurately. Its usefulness depends on selecting meaningful breakpoints without overfitting noise. ## Why a Single Regression Line Falls Short - Ordinary linear regression assumes one constant relationship between time and the measured value. - Real-world operational metrics often contain: - Sudden changes in growth rate - Traffic or usage shifts - Deployment-related behavior changes - Periods of increase followed by stabilization or decline - A single line averages these different behaviors, producing inaccurate trend estimates and potentially misleading forecasts. ## How Piecewise Regression Works - The time series is divided into multiple regions by one or more breakpoints. - Each region receives its own regression equation, such as: - Before the breakpoint: one intercept and slope - After the breakpoint: a different intercept and slope - The fitting process searches for the breakpoint that minimizes the combined prediction error across all segments. - Models may require the lines to connect at the breakpoint, preventing unrealistic discontinuities, or allow independent segments when abrupt jumps are meaningful. ## Finding Useful Breakpoints - Candidate breakpoints are evaluated by comparing the residual error produced by different segmentations. - A breakpoint is valuable when it significantly improves the fit rather than merely explaining random fluctuations. - More segments can capture complex behavior, but they also increase the risk of overfitting. - Practical implementations therefore need safeguards such as minimum segment sizes, limits on the number of breakpoints, and validation against noisy data. ## Applications in Observability - Piecewise models can improve the interpretation of infrastructure and application metrics. - They are particularly useful for identifying: - Changes in request volume - Altered resource-consumption patterns - Performance regressions - Long-term growth phases - Recovery or stabilization after an incident - By distinguishing genuine trend changes from normal variation, the method can support better anomaly detection and forecasting. ## Limitations and Tradeoffs - Noisy or sparse data can make breakpoint selection unstable. - A model with too many segments may describe historical noise instead of general behavior. - Sudden outliers can distort regression parameters unless they are handled separately. - Piecewise regression captures trend changes, but it does not automatically explain their causes; engineers still need deployment, traffic, and infrastructure context. Piecewise regression is therefore best treated as a lightweight, interpretable tool for detecting changes in metric behavior. It provides more realistic trend modeling than a single regression line while remaining simpler and easier to operate than highly complex forecasting models.

synthetic-monitoringk8sdatadogobservability+3

datadogJan 23, 2017

The trouble with mounting | Datadog (opens in new tab)

The post explains why filesystem mounting becomes surprisingly complex in containerized Linux environments. Datadog’s Agent needs to inspect host filesystems from inside a container, but mount namespaces, bind mounts, and propagation rules can make the host’s view incomplete or inconsistent. The conclusion is that reliable mounting requires understanding namespace boundaries and deliberately configuring mount propagation rather than treating mounts as ordinary directory mappings. ## Linux Mount Namespaces - Each process can have its own mount namespace, isolating its view of mounted filesystems. - A container therefore sees a filesystem tree that may differ substantially from the host’s tree. - Bind-mounting a directory into a container does not necessarily expose mounts created beneath that directory. - This is particularly problematic for paths such as `/proc`, `/sys`, `/var/lib/docker`, and other locations containing nested mounts. ## The Problem with Bind Mounts - A bind mount initially exposes only the directory tree visible at the time it is created. - Later mounts beneath the source directory may not appear in the container. - Recursive bind mounts can copy nested mounts, but they introduce their own behavior and compatibility concerns. - Mounts can also be shared, private, or “slave,” determining whether mount and unmount events propagate between namespaces. ## Mount Propagation - Linux mount propagation controls how changes in one namespace are reflected in another. - Shared mounts propagate events in both directions, while slave mounts receive changes without sending them back. - Private mounts isolate changes completely. - Choosing the wrong propagation mode can cause the Agent to miss newly mounted filesystems or, worse, allow container-side changes to affect the host. ## Datadog’s Engineering Challenge - Datadog needs host-level visibility while keeping the monitoring container isolated and safe. - The Agent must account for mounts that appear after startup, including dynamically created container and volume mounts. - Correct behavior depends on both the container runtime configuration and the host’s existing mount topology. - A robust implementation must inspect mount metadata and handle namespace and propagation details explicitly. The practical recommendation is to treat mounting as a namespace and event-propagation problem, not merely a path-sharing mechanism. Systems that need host visibility should use carefully selected recursive mounts and propagation modes, validate the resulting mount tree, and test behavior when mounts are created or removed dynamically.

synthetic-monitoringk8sdatadogobservability+4