CI/CD | Techlist.io

datadogNov 1, 2024Original article

How we built a Ruby library that saves 50% in testing time | Datadog (opens in new tab)

Lengthy CI pipelines and flaky tests often hinder developer productivity by causing unnecessary wait times and costly infrastructure usage. To address this, Datadog developed a Ruby test impact analysis library that dynamically maps tests to specific source files, allowing the CI runner to skip tests unrelated to the latest code changes. By moving beyond standard coverage tools and utilizing low-level Ruby VM interpreter events, this solution significantly reduces testing time while maintaining high performance and correctness. ## The Strategy of Test Impact Analysis * Lengthy CI pipelines (often exceeding 20 minutes) increase the likelihood of intermittent "flaky" failures that are unrelated to current code changes. * While parallelization can reduce time, it increases cloud computing costs and does not mitigate the flakiness of irrelevant tests. * Test impact analysis generates a dynamic map between each test and the source files executed during its run; if a commit doesn't touch those files, the test is safely skipped. * Success depends on three pillars: correctness (never skipping a necessary test), performance (low overhead), and seamlessness (no required code changes for the user). ## Limitations of Standard Coverage Tools * Ruby’s built-in `Coverage` module (enhanced in version 3.1 with `resume`/`suspend` methods) proved incompatible with existing total code coverage tools like `simplecov`. * Initial prototypes using the `Coverage` module showed a performance overhead of 300%, making the test suite four times slower. * The `TracePoint` API was also evaluated as an alternative to spy on code execution via the `line` event, but it still produced a significant median overhead of 200% to 400%. * Benchmarks were conducted using the `rubocop` test suite—a "hard mode" scenario with 20,000+ tests—to ensure the tool could handle high-sensitivity environments. ## Implementing a Custom C Extension * To bypass the limitations of high-level APIs, developers utilized Ruby’s C extension capabilities to hook directly into the Virtual Machine. * The library uses `rb_add_event_hook2` and `rb_thread_add_event_hook` to subscribe to the `RUBY_EVENT_LINE` event at the interpreter level. * The implementation involves a C-based `dd_cov_start` function that triggers when a test begins and a `dd_cov_stop` function to collect the results. * During execution, the tool uses `rb_sourcefile()` to identify the current file and stores it in a Ruby hash only if the file is located within the project’s root directory. For engineering teams struggling with bloated CI pipelines, adopting test impact analysis is a highly effective way to optimize resources. By utilizing tools like Datadog’s Intelligent Test Runner, which leverages low-level VM events for minimal overhead, teams can cut their testing time in half without sacrificing the reliability of their master branch.

CI/CD Datadog Performance Optimization Ruby On Rails+4

datadogNov 1, 20243 min readCurated summary

How we built a Ruby library that saves 50% in testing time

Datadog built a Ruby test impact analysis library to reduce CI time and avoid rerunning unrelated, flaky tests. The approach maps each test to the source files it executes, then runs only tests affected by a commit. Existing Ruby coverage APIs were too slow or incompatible with standard coverage tools, so Datadog developed a faster solution using Ruby VM interpreter events. ## The CI Problem - Large test suites often take 20 minutes or more and may fail because of unrelated flaky tests. - Parallel execution reduces runtime but increases cloud costs and does not eliminate flakiness. - Selective testing can reduce: - Pipeline duration - Cloud resource usage - Exposure to unrelated flaky tests - Test impact analysis determines which source files each test executes and compares them with files changed in the latest Git commit. ## Requirements for Test Impact Analysis Datadog’s library needed to provide: - **Correctness:** Never skip a test that could detect a regression. - **Performance:** Add minimal overhead because impact data must be collected on every commit and branch. - **Seamlessness:** Require no user code changes and avoid changing test behavior or interfering with existing tooling. ## Limitations of Ruby Coverage APIs - Ruby’s built-in `Coverage` module can collect per-test coverage using `resume` and `suspend`, introduced in Ruby 3.1. - A prototype using Coverage had two major problems: - It conflicted with tools such as SimpleCov that collect total code coverage. - It added up to 300% overhead, making the test suite roughly four times slower. - Datadog then tried Ruby’s `TracePoint` API, subscribing to the `line` VM event. - TracePoint avoided interference with SimpleCov and provided the required data, but still introduced roughly 200% overhead, reaching 400% in some cases. ## A Custom Coverage Tool Using Ruby VM Events - Datadog examined Ruby’s internals, including: - `coverage.c` - `rb_coverage_resume` - `rb_resume_coverages` - `rb_add_event_hook2` - Ruby’s C extension API supports registering callbacks for `RUBY_EVENT_LINE`, allowing a custom native implementation. - The proof of concept: - Registers a line-event hook for the current thread. - Records the source file for executed lines. - Ignores files outside the project root. - Removes the hook when collection stops. - Returns and resets the collected coverage data for the next test. - Implementing collection closer to the VM was intended to preserve correctness while substantially reducing the overhead of per-test impact tracking. Datadog’s experience shows that selective testing is a promising way to make CI faster and more reliable, but practical test impact analysis requires a low-level implementation. Standard coverage and tracing APIs provide useful functionality but can impose unacceptable performance costs, making a purpose-built native VM-event collector a better fit.

Read original

CI/CD Datadog Cloud Computing Performance Optimization+4

microsoftJul 18, 20243 min readCurated summary

Managed DevOps Pools – The Origin Story

Microsoft’s vast, diverse engineering organization had accumulated more than 5,000 self-hosted Azure DevOps pools, creating duplicated tooling, inconsistent reliability, security gaps, and compliance challenges. Its One Engineering System (1ES) team addressed this with 1ES Hosted Pools, a standardized service for flexible, secure, and scalable CI/CD infrastructure. Adoption reduced costs by more than 60%, cut remaining self-hosted pools to a few dozen, and eventually led to the external Managed DevOps Pools offering. ## The Scale and Challenges of Self-Hosted Infrastructure - Microsoft supports over 100,000 engineers across many businesses, programming languages, operating systems, hardware platforms, build engines, and test frameworks. - By 2021, teams had created: - More than 5,000 self-hosted Azure DevOps pools - Hundreds of thousands of agents - Teams needed capabilities unavailable from Microsoft-hosted agents, including: - Larger compute sizes - Private-network connectivity - Custom images - Stateful agents - Long-running tests - The decentralized approach caused: - Duplicate engineering effort - Uneven support and reliability - Poor resource utilization and higher costs - Inconsistent patching and security practices - Difficult and time-consuming compliance audits ## 1ES Hosted Pools - 1ES developed a standardized internal service for custom Azure DevOps infrastructure. - Teams could connect agents to private resources such as package registries, secret managers, and on-premises services. - They could bring custom images, using centrally maintained images as their base. - Business continuity features allowed backup pools and failover to other Azure regions. - Agents were stateless by default, but teams could reuse stateful agents for better performance through local caches. - Stateful agents were automatically recycled based on age or available disk space. - Teams could select Azure VM families and sizes suited to their workload. - Standby agents could be pre-warmed on schedules or automatically provisioned using historical demand. ## Operational and Business Benefits - **Lower costs:** Infrastructure bills fell by more than 60% through improved utilization, better SKU selection, and selective use of Azure Spot VMs. - **Faster development:** Teams spent less time maintaining CI/CD infrastructure and more time building products. - **Simpler compliance:** Standardized telemetry made audits easier and allowed security and compliance improvements to be deployed centrally. - **Greater mobility:** Developers changing teams no longer had to learn different infrastructure-management systems. - **Improved security:** Features such as Azure Confidential VMs, Trusted Launch, and Secure TPM became available across pools. - **Reduced fragmentation:** By 2024, Microsoft had reduced its remaining self-hosted pools from more than 5,000 to only a few dozen. ## From Internal Platform to Managed DevOps Pools - 1ES first built Hosted Pools as an internal “Host On Behalf Of” service to validate whether centralized management could reduce self-hosting. - Success inside Microsoft, combined with customer demand, led to the external **Managed DevOps Pools (MDP)** service. - Organizations using VM Scale Set agents or self-hosted agents can migrate to MDP to gain standardized scaling, security, compliance, and operational support. - The external offering initially does not include every feature available in 1ES Hosted Pools, though additional capabilities may be added later. Centralizing CI/CD infrastructure can eliminate redundant platform work while improving cost efficiency, security, compliance, and developer productivity. Managed DevOps Pools extends Microsoft’s internal solution to organizations facing similar self-hosting challenges.

Read original

CI/CD DevOps Cloud Infrastructure Azure+3

datadogMay 23, 20243 min readCurated summary

How we migrated our static analyzer from Java to Rust

Datadog migrated its static analyzer from Java to Rust after finding that ANTLR-based parsing was too slow and language support was incomplete. Rust’s strong integration with Tree-sitter enabled broader language coverage, faster scans, and lower memory usage. The migration preserved behavioral parity while tripling performance and reducing memory consumption tenfold. ## Why Performance Became a Priority - Datadog runs analysis directly in customers’ CI environments, often on resource-constrained runners. - On a two-core, 7 GB GitHub Actions runner, medium repositories took about five minutes to scan instead of the target of under three minutes. - Codiga’s previous hosted environment used large, tuned servers, which masked some performance problems. - Java also required customers to use JVM 17+, potentially conflicting with JVM versions already installed in their CI environments. - Improving Java offered limited upside, so the team considered a rewrite despite its cost and risk. ## Static Analyzer Architecture - The analyzer consists primarily of: - A parsing layer that builds an abstract syntax tree (AST). - An execution layer that analyzes the AST, reports violations, and offers fixes. - Tree-sitter generates the AST. - The existing Java binding lacked important functionality, including Tree-sitter pattern matching. - Tree-sitter’s core libraries are implemented in Rust, where support was more complete. - Analysis rules are written in JavaScript and were originally executed through GraalVM’s polyglot capabilities. - Fast parsing, pattern matching, and rule execution were central to meeting the desired CI performance. ## Migrating from Java to Rust - Rust was selected because it is a first-class part of the Tree-sitter ecosystem and provided better access to its features. - The migration required: - Feature parity with the Java implementation. - Identical analysis results and reported violations. - No execution-time regressions. - Migrating the parser was relatively straightforward because Rust support came directly from Tree-sitter. - The Rust implementation: - Tripled analyzer performance. - Reduced memory usage by a factor of ten. - JavaScript execution moved from GraalVM to `deno-core`, a Rust-based V8 integration. - Only the core JavaScript functionality was included. - Disk and network capabilities were excluded because analysis rules do not need them, improving security. ## Migration Strategy and Rust Adoption - The team treated automated equivalence and performance tests as requirements for a successful rewrite. - Rust allowed the analyzer to integrate more directly with its key dependencies rather than maintaining a separate Java binding. - The broader migration also required replacing supporting Java components with corresponding Rust libraries; the article indicates that these mappings were documented as part of the transition. Overall, the move to Rust was justified by the analyzer’s deployment model: faster execution and lower resource consumption directly improved the experience of customers running scans in constrained CI environments.

Read original

CI/CD Database Design JavaScript Rust+4

figmaApr 24, 20243 min readCurated summary

Speeding Up C++ Build Times | Figma Blog

Figma cut C++ build times roughly in half by addressing unnecessary header inclusion rather than relying solely on faster hardware or caching. The team found that compiled bytes were growing much faster than the codebase itself, making transitive header dependencies the main culprit. They combined automated include analysis with CI-based measurement to prevent both unused includes and costly dependency regressions. ## Why Build Times Were Getting Worse - In 2023, Figma’s codebase grew by about 10%, but build times increased by 50%. - C++ builds were a major productivity problem and a top concern in internal developer surveys. - Faster M1 Max machines, Ccache, and remote caching provided only temporary or insufficient improvements. - The team observed that build times were closely related to the amount of code passed to the compiler after preprocessing. ## How C++ Header Inclusion Affects Builds - The preprocessor expands every `#include` into a single large file before compilation. - Transitive dependencies are included as well: - If file C includes B, and B includes A, C receives the contents of both A and B. - As a result, a small source change can cause the compiler to process a very large amount of unrelated code. ## Removing Unnecessary Includes - Figma suspected that many files included headers they did not use directly or relied on headers only for transitive dependencies. - Removing unnecessary includes from the largest files produced: - A 31% reduction in compiled bytes. - A 25% reduction in cold build time. - These results confirmed that compiled byte volume was strongly correlated with build performance. ## DIWYDU: Automating Include Cleanup - Google’s Include What You Use (IWYU) tool was considered but proved difficult to apply retroactively to Figma’s large codebase. - Figma created a less strict alternative called **Don’t Include What You Don’t Use (DIWYDU)**. - DIWYDU: - Uses Python bindings for `libclang`. - Parses source and header files into Clang Abstract Syntax Trees. - Identifies types, functions, and variables directly used by each file. - Flags headers that are included but provide no directly used symbols. - The tool runs on feature branches to prevent unnecessary includes from accumulating. ## DIWYDU’s Limitations - It analyzes Figma-owned files but excludes Standard Template Library headers. - STL headers may define symbols through private internal includes, making direct dependency analysis difficult. - Python’s `libclang` bindings expose less of Clang’s AST than the compiler’s native C++ APIs, sometimes producing `UNEXPOSED_EXPR` nodes. - A future C++ implementation could provide more accurate AST access. - DIWYDU cannot detect cases where an included header is genuinely required but excessively large. - Such regressions may need forward declarations or header decomposition instead. ## Measuring Dependency Growth with `includes.py` - Figma built `includes.py` to measure the transitive bytes associated with each source file. - The tool is written entirely in Python and typically runs in a few seconds without invoking Clang. - It: - Crawls first-party source, header, and generated files. - Counts file sizes. - Builds a dependency graph. - Estimates the total bytes passed to the compiler for each source file. - Standard library includes are treated as zero bytes because Figma mainly accesses them through internal wrapper directories. - CI uses the measurements to compare pull requests and warn authors when changes significantly increase compiled bytes. Figma’s approach demonstrates that controlling header dependencies can deliver larger and more durable gains than simply adding hardware or cache capacity. Teams working on large C++ codebases should automate unused-include checks, measure transitive dependency size in CI, and use forward declarations or smaller headers when necessary.

Read original

CI/CD Python C++Static Analysis+4

datadogFeb 14, 2024Original article

How we use Vale to improve our documentation editing process | Datadog (opens in new tab)

To manage a high volume of technical content across dozens of products, Datadog’s documentation team has automated its editorial process using the open-source linting tool Vale. By integrating these checks directly into their CI/CD pipeline via GitHub Actions, the team ensures prose consistency and clarity while significantly reducing the manual burden on technical writers. This "shift-left" approach empowers both internal and external contributors to identify and fix style issues independently before a formal human review begins. ### Scaling Documentation Workflows * The Datadog documentation team operates at a 200:1 developer-to-writer ratio, managing over 1,400 contributors and 35 distinct products. * In 2023 alone, the team merged over 20,000 pull requests covering 650 integrations, 400 security rules, and 65 API endpoints. * On-call writers review an average of 40 pull requests per day, necessitating automation to handle triaging and style enforcement efficiently. ### Automated Prose Review with Vale * Vale is implemented as a command-line tool and a GitHub Action that scans Markdown and HTML files for style violations. * When a contributor opens a pull request, the linter provides automated comments in the "Files Changed" tab, flagging long sentences, wordy phrasing, or legacy formatting habits. * This automation reduces the "mental toll" on writers by filtering out repetitive errors before they reach the human review stage. ### Codifying Style Guides into Rules * The team transitioned from static editorial guidelines stored in Confluence and wikis to a codified repository called `datadog-vale`. * Style rules are defined using Vale’s YAML specification, allowing the team to update global standards in a single location that is immediately active in the CI pipeline. * Custom regular expressions are used to exclude specific content from validation, such as Hugo shortcodes or technical snippets that do not follow standard prose rules. ### Implementation of Specific Linting Rules * **Jargon and Filler Words:** A `words.yml` file flags "cruft" such as "easily" or "simply" to maintain a professional, objective tone. * **Oxford Comma Enforcement:** The `oxfordcomma.yml` rule uses regex to identify lists missing a serial comma and provides a suggestion to the author. * **Latin Abbreviations:** The `abbreviations.yml` rule identifies terms like "e.g." or "i.e." and suggests plain English alternatives like "for example" or "that is." * **Timelessness:** Rules flag words like "currently" or "now" to ensure documentation remains relevant without frequent updates. By open-sourcing their Vale configurations, Datadog provides a framework for other organizations to automate their style guides and foster a more efficient, collaborative documentation culture. Teams looking to improve prose quality should consider adopting a similar "docs-as-code" approach to shift editorial effort toward the beginning of the contribution lifecycle.

CI/CD Datadog GitHub Actions Markdown+4

datadogFeb 14, 20243 min readCurated summary

How we use Vale to improve our documentation editing process

Datadog’s Documentation team uses automated style linting to maintain clear, consistent prose across a large, fast-moving documentation repository. By integrating the open-source Vale linter into local authoring workflows and GitHub Actions, the team moves copy editing closer to the moment content is written. This reduces review effort, helps contributors fix issues themselves, and makes the team’s style guide executable rather than scattered across multiple documents. ## Documentation at Scale - The Documentation team grew from 7 to 14 writers while supporting roughly 200 developers per writer. - The repository includes documentation for 35 products and more than 1,400 internal and external contributors. - In 2023, the team merged more than 20,000 pull requests covering: - 30+ products - 65 API endpoints - 95 Marketplace integrations - 400 security compliance rules - 400 workflow actions - 650 integrations - An on-call writer reviews more than 40 pull requests per day, making automated consistency checks especially valuable. ## Why Manual Style Enforcement Falls Short - Writers must catch issues such as: - Jargon and wordy phrasing - Malapropisms - Mismatched tenses - Gendered language - Typewriter-era formatting habits - Organization-specific preferences - Contributors and AI writing tools may not know Datadog’s conventions, such as using serial commas, avoiding “via,” or eliminating time-sensitive words like “currently.” - Previously, style guidance had to be maintained in Confluence, review documentation, contributing guides, and repository wiki pages. ## Vale in Authoring and CI - Datadog adopted Vale, an open-source command-line prose linter, through the `datadog-vale` project. - A GitHub Action runs Vale against Markdown and HTML files in pull requests. - The repository’s `vale.ini` file identifies: - Where style rules are stored - Which rules should run - Which content formats should be checked - Automated comments appear in GitHub’s **Files Changed** view, allowing contributors to correct issues before a writer reviews the pull request. - Vale has reduced editing time and the mental burden on writers while improving contributor self-service. ## Turning the Style Guide into Rules - Existing editorial guidelines were converted into YAML-based Vale rules. - New rules can be added once and enforced everywhere, avoiding duplicated documentation. - Regular expressions exclude content that should not be linted, such as Hugo shortcodes. - Rules can identify both broad writing problems and precise organizational preferences. ## Examples of Vale Rules - A `words.yml` file can flag unnecessary jargon or “cruft” such as “easily” and “simply.” - An `oxfordcomma.yml` rule detects sentences that omit the Oxford comma and provides a correction message and link to the relevant style guidance. - An `abbreviations.yml` rule replaces Latin abbreviations with plain-English alternatives: - `e.g.` → “for example” - `i.e.` → “that is” - `etc.` → “and more” - Vale rules can define severity levels such as `suggestion` or `error`, include explanatory messages, link to documentation, and optionally perform replacements. Datadog’s approach demonstrates that documentation quality can be improved by treating prose standards like code standards: encode them as rules, run them continuously, and give authors immediate, actionable feedback. Teams with large contributor bases can use Vale and CI to make their style guide consistent, discoverable, and easier to maintain.

Read original

CI/CD Datadog GitHub Actions Markdown+4

figmaAug 28, 20232 min readCurated summary

Keeping Figma Fast | Figma Blog

Figma’s original performance-testing system relied on a single MacBook running a few repeated scenarios, which worked when the product and team were smaller. As Figma expanded into plugins, FigJam, Dev Mode, and many other features, that approach became too limited and fragile—the laptop eventually overheated while the company was remote. Figma concluded it needed a scalable, proactive testing framework capable of catching regressions across its growing codebase. ## From One Laptop to a Growing Product - In 2018, one MacBook repeatedly ran performance scenarios and reported timing changes to a shared dashboard. - A major renderer restructuring and WebAssembly fixes made Figma three times faster. - Over five years, the product grew substantially, adding: - Plugins and Community features - FigJam - Dev Mode - Numerous ongoing product updates - The existing test files could no longer represent the product’s expanding feature set and edge cases. ## Limits of the Original Testing Process - Granular performance tests measure specific behavior at scale, such as panning through a file while 100 users edit layers and type simultaneously. - Creating feature-specific tests became impractical as Figma grew beyond 400 engineers and managers. - Increasing release velocity made it difficult for any individual or single machine to track every performance-affecting change. - During the shift to remote work in 2020, the office MacBook remained running unattended and eventually overheated. - Attempts to reproduce the setup on another laptop were unsuccessful, demonstrating that the system was not operationally scalable. ## Requirements for a New System Figma used the overhaul to define an ideal performance-testing framework: - **Test every proposed code change:** Performance checks should run against changes in the main monorepo, allowing regressions to be found during development rather than after users encounter them. - **Support proactive performance work:** Small delays can significantly disrupt users who spend hours working in Figma. - **Run tests in parallel:** Dozens of stress scenarios would need to execute simultaneously, similar to Figma’s existing cloud-based CI testing. - **Finish quickly:** Performance guardrail checks were required to complete in under 10 minutes. - **Scale across real hardware:** Running every pull request on physical machines could require roughly 100 identical runners at peak capacity. Figma’s experience shows that performance testing must evolve alongside product complexity. A lightweight single-machine setup can be effective initially, but larger teams and faster release cycles require automated, parallel, hardware-aware testing integrated directly into CI.

Read original

CI/CD Webassembly Monorepo Performance Testing+2

datadogJun 30, 20233 min readCurated summary

How we migrated our acceptance tests to use Synthetic Monitoring

Datadog’s Frontend Developer Experience team migrated 565 flaky, maintenance-heavy Puppeteer acceptance tests to Synthetic Monitoring. The change replaced manually scripted browser interactions with recorded tests that could run reliably from CI through a dedicated CLI. The year-long migration improved maintainability and built engineer confidence through documentation, gradual adoption, and non-blocking rollout tooling. ## Why the Existing Acceptance Tests Were Failing - Tests ran in Node.js on a custom Puppeteer-based runner. - End-to-end tests were flaky because they depended on browsers, virtual graphics, dedicated machines, navigation, and application timing. - Even simple actions required extensive scripting: - Confirming an element existed - Checking that it was enabled - Performing the interaction - Handling compatibility with Puppeteer - Custom UI elements, such as dropdowns, made reliable automation substantially harder. - Product changes frequently required updates to both tests and the testing infrastructure. - The six CI jobs took up to 14 minutes, with total machine time reaching 35 minutes per commit. ## Synthetic Monitoring as the Replacement - The team adopted Datadog’s own Synthetic Monitoring product to record page interactions rather than manually script them. - They created `synthetics-ci`, a CLI that: - Finds files named `*.synthetics.json` - Accepts configuration overrides - Triggers Synthetic tests - Polls for result statuses - Prints human-readable output - Because the tool represented a broader pattern for using Datadog from CI/CD, it was generalized into `datadog-ci`. ## Scope of the Migration At the start of the migration in June 2021, the frontend repository had: - 300 engineers working in one repository - Approximately 90 new pull requests and 1,120 commits every day - Six acceptance-test CI jobs - 35 minutes of machine time per commit - 84 relevant files - 565 tests - About 100,000 lines of test and infrastructure code The scale of the repository and its rapid development activity meant the migration needed to be gradual and carefully coordinated. ## Building Trust and Adoption - The team wrote documentation covering: - How to write effective Synthetic tests - Which behaviors were worth testing - Which testing patterns to avoid - They demonstrated the system in company-wide and frontend-focused meetings. - Teams learned how to: - Record tests through the UI - Use scheduled tests in CI - Replace existing acceptance tests - Reduce ongoing maintenance - The team worked directly with groups that owned the largest test collections. - Jira tickets tracked the migration of every acceptance test and assigned ownership to the appropriate team. ## Gradual CI Integration - A non-blocking CI job allowed teams to introduce Synthetic tests without risking the entire pipeline. - Failures appeared as pull-request comments rather than blocking merges. - This gave engineers time to understand and trust the new system. - Once tests ran reliably, the team made the pipeline blocking. - The old acceptance-testing platform could then be retired progressively instead of being removed all at once. The migration took roughly one year and succeeded through a combination of better tooling, clear communication, incremental rollout, and shared ownership across frontend teams. For large organizations replacing a critical testing system, introducing the new workflow safely before enforcing it can make adoption far less disruptive.

Read original

CI/CD Datadog Synthetic Monitoring Node.Js+3

datadogFeb 10, 20212 min readCurated summary

Engineering spotlight: Maël Nison | Datadog

Datadog announces that it has been named a Leader in the 2026 Gartner® Magic Quadrant™ for Observability Platforms. The announcement positions Datadog as a broad observability platform spanning infrastructure, applications, logs, security, digital experience, software delivery, service management, and AI. The provided content does not include Gartner’s detailed evaluation or the blog post’s supporting arguments. ## Recognition and Platform Scope - Datadog highlights its leadership placement in Gartner’s observability-platform research. - Its platform covers: - Infrastructure and container monitoring - Application performance monitoring and profiling - Database, data-stream, and jobs monitoring - Log management and observability pipelines - Cloud, application, workload, and code security - Browser and mobile real user monitoring - Synthetic monitoring, session replay, and error tracking - CI visibility, testing, code coverage, and feature flags - Incident response, service catalogs, SLOs, and workflow automation ## AI and Automation - Datadog presents AI as an integrated part of its platform through: - Bits AI agents and investigation tools - AI integrations and agent observability - GPU monitoring - MCP Server and agent-building capabilities - AI-assisted security and developer workflows - Additional automation features include Watchdog, fleet automation, workflow automation, and incident-management tools. ## Overall Positioning - The product catalog emphasizes a unified approach to monitoring technology environments rather than separate tools for infrastructure, applications, security, and user experience. - The platform also includes dashboards, alerts, notebooks, governance controls, access management, and mobile access. The announcement’s central message is that Datadog combines extensive observability coverage with security, delivery, service-management, and AI capabilities. Readers seeking the actual Gartner assessment should consult the linked Magic Quadrant resource, since the supplied text contains only the announcement and navigation information.

Read original

CI/CD Kubernetes Datadog Observability+4

datadogJun 3, 20193 min readCurated summary

Secure publication of Datadog Agent integrations with TUF and in-toto

Datadog built a compromise-resilient CI/CD system to publish Agent integrations independently of full Agent releases. It combines in-toto for end-to-end supply-chain verification with TUF for secure key and metadata distribution. Together, these technologies ensure that users install only integrations derived from developer-approved source code, even if parts of the infrastructure are compromised. ## Independent Integration Publishing - Agent integrations were traditionally bundled into full Agent releases. - This delayed important integration updates and prevented users from trying new integrations immediately. - Datadog wanted automation to build and publish integrations on demand without fully trusting the automation itself. ## End-to-End Verification with in-toto - TLS and GPG package signatures help prevent man-in-the-middle attacks but do not protect against compromised build or publishing infrastructure. - in-toto defines the software supply chain as a fixed sequence of signed steps. - Each step records the inputs it received and the outputs it produced, allowing the Agent to verify that only authorized parties performed the required work. - The integration pipeline includes: - Developers signing Python and YAML source files. - CI/CD packaging the source into Python wheels without modifying existing wheels. - A signing step applying TUF signatures to the wheels. - The Datadog Agent verifying that the downloaded wheel matches the developer-signed source. ## Secure Distribution with TUF - in-toto does not itself provide a secure way to distribute, revoke, or replace verification keys. - TUF supplies signed, compromise-resilient metadata for: - The root of trust for wheels and supply-chain metadata. - The in-toto-defined workflow. - Public keys used to verify the workflow. - TUF protects against tampering, rollback attacks, and indefinite replay of outdated metadata. - Offline trust bootstrapping and protected developer keys are essential to the overall security model. ## Hardware-Protected Developer Signing - Developers use Yubikeys to generate and store GPG signing keys. - Private keys cannot be exported from the device, assuming correct firmware. - Signing requires both a secret PIN and physical interaction with the Yubikey. - A command-line tool integrates in-toto and GPG, preserving a convenient developer workflow while reducing key-compromise risk. ## Transparent Verification for Users - The Datadog Agent automatically invokes TUF and in-toto when downloading or updating integrations. - Users need no workflow changes under normal conditions. - If metadata, signatures, or supply-chain steps fail verification, installation is blocked and the Agent reports the failure. Datadog’s approach demonstrates that secure automated publishing requires layered controls: in-toto verifies how software was produced, while TUF securely manages the trust and distribution mechanisms needed to validate it.

Read original

CI/CD Database Design Datadog Python+4

datadogAug 15, 20172 min readCurated summary

Improving cloud security visibility with ChatOps | Datadog

Datadog announces that it was named a Leader in the 2026 Gartner® Magic Quadrant™ for Observability Platforms. The provided content does not include the blog post’s body or Gartner’s evaluation details; it mainly contains Datadog’s navigation menu and product links. ## Announcement - Datadog highlights its recognition as a Leader in Gartner’s 2026 Observability Platforms Magic Quadrant. - The linked resource appears to be a Gartner-related announcement rather than a technical deep dive. ## Datadog’s Product Scope The navigation reflects a broad observability and operations platform covering: - Infrastructure monitoring, metrics, containers, Kubernetes, networks, serverless systems, and cloud costs - Application performance monitoring, profiling, dynamic instrumentation, and agent observability - Database, data-stream, jobs, and quality monitoring - Log management, sensitive-data scanning, audit trails, and observability pipelines - Security capabilities including cloud security, SIEM, workload protection, code security, and vulnerability management - Digital experience tools such as real-user monitoring, session replay, synthetic monitoring, and error tracking - Software delivery, CI visibility, testing, feature flags, and code coverage - Incident response, service catalogs, SLOs, workflow automation, and case management - AI agents, GPU monitoring, AI integrations, and investigation tools The supplied excerpt does not provide enough information to summarize Gartner’s criteria, Datadog’s strengths or weaknesses, or the report’s comparative findings.

Read original

CI/CD Kubernetes Datadog Observability+4