datadogNov 1, 2024

How we built a Ruby library that saves 50% in testing time | Datadog (opens in new tab)

Lengthy CI pipelines and flaky tests often hinder developer productivity by causing unnecessary wait times and costly infrastructure usage. To address this, Datadog developed a Ruby test impact analysis library that dynamically maps tests to specific source files, allowing the CI runner to skip tests unrelated to the latest code changes. By moving beyond standard coverage tools and utilizing low-level Ruby VM interpreter events, this solution significantly reduces testing time while maintaining high performance and correctness. ## The Strategy of Test Impact Analysis * Lengthy CI pipelines (often exceeding 20 minutes) increase the likelihood of intermittent "flaky" failures that are unrelated to current code changes. * While parallelization can reduce time, it increases cloud computing costs and does not mitigate the flakiness of irrelevant tests. * Test impact analysis generates a dynamic map between each test and the source files executed during its run; if a commit doesn't touch those files, the test is safely skipped. * Success depends on three pillars: correctness (never skipping a necessary test), performance (low overhead), and seamlessness (no required code changes for the user). ## Limitations of Standard Coverage Tools * Ruby’s built-in `Coverage` module (enhanced in version 3.1 with `resume`/`suspend` methods) proved incompatible with existing total code coverage tools like `simplecov`. * Initial prototypes using the `Coverage` module showed a performance overhead of 300%, making the test suite four times slower. * The `TracePoint` API was also evaluated as an alternative to spy on code execution via the `line` event, but it still produced a significant median overhead of 200% to 400%. * Benchmarks were conducted using the `rubocop` test suite—a "hard mode" scenario with 20,000+ tests—to ensure the tool could handle high-sensitivity environments. ## Implementing a Custom C Extension * To bypass the limitations of high-level APIs, developers utilized Ruby’s C extension capabilities to hook directly into the Virtual Machine. * The library uses `rb_add_event_hook2` and `rb_thread_add_event_hook` to subscribe to the `RUBY_EVENT_LINE` event at the interpreter level. * The implementation involves a C-based `dd_cov_start` function that triggers when a test begins and a `dd_cov_stop` function to collect the results. * During execution, the tool uses `rb_sourcefile()` to identify the current file and stores it in a Ruby hash only if the file is located within the project’s root directory. For engineering teams struggling with bloated CI pipelines, adopting test impact analysis is a highly effective way to optimize resources. By utilizing tools like Datadog’s Intelligent Test Runner, which leverages low-level VM events for minimal overhead, teams can cut their testing time in half without sacrificing the reliability of their master branch.

test-impact-analysisdatadogperformance-optimizationcontinuous-integration+4

datadogNov 1, 2024

How we built a Ruby library that saves 50% in testing time (opens in new tab)

Datadog built a Ruby test impact analysis library to reduce CI time and avoid rerunning unrelated, flaky tests. The approach maps each test to the source files it executes, then runs only tests affected by a commit. Existing Ruby coverage APIs were too slow or incompatible with standard coverage tools, so Datadog developed a faster solution using Ruby VM interpreter events. ## The CI Problem - Large test suites often take 20 minutes or more and may fail because of unrelated flaky tests. - Parallel execution reduces runtime but increases cloud costs and does not eliminate flakiness. - Selective testing can reduce: - Pipeline duration - Cloud resource usage - Exposure to unrelated flaky tests - Test impact analysis determines which source files each test executes and compares them with files changed in the latest Git commit. ## Requirements for Test Impact Analysis Datadog’s library needed to provide: - **Correctness:** Never skip a test that could detect a regression. - **Performance:** Add minimal overhead because impact data must be collected on every commit and branch. - **Seamlessness:** Require no user code changes and avoid changing test behavior or interfering with existing tooling. ## Limitations of Ruby Coverage APIs - Ruby’s built-in `Coverage` module can collect per-test coverage using `resume` and `suspend`, introduced in Ruby 3.1. - A prototype using Coverage had two major problems: - It conflicted with tools such as SimpleCov that collect total code coverage. - It added up to 300% overhead, making the test suite roughly four times slower. - Datadog then tried Ruby’s `TracePoint` API, subscribing to the `line` VM event. - TracePoint avoided interference with SimpleCov and provided the required data, but still introduced roughly 200% overhead, reaching 400% in some cases. ## A Custom Coverage Tool Using Ruby VM Events - Datadog examined Ruby’s internals, including: - `coverage.c` - `rb_coverage_resume` - `rb_resume_coverages` - `rb_add_event_hook2` - Ruby’s C extension API supports registering callbacks for `RUBY_EVENT_LINE`, allowing a custom native implementation. - The proof of concept: - Registers a line-event hook for the current thread. - Records the source file for executed lines. - Ignores files outside the project root. - Removes the hook when collection stops. - Returns and resets the collected coverage data for the next test. - Implementing collection closer to the VM was intended to preserve correctness while substantially reducing the overhead of per-test impact tracking. Datadog’s experience shows that selective testing is a promising way to make CI faster and more reliable, but practical test impact analysis requires a low-level implementation. Standard coverage and tracing APIs provide useful functionality but can impose unacceptable performance costs, making a purpose-built native VM-event collector a better fit.

test-impact-analysisdatadogcloud-computingperformance-optimization+4