How we built a Ruby library that saves 50% in testing time

Datadog built a Ruby test impact analysis library to reduce CI time and avoid rerunning unrelated, flaky tests. The approach maps each test to the source files it executes, then runs only tests affected by a commit. Existing Ruby coverage APIs were too slow or incompatible with standard coverage tools, so Datadog developed a faster solution using Ruby VM interpreter events.

The CI Problem

• Large test suites often take 20 minutes or more and may fail because of unrelated flaky tests.
• Parallel execution reduces runtime but increases cloud costs and does not eliminate flakiness.
• Selective testing can reduce:
  • Pipeline duration
  • Cloud resource usage
  • Exposure to unrelated flaky tests
• Test impact analysis determines which source files each test executes and compares them with files changed in the latest Git commit.

Requirements for Test Impact Analysis

Datadog’s library needed to provide:

• Correctness: Never skip a test that could detect a regression.
• Performance: Add minimal overhead because impact data must be collected on every commit and branch.
• Seamlessness: Require no user code changes and avoid changing test behavior or interfering with existing tooling.

Limitations of Ruby Coverage APIs

• Ruby’s built-in Coverage module can collect per-test coverage using resume and suspend, introduced in Ruby 3.1.
• A prototype using Coverage had two major problems:
  • It conflicted with tools such as SimpleCov that collect total code coverage.
  • It added up to 300% overhead, making the test suite roughly four times slower.
• Datadog then tried Ruby’s TracePoint API, subscribing to the line VM event.
• TracePoint avoided interference with SimpleCov and provided the required data, but still introduced roughly 200% overhead, reaching 400% in some cases.

A Custom Coverage Tool Using Ruby VM Events

• Datadog examined Ruby’s internals, including:
  • coverage.c
  • rb_coverage_resume
  • rb_resume_coverages
  • rb_add_event_hook2
• Ruby’s C extension API supports registering callbacks for RUBY_EVENT_LINE, allowing a custom native implementation.
• The proof of concept:
  • Registers a line-event hook for the current thread.
  • Records the source file for executed lines.
  • Ignores files outside the project root.
  • Removes the hook when collection stops.
  • Returns and resets the collected coverage data for the next test.
• Implementing collection closer to the VM was intended to preserve correctness while substantially reducing the overhead of per-test impact tracking.

Datadog’s experience shows that selective testing is a promising way to make CI faster and more reliable, but practical test impact analysis requires a low-level implementation. Standard coverage and tracing APIs provide useful functionality but can impose unacceptable performance costs, making a purpose-built native VM-event collector a better fit.