How we migrated our acceptance tests to use Synthetic Monitoring (opens in new tab)

Datadog’s Frontend Developer Experience team migrated 565 flaky, maintenance-heavy Puppeteer acceptance tests to Synthetic Monitoring. The change replaced manually scripted browser interactions with recorded tests that could run reliably from CI through a dedicated CLI. The year-long migration improved maintainability and built engineer confidence through documentation, gradual adoption, and non-blocking rollout tooling.

Why the Existing Acceptance Tests Were Failing

• Tests ran in Node.js on a custom Puppeteer-based runner.
• End-to-end tests were flaky because they depended on browsers, virtual graphics, dedicated machines, navigation, and application timing.
• Even simple actions required extensive scripting:
  • Confirming an element existed
  • Checking that it was enabled
  • Performing the interaction
  • Handling compatibility with Puppeteer
• Custom UI elements, such as dropdowns, made reliable automation substantially harder.
• Product changes frequently required updates to both tests and the testing infrastructure.
• The six CI jobs took up to 14 minutes, with total machine time reaching 35 minutes per commit.

Synthetic Monitoring as the Replacement

• The team adopted Datadog’s own Synthetic Monitoring product to record page interactions rather than manually script them.
• They created synthetics-ci, a CLI that:
  • Finds files named *.synthetics.json
  • Accepts configuration overrides
  • Triggers Synthetic tests
  • Polls for result statuses
  • Prints human-readable output
• Because the tool represented a broader pattern for using Datadog from CI/CD, it was generalized into datadog-ci.

Scope of the Migration

At the start of the migration in June 2021, the frontend repository had:

• 300 engineers working in one repository
• Approximately 90 new pull requests and 1,120 commits every day
• Six acceptance-test CI jobs
• 35 minutes of machine time per commit
• 84 relevant files
• 565 tests
• About 100,000 lines of test and infrastructure code

The scale of the repository and its rapid development activity meant the migration needed to be gradual and carefully coordinated.

Building Trust and Adoption

• The team wrote documentation covering:
  • How to write effective Synthetic tests
  • Which behaviors were worth testing
  • Which testing patterns to avoid
• They demonstrated the system in company-wide and frontend-focused meetings.
• Teams learned how to:
  • Record tests through the UI
  • Use scheduled tests in CI
  • Replace existing acceptance tests
  • Reduce ongoing maintenance
• The team worked directly with groups that owned the largest test collections.
• Jira tickets tracked the migration of every acceptance test and assigned ownership to the appropriate team.

Gradual CI Integration

• A non-blocking CI job allowed teams to introduce Synthetic tests without risking the entire pipeline.
• Failures appeared as pull-request comments rather than blocking merges.
• This gave engineers time to understand and trust the new system.
• Once tests ran reliably, the team made the pipeline blocking.
• The old acceptance-testing platform could then be retired progressively instead of being removed all at once.

The migration took roughly one year and succeeded through a combination of better tooling, clear communication, incremental rollout, and shared ownership across frontend teams. For large organizations replacing a critical testing system, introducing the new workflow safely before enforcing it can make adoption far less disruptive.