Background Coding Agents: Supercharging Downstream Consumer Dataset Migrations (Honk, Part 4) | Spotify Engineering (opens in new tab)
Spotify used its Honk background coding agent with Backstage and Fleet Management to automate migrations from two deprecated datasets to new versions. The effort targeted roughly 1,800 downstream pipelines and produced 240 automated pull requests, potentially saving about 10 engineering weeks. The experience showed that agents perform best when repositories follow standardized patterns, prompts contain precise technical context, and automated testing is available. ## The Challenge of Large-Scale Dataset Migrations - Two heavily used datasets needed replacement to support new dimensions and features. - The datasets had approximately 1,800 direct downstream pipelines and affected thousands more indirectly. - Migrations spanned three frameworks: - BigQuery Runner - dbt - Scala-based Scio - Manual migration was estimated to require around 10 engineering weeks within a six-month deadline. ## Using Backstage to Identify Consumers - Backstage’s endpoint lineage pages revealed downstream dataset consumers. - Its Codesearch plugin located relevant repositories across Spotify’s GitHub Enterprise environment. - The Fleetshift plugin used those results to organize and orchestrate repository migrations. - Backstage also provided a centralized view for tracking progress and opening generated pull requests. ## Context Engineering for Honk - Honk needed detailed, self-contained prompts because it could not access external documentation, dataset schemas, MCPs, or custom Claude skills during execution. - Scio was excluded because its flexible, inconsistent implementations made it difficult to describe all migration cases in one reliable prompt. - BigQuery Runner and dbt were more standardized, making them better candidates for automation. - An initial prompt based on a human migration guide was insufficient and caused incorrect assumptions about field mappings. - Explicit mapping tables in the context file significantly improved results. - Prompts also specified cases where fields should not be migrated automatically. - Honk left those fields unchanged. - It added comments linking to human migration guidance for later review. ## Testing and Automated Pull Requests - BigQuery Runner and dbt repositories generally lacked build-time unit tests. - As a result, Honk could not automatically verify and correct its changes, one of its key capabilities. - Downstream teams had to manually test the generated pull requests before merging. - Despite this limitation, the team successfully created 240 automated migration PRs. - Fleetshift’s Backstage interface simplified monitoring, troubleshooting, repository navigation, and communication with owning teams. ## Lessons for Future Agent-Driven Maintenance - Large-scale automation depends on standardizing frameworks and data practices across repositories. - Consistent testing and validation requirements are essential so agents can verify their own changes. - Future Honk functionality will allow agents to gather context from sources such as JIRA tickets and documentation before editing code. - Better context gathering should reduce the need for exhaustive prompt files and improve migration quality. Spotify’s experience suggests that background coding agents can substantially reduce migration toil, but their effectiveness depends on disciplined standardization, explicit migration rules, and strong automated testing.