Background Coding Agents: Predictable Results Through Strong Feedback Loops (Honk, Part 3) | Spotify Engineering
Spotify argues that unsupervised coding agents become reliable only when surrounded by strong, automated feedback loops. Its “Honk” system uses component-specific verifiers, mandatory pre-PR checks, and an LLM judge to catch build failures, test failures, scope creep, and functionally incorrect changes. The conclusion is that constrained, sandboxed agents with rich verification are more predictable than flexible agents operating independently. ## Failure Modes at Scale - Agents may fail to produce a pull request, which is inconvenient but usually manageable. - They may produce PRs that fail CI, leaving engineers to repair incomplete work. - Most seriously, they may produce PRs that pass CI but are functionally wrong and potentially reach production. - These failures are more likely when components lack tests, agents modify code beyond the prompt, or agents cannot correctly run builds and tests. - Reviewing invalid or nonsensical PRs can become a significant engineering time sink. ## Verification Loops - Honk uses independent verifiers that provide incremental feedback while the agent works. - Verifiers activate automatically based on the repository contents; for example, a Maven verifier runs when a root-level `pom.xml` is present. - The agent sees an abstract MCP tool rather than the implementation details of Maven, test runners, or build systems. - Verifiers handle formatting, compilation, testing, and output parsing, returning concise error messages instead of consuming the agent’s context with raw logs. - All applicable verifiers run before a PR is opened. In Claude Code, this is enforced with a stop hook. - If verification fails, the PR is blocked and the user receives an error. ## An LLM as a Judge - Deterministic checks cannot detect every problem, especially when an agent makes unnecessary refactors or disables flaky tests. - Honk therefore sends the original prompt and proposed diff to a separate LLM judge. - The judge runs after the regular verifiers and can veto changes that exceed the requested scope. - Across thousands of sessions, the judge rejects roughly one quarter of proposed changes. - Agents successfully correct about half of the vetoed changes. - Spotify has not yet built formal evaluations for the judge, but observed that scope violations are its most common reason for rejection. ## Constrained Agents and Sandboxing - The agent has limited responsibilities: inspect the relevant code, edit files, and invoke verification tools. - Surrounding infrastructure handles prompt creation, pushing code, and user communication through systems such as Slack. - Restricting the agent’s capabilities improves predictability and provides security benefits. - Agents run in heavily sandboxed containers with limited permissions, few installed binaries, and almost no access to surrounding systems. - Spotify reports that agents solve increasingly complex tasks reliably when these feedback loops are present, but often produce unusable code without them. ## Future Expansion - Spotify plans to support more hardware and operating systems. - Current verifiers run only on Linux x86, limiting support for systems that require macOS, such as iOS applications, or ARM64 environments. - The company also intends to integrate Honk more deeply with existing CI/CD pipelines. The practical recommendation is to treat autonomous coding as an infrastructure and verification problem, not merely a prompting problem: keep agents narrowly scoped, isolate them securely, and require layered automated checks before accepting their changes.
Read original(opens in new tab)
