spotify·May 18, 2026·3 min readCurated summary

Coding Is No Longer the Constraint: Scaling Developer Experience to Teams and Agents at Spotify | Spotify Engineering

Spotify argues that AI has shifted software development’s main constraint from writing code to coordinating people, systems, and decisions. Years of investment in standardized platforms, automation, and developer experience enabled Spotify to adopt AI coding tools at extraordinary scale. The company’s experience suggests that consistent infrastructure and strong feedback loops are essential for making both human developers and coding agents effective. ## Rapid AI Adoption - More than 99% of Spotify engineers use AI coding tools weekly. - 94% report improved productivity. - Pull request frequency has increased by 76%, with most PRs created by developers working alongside AI agents. - Adoption accelerated sharply after the release of Claude Opus 4.5. ## Fleet Management Before AI Agents - Spotify’s codebase was growing seven times faster than its engineering workforce. - Developers increasingly spent time on dependency upgrades, API migrations, and vulnerability fixes. - Fleet Management automated changes across hundreds or thousands of components. - Its orchestration system, Fleetshift, has merged more than 2.5 million maintenance PRs, most without human intervention. - This approach reduced migrations from work taking weeks or months across many teams to centrally managed operations. ## Honk: A Background Coding Agent - Deterministic scripts struggled with complex refactoring and the edge cases found across large codebases. - Spotify created Honk, a background coding agent powered by Claude through the Agent SDK. - Honk runs in Kubernetes pods, allowing many coding sessions to execute concurrently. - It can use trusted tools and run builds in CI across multiple operating systems. - Fleetshift identifies targets, schedules work, and tracks PRs, while Honk performs the code changes. - A recent Java migration across Spotify’s backend services took three days. - Engineers can invoke Honk through Slack, where it uses conversation context to create and return PRs. - Honk v2 adds shared sessions, team projects, and agent orchestration through Chirp. ## Standardization Improves Agent Performance - Spotify’s principle of limiting the number of technologies it supports reduces decisions and improves collaboration. - Consistent service architectures and design patterns also give AI agents better reference material. - Agents perform worse in fragmented codebases with inconsistent conventions. - Backstage provides a unified internal developer portal and catalog for software components. - Spotify exposes Backstage capabilities to agents through MCP integrations and command-line tools. - Agents can discover component ownership, read documentation, and contact responsible teams. ## Guardrails Through Backstage - Backstage’s Soundcheck and “golden state” define recommended technologies and practices. - Teams can assess their components against these standards. - Static analysis and linting provide immediate feedback when developers or agents use unsuitable patterns. - This creates a feedback loop that helps agents correct their work and drives consistency across the organization. Spotify’s experience indicates that scaling AI development requires more than giving engineers access to models. Organizations should invest in standardized platforms, searchable component metadata, automated fleet-wide workflows, and strong validation systems so agents can operate reliably at team scale.

Read original(opens in new tab)

CI/CDAI AgentsKubernetesSlack+4

spotify·Feb 19, 2026·3 min readCurated summary

Background Coding Agents: Predictable Results Through Strong Feedback Loops (Honk, Part 3) | Spotify Engineering

Spotify argues that unsupervised coding agents become reliable only when surrounded by strong, automated feedback loops. Its “Honk” system uses component-specific verifiers, mandatory pre-PR checks, and an LLM judge to catch build failures, test failures, scope creep, and functionally incorrect changes. The conclusion is that constrained, sandboxed agents with rich verification are more predictable than flexible agents operating independently. ## Failure Modes at Scale - Agents may fail to produce a pull request, which is inconvenient but usually manageable. - They may produce PRs that fail CI, leaving engineers to repair incomplete work. - Most seriously, they may produce PRs that pass CI but are functionally wrong and potentially reach production. - These failures are more likely when components lack tests, agents modify code beyond the prompt, or agents cannot correctly run builds and tests. - Reviewing invalid or nonsensical PRs can become a significant engineering time sink. ## Verification Loops - Honk uses independent verifiers that provide incremental feedback while the agent works. - Verifiers activate automatically based on the repository contents; for example, a Maven verifier runs when a root-level `pom.xml` is present. - The agent sees an abstract MCP tool rather than the implementation details of Maven, test runners, or build systems. - Verifiers handle formatting, compilation, testing, and output parsing, returning concise error messages instead of consuming the agent’s context with raw logs. - All applicable verifiers run before a PR is opened. In Claude Code, this is enforced with a stop hook. - If verification fails, the PR is blocked and the user receives an error. ## An LLM as a Judge - Deterministic checks cannot detect every problem, especially when an agent makes unnecessary refactors or disables flaky tests. - Honk therefore sends the original prompt and proposed diff to a separate LLM judge. - The judge runs after the regular verifiers and can veto changes that exceed the requested scope. - Across thousands of sessions, the judge rejects roughly one quarter of proposed changes. - Agents successfully correct about half of the vetoed changes. - Spotify has not yet built formal evaluations for the judge, but observed that scope violations are its most common reason for rejection. ## Constrained Agents and Sandboxing - The agent has limited responsibilities: inspect the relevant code, edit files, and invoke verification tools. - Surrounding infrastructure handles prompt creation, pushing code, and user communication through systems such as Slack. - Restricting the agent’s capabilities improves predictability and provides security benefits. - Agents run in heavily sandboxed containers with limited permissions, few installed binaries, and almost no access to surrounding systems. - Spotify reports that agents solve increasingly complex tasks reliably when these feedback loops are present, but often produce unusable code without them. ## Future Expansion - Spotify plans to support more hardware and operating systems. - Current verifiers run only on Linux x86, limiting support for systems that require macOS, such as iOS applications, or ARM64 environments. - The company also intends to integrate Honk more deeply with existing CI/CD pipelines. The practical recommendation is to treat autonomous coding as an infrastructure and verification problem, not merely a prompting problem: keep agents narrowly scoped, isolate them securely, and require layered automated checks before accepting their changes.

Read original(opens in new tab)

CI/CDLarge Language ModelsAI AgentsModel Context Protocol+4

spotify·Feb 19, 2026·3 min readCurated summary

How We Release the Spotify App: A Look Under the Hood (Part 2) | Spotify Engineering

Spotify’s Release Manager Dashboard replaced a Jira-heavy workflow with a unified command center for mobile and desktop releases. It reduces context switching and cognitive load by aggregating release status, bugs, testing, build health, and usage metrics into one interface. A backend that caches and pre-aggregates data from roughly ten systems makes the dashboard fast and affordable. ## From Jira to a Release Command Center - Previously, Release Managers depended on Jira tickets, multiple browser tabs, and Slack conversations. - This made it easy to miss details and required constant context switching. - The dashboard was designed to: - Prioritize the Release Manager’s workflow. - Remain understandable to anyone familiar with Spotify’s release process. - Reduce cognitive load. - Support fast, accurate decisions. ## Release Data Spotify treats each platform-and-version combination as a **track**. Android, iOS, and Desktop share some libraries but are released independently. - Track-specific information includes: - Current release state. - Release-blocking bugs. - Team regression-testing sign-offs. - Final release candidate build status. - Build verification test results. - App Store upload status. - Crash, ANR, and CPU-exception rates per song. - Daily active users. - The dashboard also highlights: - Blocking bugs without an assigned version. - Bugs without a priority. - Reports from internal users and alpha/beta testers. - Release management includes finding appropriate owners for unassigned issues, even when temporary team ownership is needed. ## React, TypeScript, and Backstage - The dashboard is a Backstage plugin built with React and TypeScript. - Spotify’s Backstage ecosystem already provides: - Software Catalog functionality for distributing builds to app stores. - App-build and crash plugins with deeper detail. - Shared UI components and data across developer tools. - The interface provides a quick health overview, with drill-down capabilities for investigating blockers. - Status colors communicate urgency: - **Green:** Ready for the next stage. - **Yellow:** Something still needs attention. - **Red:** An error requires corrective action. ## Backend Aggregation and Performance - A dedicated backend acts as an API gateway for approximately ten existing systems. - It consolidates their data into one consistent API for the dashboard. - The initial implementation queried large amounts of data on every reload, making it slow and expensive. - Caching and five-minute pre-aggregation reduced load time to about eight seconds while significantly lowering operating costs. ## Dashboard Sections ### Production - Shows the currently deployed Android, iOS, and Desktop versions. - Since these releases have completed the release process, only production metrics are displayed. - Metrics include: - Crash data. - Rolling daily active users over the previous 24 hours. - This helps Release Managers detect problems shortly after rollout. ### Current - Displays the branched version that has not yet reached production. - Tracks release blockers such as: - Open blocking bugs. - Incomplete regression testing. - Crash rates above release thresholds. - Builds that do not contain the latest release-branch commits. - Yellow indicators represent pending work, while red indicators call for direct investigation or action. - The ITGC section confirms that full production rollout is permitted only after: - ITGC tests pass. - Reporting is correct. - Data loss remains below the defined threshold. - A Release Status Ping link generates a Slack update with the release’s current state. ### Upcoming - Mirrors the Current release view for the next planned version. - Sections that are not yet relevant are shown in a grayed-out state. The dashboard illustrates how a specialized aggregation layer and focused UI can turn a fragmented release process into a clear operational workflow. For organizations managing complex, multi-platform releases, combining cached cross-system data with color-coded status and drill-down details can improve both speed and release safety.

Read original(opens in new tab)

CI/CDReactTypeScriptJira+4