spotifyJul 20, 2026

Content Ingestion & Podcast Video Incident Report | Spotify Engineering (opens in new tab)

Spotify’s June 24 publishing incident delayed video podcast episodes for several hours after transcoding capacity was overwhelmed. The disruption resulted from insufficient capacity headroom, a concurrent batch job, increased processing costs, and a scheduling bug that reduced throughput by about 10%. Spotify cleared the backlog by the following morning and has since expanded capacity, improved monitoring, and launched a broader publishing reliability initiative. ## What Caused the Delay - A spike in new video podcast submissions pushed transcoding infrastructure close to its maximum capacity. - A scheduled batch job was reprocessing existing episodes, consuming capacity needed for new publications. - Recent video-quality improvements increased the processing time and compute required per episode. - A resource-scheduling bug following an infrastructure migration underused available hardware, reducing throughput by approximately 10%. - Creators re-uploaded missing episodes, unintentionally adding more load because Spotify did not clearly confirm that uploads had been received and queued. ## Incident Timeline and Response - Early monitoring alerts fired at 13:30 UTC, but they were not initially recognized as a broader capacity problem. - At 15:00, a delivery spike brought transcoding close to capacity. - The batch job was stopped at 16:35 to free resources. - Formal incident response began at 17:34 after queue thresholds were exceeded. - A scheduling fix was deployed at 20:49, and an additional processing cluster came online at 00:14 on June 25. - All queues were cleared by 01:02, with normal pipeline operation confirmed at 07:30. - Approximately four hours passed between the first alerts and formal incident response, highlighting a major monitoring and escalation gap. ## Remediation Measures - Spotify increased transcoding capacity by approximately 67% to provide more room for traffic spikes and batch processing. - The resource-scheduling bug was fixed, restoring use of previously underutilized compute capacity. - Monitoring was improved to warn earlier when capacity is approaching operational limits. ## Broader Reliability Program - Capacity planning is being expanded to account for burst traffic, background workloads, and recovery needs—not only typical steady-state demand. - Publishing systems will prioritize real-time creator uploads over background operations. - Spotify is extending rate limiting and backpressure mechanisms across the pipeline so unexpected load can be handled more gracefully. - A dedicated cross-team effort is coordinating these improvements across the publishing infrastructure. Spotify’s main lesson is that reliable publishing requires both additional capacity and better operational safeguards. Earlier detection, clearer upload acknowledgments, stronger workload prioritization, and deliberate burst-capacity planning should reduce the likelihood and impact of similar incidents.

monitoringincident-responserate-limitingvideo-transcoding+4

spotifyJun 3, 2026

Encoding Your Domain Expert: The Context Layer Behind Spotify's Data Assistant | Spotify Engineering (opens in new tab)

Spotify’s data assistant, Vedder, relies less on model size than on carefully curated domain context. With more than 70,000 datasets, schemas alone cannot capture business definitions, data quality issues, or preferred query patterns. Spotify’s solution is a cluster-based context layer owned by domain experts, making AI-generated SQL more reliable, transparent, and maintainable. ## Why Schemas Alone Are Not Enough - Spotify has petabytes of data across more than 70,000 datasets, making it impossible to provide an LLM with the entire warehouse. - Even large context windows cannot represent all available schemas effectively. - Schema types and column names omit critical meaning, such as: - Which values represent test or legacy data - What “active user” means in a particular domain - Which tables or columns are authoritative - Without this context, an AI assistant may confidently choose the wrong dataset. ## Spotify’s Data Agent - Users ask questions in natural language, and the agent: - Selects the relevant context - Generates SQL - Executes it against the warehouse - Returns the answer, query, and sources - It uses a ReAct loop to reason, call tools, inspect results, and revise its approach. - Users can see how an answer was produced rather than receiving an opaque result. - The assistant is available through: - Slack - An MCP server for IDEs and AI tools - A dedicated web interface - Since August 2025, it has supported more than 2,100 users, 13,000 conversations, and 60,000 messages across 177 domain clusters. ## The Cluster Model Spotify organizes data domains into “clusters,” each owned by a named team of experts. A cluster contains: - **Datasets** - Relevant warehouse tables with schemas and profiling - Column cardinality, common values, and partition information - Details that help the model construct accurate filters and queries - **Pairs** - Expert-approved natural-language questions paired with SQL - Examples of both query patterns and domain semantics - **Docs** - Business terminology and definitions - Known data pitfalls - Guidance about which columns to use or avoid Clusters can represent organizations, initiatives, or specialized areas of interest. Domain experts decide what belongs in each cluster and which examples best represent correct practice. ## Why Human Curation Matters - Spotify considered automatically generating training pairs from historical query logs. - That approach produced unreliable results because query history contains: - Exploratory analysis - Debugging queries - One-off investigations - Incorrect table choices - Technically valid but misleading patterns - Cluster curators accepted only 12.5% of the proposed question-SQL pairs. - Experts therefore determine what is canonical and trustworthy, while the model uses that curated knowledge to answer more users. - The goal is not to replace data specialists, but to scale their judgment and expertise. ## Keeping Context Current - Data models and business logic change continuously. - Cluster health scores monitor signals such as: - Underlying data quality - Whether curated SQL still works after schema changes - Coverage of users’ real questions - Reproducibility of generated SQL - Renamed columns or deprecated tables can immediately reduce the validity of existing examples. - Cluster owners use health dashboards and recommended actions to prioritize maintenance. ## Learning from Every Conversation - Vedder records conversations, queries, answers, generated SQL, and user feedback. - Cluster owners use this information to identify missing documentation, weak examples, and emerging needs. - Each approved example or clarified definition improves future answers. - The system treats context as an ongoing product that requires ownership and maintenance, not a one-time upload of metadata. Spotify’s approach suggests that trustworthy enterprise AI depends on a maintained context layer: curated datasets, expert-approved examples, clear documentation, and continuous feedback. The model supplies reasoning and automation, but domain experts remain responsible for defining what the data means.

aillmmodel-context-protocolslack+4

spotifyMay 18, 2026

When Can LLMs Replace Humans in A/B Tests? | Spotify Engineering (opens in new tab)

LLMs can approximate human outcomes in A/B tests only when strong assumptions hold; unlike randomized user experiments, those assumptions are not guaranteed by design. In the Upworthy headline dataset, raw GPT-4o-mini predictions recovered just 39% of the human treatment effect, but appropriate calibration and repeated sampling substantially improved accuracy. However, the assumptions are hardest to justify for genuinely new products or interventions—the cases where replacing human tests would be most valuable. ## Raw LLM Predictions Underestimate Treatment Effects - Researchers used the Upworthy Research Archive, containing click-through rates from thousands of headline experiments. - GPT-4o-mini predicted click-through rates for treatment and control headlines. - Treating those predictions as human outcomes recovered only 39% of the observed human treatment effect. - The error was systematic rather than random: LLM predictions attenuated effects toward zero. - This could lead companies to underestimate product improvements and make poor shipping decisions. ## Conditions for Valid LLM Surrogates Two assumptions are required: - **Surrogacy:** LLM predictions must capture everything about a treatment that affects human behavior. Once predictions and relevant pre-treatment characteristics are accounted for, treatment assignment should provide no additional information about the human outcome. - **Comparability:** The relationship between LLM predictions and human behavior—the calibration function—must remain stable between historical experiments and the new experiment. - If either condition fails, more LLM samples will not solve the problem; the procedure estimates an effect on the model rather than the effect on users. ## Calibration Methods Matter - Linear calibration with ordinary least squares failed a falsification test, producing estimates 3.8 standard errors away from the human benchmark. - Random forests and gradient-boosted trees performed better because they could model nonlinear relationships between LLM predictions and human click behavior. - Repeatedly sampling the LLM and averaging its outputs reduces temperature-driven noise, lowering bias toward zero and reducing variance. ## Limits for New Interventions - Historical data can partially test surrogacy and comparability, but neither assumption can be verified for a treatment never previously tested. - Trust declines as a new treatment becomes more different from past experiments. - The Upworthy setting is unusually favorable: binary outcomes, text-only treatments, similar headline variants, and an LLM trained on extensive textual data. - These assumptions are much less plausible for changes to layouts, recommendation algorithms, pricing, or entirely new product concepts. Human A/B tests remain essential for genuine product innovation. LLM-based testing is most defensible for incremental changes that resemble well-understood historical treatments, with human experiments used to validate the approach and calibrate its predictions.

llmmachine-learningab-testingcausal-inference+4

spotifyMay 18, 2026

Coding Is No Longer the Constraint: Scaling Developer Experience to Teams and Agents at Spotify | Spotify Engineering (opens in new tab)

Spotify argues that AI has shifted software development’s main constraint from writing code to coordinating people, systems, and decisions. Years of investment in standardized platforms, automation, and developer experience enabled Spotify to adopt AI coding tools at extraordinary scale. The company’s experience suggests that consistent infrastructure and strong feedback loops are essential for making both human developers and coding agents effective. ## Rapid AI Adoption - More than 99% of Spotify engineers use AI coding tools weekly. - 94% report improved productivity. - Pull request frequency has increased by 76%, with most PRs created by developers working alongside AI agents. - Adoption accelerated sharply after the release of Claude Opus 4.5. ## Fleet Management Before AI Agents - Spotify’s codebase was growing seven times faster than its engineering workforce. - Developers increasingly spent time on dependency upgrades, API migrations, and vulnerability fixes. - Fleet Management automated changes across hundreds or thousands of components. - Its orchestration system, Fleetshift, has merged more than 2.5 million maintenance PRs, most without human intervention. - This approach reduced migrations from work taking weeks or months across many teams to centrally managed operations. ## Honk: A Background Coding Agent - Deterministic scripts struggled with complex refactoring and the edge cases found across large codebases. - Spotify created Honk, a background coding agent powered by Claude through the Agent SDK. - Honk runs in Kubernetes pods, allowing many coding sessions to execute concurrently. - It can use trusted tools and run builds in CI across multiple operating systems. - Fleetshift identifies targets, schedules work, and tracks PRs, while Honk performs the code changes. - A recent Java migration across Spotify’s backend services took three days. - Engineers can invoke Honk through Slack, where it uses conversation context to create and return PRs. - Honk v2 adds shared sessions, team projects, and agent orchestration through Chirp. ## Standardization Improves Agent Performance - Spotify’s principle of limiting the number of technologies it supports reduces decisions and improves collaboration. - Consistent service architectures and design patterns also give AI agents better reference material. - Agents perform worse in fragmented codebases with inconsistent conventions. - Backstage provides a unified internal developer portal and catalog for software components. - Spotify exposes Backstage capabilities to agents through MCP integrations and command-line tools. - Agents can discover component ownership, read documentation, and contact responsible teams. ## Guardrails Through Backstage - Backstage’s Soundcheck and “golden state” define recommended technologies and practices. - Teams can assess their components against these standards. - Static analysis and linting provide immediate feedback when developers or agents use unsuitable patterns. - This creates a feedback loop that helps agents correct their work and drives consistency across the organization. Spotify’s experience indicates that scaling AI development requires more than giving engineers access to models. Organizations should invest in standardized platforms, searchable component metadata, automated fleet-wide workflows, and strong validation systems so agents can operate reliably at team scale.

k8sai-agentslackclaude-code+4

spotifyApr 22, 2026

Better Experiments with LLM Evals — A funnel, not a fork | Spotify Engineering (opens in new tab)

LLM evaluations and A/B experiments serve different purposes and should work as a funnel rather than alternatives. Evals verify whether a change meets qualitative standards and filter out weak ideas, while experiments validate whether it improves real user and business outcomes. Their greatest value comes from a feedback loop that continuously calibrates eval scores against online results. ## Evals as Verification, Experiments as Validation - LLM judges can assess relevance, coherence, tone, intent alignment, and other qualities at scale. - Evals identify promising candidates before they consume experiment capacity. - Experiments determine whether users actually respond better to the change. - An eval can uncover unexpected failure patterns, guide product fixes, and later verify that violations have decreased. - It cannot establish whether those fixes improve long-term outcomes such as engagement, trust, or retention. ## The Importance of Guardrails - Evals typically measure selected quality dimensions, not every possible business risk. - Spotify reports that roughly 42% of launched experiments are rolled back because of regressions in secondary metrics such as crashes, session length, or retention. - Online experiments expose production effects and monitor guardrail metrics that offline evals may miss. - Although only about 12% of Spotify’s A/B tests produce a shipped positive result, approximately 64% generate useful learning. ## Calibrating Evals Against Outcomes - Evals are proxies, so their scores must be validated against the outcomes they are intended to predict. - LLM judges introduce an additional calibration layer on top of conventional metrics such as precision, recall, and ranking scores. - Judges can reward superficial patterns or miss improvements that matter in real-world use. For example, Qodo’s coding evals did not detect improvements in Anthropic’s Opus 4.5 on longer tasks. - Both judges and quantitative metrics can drift, making offline-to-online comparison essential. - Continuous calibration clarifies when evals are reliable verification tools and whether they may eventually become strong enough to support validation. ## Closing the Evaluation Loop - Run evals early and frequently to select the strongest treatments. - Use experiments to confirm user impact, assess business outcomes, and detect unintended regressions. - Apply the same LLM evals to A/B test data to compare qualitative scores with actual user results. - A mismatch between eval scores and experiment outcomes is valuable diagnostic information: - Improved scores and improved outcomes validate the judge. - Improved scores without better outcomes show that the judge is measuring something that may not drive value. - Use lightweight experiments for iteration and learning, and more rigorous tests for launch decisions. The practical recommendation is to treat LLM evals as an upstream filter and a downstream calibration tool—not a replacement for experimentation. This funnel improves test efficiency while making both the judges and product decisions more trustworthy over time.

machine-learninga-b-testingllm-evalsonline-experiments+3

spotifyApr 22, 2026

Building a Natural Language Interface to the Spotify Ads API with Claude Code Plugins | Spotify Engineering (opens in new tab)

The post describes an open-source Claude Code plugin that lets users manage Spotify advertising campaigns through natural-language requests. It translates high-level intent into validated, multi-step Spotify Ads API calls, handling targeting lookups, budget conversions, entity dependencies, and missing information. The authors favor a Markdown- and CLI-based design over MCP because it keeps the system transparent, lightweight, customizable, and grounded in Spotify’s OpenAPI specification. ## Natural-Language Campaign Creation - Users can request campaigns conversationally, such as creating an audio campaign targeting a specific age range and location with a daily budget. - The agent orchestrates the campaign lifecycle: - Creates the campaign. - Creates an ad set with targeting and budget. - Creates the ad and attaches creative assets. - It handles implementation details automatically: - Looks up geographic targeting IDs. - Converts dollar amounts into API micro-units. - Validates audience size. - Passes entity IDs between dependent API calls. - Prompts for missing required information. ## Claude Code Plugin Architecture - **Skills** provide slash commands, with each command defined in Markdown alongside its endpoints, request formats, and output behavior. - **Agents** interpret freeform requests and decompose them into the necessary API operations. - **Hooks** intercept tool calls to refresh OAuth tokens and inject HTTP headers. - **Settings** store local user configuration, including credentials, ad accounts, and environment preferences. - Because all components are human-readable Markdown, the plugin has no compilation, bundling, or package-management step. - API behavior can often be corrected by updating documentation or instructions rather than changing compiled code. ## CLI and OpenAPI Instead of MCP - The authors avoided MCP because the Spotify Ads API has more than 30 resource types and complex nested schemas. - Defining every endpoint as an MCP tool would create a large static registry and consume context even when most tools were irrelevant. - The plugin loads only the API documentation needed for a particular task. - API calls are issued as visible `curl` commands, allowing users to inspect, copy, modify, and reproduce them. - Spotify’s roughly 8,600-line OpenAPI v3 specification serves as the single source of truth. - Updating one bundled specification file is simpler than maintaining a separate translation into MCP schemas. ## Domain-Specific Agent Behavior - The request-builder agent is defined in `agents/spotify-ads-request-builder.md`. - It teaches the model Spotify-specific conversions, including: - Dollar values to micro-amounts. - Natural-language dates to ISO 8601. - Platform names to API enum values. - It performs multi-step orchestration for campaign, ad set, and ad creation. - It resolves locations such as “Connecticut” through geo-targeting search endpoints and builds the required `geo_targets` structure. - It performs pre-flight audience estimates to ensure targeting meets minimum size requirements. - It is designed to control execution carefully and validate requests before making changes that could affect advertising budgets. ## Practical Implication The plugin demonstrates that a large advertising API can be made approachable without hiding its mechanics. A Markdown-based Claude Code integration, backed by the official OpenAPI specification and transparent CLI requests, offers a practical balance between natural-language convenience, developer control, auditability, and maintainability.

database-designpythonnlpclaude-code+4

spotifyApr 6, 2026

Background Coding Agents: Supercharging Downstream Consumer Dataset Migrations (Honk, Part 4) | Spotify Engineering (opens in new tab)

Spotify used its Honk background coding agent with Backstage and Fleet Management to automate migrations from two deprecated datasets to new versions. The effort targeted roughly 1,800 downstream pipelines and produced 240 automated pull requests, potentially saving about 10 engineering weeks. The experience showed that agents perform best when repositories follow standardized patterns, prompts contain precise technical context, and automated testing is available. ## The Challenge of Large-Scale Dataset Migrations - Two heavily used datasets needed replacement to support new dimensions and features. - The datasets had approximately 1,800 direct downstream pipelines and affected thousands more indirectly. - Migrations spanned three frameworks: - BigQuery Runner - dbt - Scala-based Scio - Manual migration was estimated to require around 10 engineering weeks within a six-month deadline. ## Using Backstage to Identify Consumers - Backstage’s endpoint lineage pages revealed downstream dataset consumers. - Its Codesearch plugin located relevant repositories across Spotify’s GitHub Enterprise environment. - The Fleetshift plugin used those results to organize and orchestrate repository migrations. - Backstage also provided a centralized view for tracking progress and opening generated pull requests. ## Context Engineering for Honk - Honk needed detailed, self-contained prompts because it could not access external documentation, dataset schemas, MCPs, or custom Claude skills during execution. - Scio was excluded because its flexible, inconsistent implementations made it difficult to describe all migration cases in one reliable prompt. - BigQuery Runner and dbt were more standardized, making them better candidates for automation. - An initial prompt based on a human migration guide was insufficient and caused incorrect assumptions about field mappings. - Explicit mapping tables in the context file significantly improved results. - Prompts also specified cases where fields should not be migrated automatically. - Honk left those fields unchanged. - It added comments linking to human migration guidance for later review. ## Testing and Automated Pull Requests - BigQuery Runner and dbt repositories generally lacked build-time unit tests. - As a result, Honk could not automatically verify and correct its changes, one of its key capabilities. - Downstream teams had to manually test the generated pull requests before merging. - Despite this limitation, the team successfully created 240 automated migration PRs. - Fleetshift’s Backstage interface simplified monitoring, troubleshooting, repository navigation, and communication with owning teams. ## Lessons for Future Agent-Driven Maintenance - Large-scale automation depends on standardizing frameworks and data practices across repositories. - Consistent testing and validation requirements are essential so agents can verify their own changes. - Future Honk functionality will allow agents to gather context from sources such as JIRA tickets and documentation before editing code. - Better context gathering should reduce the need for exhaustive prompt files and improve migration quality. Spotify’s experience suggests that background coding agents can substantially reduce migration toil, but their effectiveness depends on disciplined standardization, explicit migration rules, and strong automated testing.

database-designdata-pipelinebigquerydbt+4

spotifyApr 6, 2026

Indexing the Data Lake for Online Point Queries | Spotify Engineering (opens in new tab)

Random Access Parquet (RAP) enables low-latency point queries directly against massive data lakes. It addresses the mismatch between fast cloud storage and query engines such as Trino or BigQuery, whose planning and scheduling overhead can make single-row lookups take seconds. By indexing keys to exact Parquet files and rows, RAP avoids broad scans and retrieves only the required data while preserving a shared source of truth for analytics, ML, and online services. ## Why Conventional Query Engines Struggle - Online applications and AI agents need fast access to per-user histories, often stored across billions of records. - Keeping all data in systems such as Bigtable or DynamoDB is prohibitively expensive when the lake contains exabytes. - Object storage latency is increasingly suitable for online workloads: - GCS requests typically take 30–100 ms. - S3 Express One Zone and GCS Rapid Storage can offer single-digit millisecond latency. - Distributed SQL engines introduce seconds of scheduling and planning overhead, making them better suited to analytical throughput than point lookups. ## Narrowing the Search Still Leaves a Problem - A 90-day query over daily listening data could involve approximately 90,000 Parquet files. - Key-based partitioning can reduce candidates substantially; with 1,000 buckets per day, the set may fall to 90 files. - Bloom filters can eliminate files that do not contain the requested user, potentially reducing the set to around 12 files. - The remaining files still require multiple dependent reads: - Fetch the Parquet footer. - Parse row-group metadata. - Scan the key column. - Locate relevant pages through column and page indexes. - Read the corresponding value pages. - These sequential, dependent requests consume both latency and cloud-storage bandwidth. ## The RAP Approach - RAP replaces scanning with direct lookup. - An external index maps each key to: - The relevant Parquet file. - The row numbers containing the key. - Optionally, the number of values for pagination. - Cached metadata maps row numbers to page locations. - The reader then issues precise ranged reads for the required pages. - Reads can be performed in parallel because they no longer depend on a chain of discovery operations. - The approach benefits cloud storage, SSDs, and memory because it removes dependent reads at every tier. ## The External Index - RAP works with existing, unmodified Parquet files. - An index builder reads file footers and page locations, scans key columns, and writes key-to-location mappings. - New pipeline output adds index fragments rather than modifying existing index data. - The index is a multimap, allowing one key to occur in multiple files and partitions. - Index size is typically much smaller than the data: - Terabytes of data may produce gigabytes of index. - Petabytes of data may produce terabytes of index. - Large indexes can be distributed using hash bucketing. - Unlike Parquet Bloom filters and PageIndex structures, the external index is definitive: it identifies exact files and rows instead of merely narrowing a scan. - With unmodified files, RAP may still need to read an entire page—for example, several megabytes to retrieve a small value—so write-time preparation can further reduce read cost. ## Preparing Parquet for Faster Point Reads - Once the reader knows the exact key, row, and columns, file layout can prioritize smaller and fewer final reads rather than in-file discovery. - Relevant optimizations fall into three broad categories: - Concentrating a key’s data. - Reducing bytes per read. - Reducing the number of reads. - Sorting by key places a key’s rows together, minimizing the number of pages required. - Hash bucketing deterministically places each key in one file per partition. - Co-grouping can store one row per key with values in repeated or nested structures, such as an array of timestamp, track URI, and duration fields. - Coarser partitioning can reduce how many files a key spans. RAP therefore provides a way to serve interactive point queries from the same Parquet data used for batch analytics, reducing duplication and avoiding specialized serving copies. For latency-sensitive workloads, indexing should be combined with write-time layout decisions that cluster keys and minimize page reads.

apache-parquetdata-lakerandom-access-parquetpoint-queries+3

spotifyMar 12, 2026

Let’s Talk Agentic Development: Spotify x Anthropic Live | Spotify Engineering (opens in new tab)

Agentic development is reshaping software engineering at Spotify and Anthropic, from how developers write code to how organizations manage delivery. The discussion highlights Claude-powered agents, enterprise-scale context management, and the need to rethink testing, review, and accountability. The speakers conclude that agents will soon handle more of the full software lifecycle, including maintenance and deletion. ## The Opus 4.5 Inflection Point - Spotify observed a sharp increase in agent-driven development after Opus 4.5 went online on November 25, 2025. - Engineers increasingly shifted from working primarily in IDEs to using terminals and agent-based workflows. - The change was presented as a practical transformation in daily engineering work, not merely an experimental trend. ## Honk: Spotify’s Background Coding Agent - Spotify employees can invoke Honk by mentioning it in Slack. - Honk evolved from deterministic code migrations into a Slack-native agent capable of complex migrations across thousands of repositories. - Teams can discuss a problem in Slack and ask Honk to investigate or implement a solution directly. - Spotify is continuing to explore how background coding agents can operate at larger scale. ## Context Engineering and Control - Scaling agents across many repositories requires consistent, reproducible configuration. - Anthropic recommends well-structured `CLAUDE.md` files and reusable skills that describe engineering roles, domains, and expected workflows. - The emphasis is on simple, standardized context rather than overly complex orchestration. - Both companies are still identifying gaps in how agents receive context and how their actions are coordinated across enterprise systems. ## Testing, Reviews, and Accountability - Agent-generated code can be produced faster than humans can review it, creating new bottlenecks. - Organizations must reconsider testing, governance, and approval processes as output volume increases. - Accountability should remain tied to the outcome, regardless of whether code was produced by a human or an agent. - The discussion frames agent adoption as an organizational change, not just a tooling upgrade. ## The Next Stage of Agentic Development - The current phase has focused largely on code creation; the next phase will expand into maintenance, deletion, and other less popular but essential engineering work. - Spotify is evolving Backstage from a human-oriented developer portal into an agent-first platform. - MCP connections are expected to replace more manual developer workflows. - Anthropic’s internal “ant-fooding” practice continues to generate product ideas from employees using its own tools, including Claude Code and Cowork. Organizations adopting agentic development should start with reliable feedback loops, standardized context, and clear human accountability. The most significant gains will come when agents are integrated across the entire software lifecycle rather than used only for writing new code.

slackclaudecontext-engineeringai-infrastructure+4

spotifyFeb 19, 2026

Congratulations to the recipients of the 2025 Spotify FOSS Fund | Spotify Engineering (opens in new tab)

Spotify’s 2025 FOSS Fund provided €30,000 to FFmpeg, €15,000 to Mock Service Worker (MSW), and support to the Xiph.Org Foundation. The grants recognize both large, foundational projects and smaller efforts maintained by individuals, emphasizing that Spotify’s technology depends on volunteer-driven open source. Maintainers describe the funds as valuable for development, infrastructure, contributor support, and long-term sustainability. ## Spotify’s FOSS Fund - Established in 2022 to support open source projects that Spotify relies on and its developers value. - The 2025 recipients were: - FFmpeg - Mock Service Worker (MSW) - Xiph.Org Foundation - FFmpeg and Xiph.Org support technologies fundamental to streaming, while MSW is primarily maintained by its creator. - Despite differences in scale, all three projects depend heavily on volunteer contributions. ## FFmpeg: €30,000 - FFmpeg has been actively developed for more than 25 years and supports: - Multimedia encoding and decoding - Transcoding - Muxing and demuxing - Filtering - Streaming and playback - Its goal is to play every multimedia file ever created, making it widely used by hobbyists and companies such as YouTube, Netflix, X, and Spotify. - Maintainer Kieran Kunhya said the project is written almost entirely by volunteers. - The funding will support: - Hardware purchases - Conference travel - Development projects - FFmpeg’s maintainers argue that company donations also raise awareness of the project’s importance and demonstrate responsibility toward critical dependencies. - They identify reliable, long-term funding and dedicated company developers as major unmet needs in open source sustainability. ## Mock Service Worker: €15,000 - MSW is a JavaScript library for API mocking used by Spotify for unit testing and across its web stack. - Creator Artem Zakharchenko is the project’s sole full-time maintainer, supported by a small group of active contributors. - Major 2025 improvements included: - A new Interceptors architecture - Custom mock agents operating at the Node.js `node:net` level - First-class Server-Sent Events support - Planned work for 2026 includes: - A redesigned internal architecture - A new approach to request interception - Remote request interception, a technically difficult feature under development for several years - The previous Spotify grant primarily helped Artem support himself while working full time on open source. - The new funding will directly support continued MSW development and efforts to make external contributions more financially sustainable. - Artem notes that many contributors have been reluctant to accept payment, creating an ongoing challenge in building a sustainable maintainer and contributor community. ## Xiph.Org Foundation - The post identifies the Xiph.Org Foundation as a 2025 recipient and groups it with FFmpeg as a project representing technology fundamental to streaming. - The provided text does not include further details about the foundation’s grant amount, projects, or maintainer comments. Spotify’s grants illustrate how companies can support both major infrastructure projects and individual maintainers. Recurring funding, paid development time, contributor compensation, and practical resources such as hardware and conference travel remain important to the long-term health of open source.

typescriptjavascriptunit-testingstreaming+4

spotifyFeb 19, 2026

Why We Use Separate Tech Stacks for Personalization and Experimentation | Spotify Engineering (opens in new tab)

Personalization and experimentation overlap, especially with contextual bandits, but they serve different purposes. Personalization chooses the best experience for each user, while experimentation evaluates whether a system or product change improves outcomes overall. Spotify therefore keeps personalization in its ML stack and uses its experimentation platform, Confidence, to measure the impact of those systems. ## What Personalization Does - Personalization adapts products to individual users instead of optimizing for an “average” user. - Spotify uses it for: - Personalized playlists and discovery recommendations - Search results based on listening and search history - Home-screen shortcuts tailored to user behavior - Recommendation systems use models ranging from regression and random forests to neural networks, LLMs, and reinforcement learning. - These systems combine user characteristics, historical behavior, and real-time context to select recommendations and improve them from user responses. ## Where Experimentation and Personalization Overlap - Traditional A/B tests randomly assign users to variants and identify the best option on average. - Multi-armed bandits adapt traffic allocation, sending more users to better-performing treatments. - Contextual bandits use user features to select different treatments for different users. - This creates a conceptual transition: - A/B tests seek the best average treatment. - Multi-armed bandits efficiently find the best average treatment. - Contextual bandits seek the best treatment for each user or context. - Once contextual information determines treatment assignment, the goal is no longer simply measuring one variant’s average effect. - The relevant comparison becomes the value of the personalization system versus a static treatment or an earlier personalization system. ## Why Contextual Bandits Are Not Experimentation Platforms - A contextual bandit can personalize a checkout button based on factors such as cart contents, age, and location. - The bandit is itself a product feature or recommendation system, not the experiment used to evaluate it. - Teams still need experiments to compare different versions of the bandit and determine whether personalization improves user outcomes. - Measuring average or conditional treatment effects may be less important than evaluating the overall benefit of the personalization system. ## Why Spotify Separates the Stacks - Personalization systems require specialized ML infrastructure for: - Training diverse model types - Managing rich feature sets - Serving models with low latency - Collecting real-time data - Computing recommendations at request time - Putting these capabilities into experimentation tools would either greatly expand their scope or limit the sophistication of personalization. - Combining unrelated responsibilities can create technical debt and operational complexity. - Spotify’s ML platform standardizes the development and deployment of personalization systems. - Its experimentation platform, Confidence, evaluates those systems alongside thousands of other product experiments. ## How the Systems Work Together - Personalization algorithms, including contextual bandits, remain in the ML stack. - The experimentation stack treats a personalization system as a treatment to be evaluated. - This separation lets teams improve recommendation logic independently while using controlled experiments to measure its product impact. - It also avoids confusing dependencies that arise when a bandit is both the experiment mechanism and the system being evaluated. Spotify’s approach is to separate building personalized experiences from measuring their effectiveness. Teams should use ML infrastructure for model development and real-time decision-making, then use an experimentation platform to compare personalization systems and quantify their impact.

machine-learningrecommendation-systemsreinforcement-learningab-testing+4

spotifyFeb 19, 2026

Background Coding Agents: Predictable Results Through Strong Feedback Loops (Honk, Part 3) | Spotify Engineering (opens in new tab)

Spotify argues that unsupervised coding agents become reliable only when surrounded by strong, automated feedback loops. Its “Honk” system uses component-specific verifiers, mandatory pre-PR checks, and an LLM judge to catch build failures, test failures, scope creep, and functionally incorrect changes. The conclusion is that constrained, sandboxed agents with rich verification are more predictable than flexible agents operating independently. ## Failure Modes at Scale - Agents may fail to produce a pull request, which is inconvenient but usually manageable. - They may produce PRs that fail CI, leaving engineers to repair incomplete work. - Most seriously, they may produce PRs that pass CI but are functionally wrong and potentially reach production. - These failures are more likely when components lack tests, agents modify code beyond the prompt, or agents cannot correctly run builds and tests. - Reviewing invalid or nonsensical PRs can become a significant engineering time sink. ## Verification Loops - Honk uses independent verifiers that provide incremental feedback while the agent works. - Verifiers activate automatically based on the repository contents; for example, a Maven verifier runs when a root-level `pom.xml` is present. - The agent sees an abstract MCP tool rather than the implementation details of Maven, test runners, or build systems. - Verifiers handle formatting, compilation, testing, and output parsing, returning concise error messages instead of consuming the agent’s context with raw logs. - All applicable verifiers run before a PR is opened. In Claude Code, this is enforced with a stop hook. - If verification fails, the PR is blocked and the user receives an error. ## An LLM as a Judge - Deterministic checks cannot detect every problem, especially when an agent makes unnecessary refactors or disables flaky tests. - Honk therefore sends the original prompt and proposed diff to a separate LLM judge. - The judge runs after the regular verifiers and can veto changes that exceed the requested scope. - Across thousands of sessions, the judge rejects roughly one quarter of proposed changes. - Agents successfully correct about half of the vetoed changes. - Spotify has not yet built formal evaluations for the judge, but observed that scope violations are its most common reason for rejection. ## Constrained Agents and Sandboxing - The agent has limited responsibilities: inspect the relevant code, edit files, and invoke verification tools. - Surrounding infrastructure handles prompt creation, pushing code, and user communication through systems such as Slack. - Restricting the agent’s capabilities improves predictability and provides security benefits. - Agents run in heavily sandboxed containers with limited permissions, few installed binaries, and almost no access to surrounding systems. - Spotify reports that agents solve increasingly complex tasks reliably when these feedback loops are present, but often produce unusable code without them. ## Future Expansion - Spotify plans to support more hardware and operating systems. - Current verifiers run only on Linux x86, limiting support for systems that require macOS, such as iOS applications, or ARM64 environments. - The company also intends to integrate Honk more deeply with existing CI/CD pipelines. The practical recommendation is to treat autonomous coding as an infrastructure and verification problem, not merely a prompting problem: keep agents narrowly scoped, isolate them securely, and require layered automated checks before accepting their changes.

llmai-agentmodel-context-protocolclaude-code+4

spotifyFeb 19, 2026

Inside the Archive: The Tech Behind Your 2025 Wrapped Highlights | Spotify Engineering (opens in new tab)

Spotify’s 2025 Wrapped Archive identified up to five remarkable listening days for each eligible user and turned them into personalized, LLM-generated stories. A distributed pipeline, carefully designed prompts, model distillation, and massive-scale pre-generation made it possible to create roughly 1.4 billion reports before launch. The system prioritized factual grounding, creative consistency, safety, and reliable parallel storage. ## Identifying Remarkable Listening Days - Spotify used a priority-ordered set of heuristics to evaluate each user’s full year of listening. - Straightforward categories included: - Biggest Music Listening Day - Biggest Podcast Listening Day - Biggest Discovery Day, based on first-time artists - Biggest Top Artist Day - Biggest Top Genre Day - More nuanced categories detected: - Nostalgic listening and throwback-heavy sessions - Unusual listening patterns that differed from a user’s typical taste - Contextual dates such as birthdays and New Year’s Day - Candidate days were ranked by narrative potential and statistical strength, reducing hundreds of millions of events to as many as five standout days per user. - A distributed data pipeline aggregated the results and stored listening data in object storage. - Messaging queues then moved each user’s data asynchronously into report generation. ## Prompt Engineering for Reliable Stories - Spotify spent more than three months iterating on prompts and evaluating edge cases. - The system prompt established: - Traceability to real listening behavior - A witty, sincere, and quietly playful tone - Safety constraints excluding references to drugs, alcohol, sex, violence, and offensive language - User prompts supplied: - Detailed daily listening logs - Precomputed statistics, since LLMs are unreliable at arithmetic - Overall Wrapped data - The remarkable-day category - Previously generated reports to reduce repetition - The user’s country for appropriate spelling and vocabulary - Outputs were improved through prototype comparisons, LLM-based judging, human review, and feedback from creative, technical, and safety teams. ## Distilling the Model for Scale - Larger frontier models produced strong results during prototyping but were too expensive for more than a billion generations. - Spotify generated high-quality reference outputs and curated them into a reviewed “gold” dataset. - A smaller, faster production model was fine-tuned on that dataset. - Direct Preference Optimization (DPO), based on curated human A/B evaluations, further aligned the smaller model with the preferred output style. - The resulting model achieved preference performance comparable to the larger baseline. ## Generating 1.4 Billion Reports - Approximately 350 million users were eligible, with up to five reports each. - Spotify pre-generated about 1.4 billion reports before Wrapped launch. - The system sustained thousands of model requests per second over several days. - After remarkable days were computed, snapshots were published to a pub/sub queue. - Reports were generated sequentially per user so earlier reports could inform later ones and prevent repetition. - Real-time dashboards tracked throughput, reliability, errors, and projected completion time. - The generation engine ran continuously for four days, followed by checks for missing reports, inconsistencies, and necessary re-generation. ## Designing Storage for Concurrent Writes - Completed reports were stored in a distributed, column-oriented key-value database optimized for high-throughput writes. - Each user occupied a single row, with separate columns representing completed remarkable days. - Instead of maintaining a serialized list—which could cause race conditions during read-modify-write operations—each date received its own column qualifier in `YYYYMMDD` format. - Independent reports could therefore be written concurrently to separate cells without locks or coordination. - Report content was written first, followed by lightweight metadata marking the report complete. - This ordering prevented the system from exposing a completion marker before the underlying report was safely stored. ## Practical Conclusion Building Wrapped Archive required treating creative AI generation as a large-scale production system: ground outputs in structured data, use smaller specialized models when volume demands it, evaluate continuously, and design storage schemas that make concurrency safe by default.

database-designllmprompt-engineeringhigh-throughput+4

spotifyFeb 19, 2026

How We Release the Spotify App: A Look Under the Hood (Part 2) | Spotify Engineering (opens in new tab)

Spotify’s Release Manager Dashboard replaced a Jira-heavy workflow with a unified command center for mobile and desktop releases. It reduces context switching and cognitive load by aggregating release status, bugs, testing, build health, and usage metrics into one interface. A backend that caches and pre-aggregates data from roughly ten systems makes the dashboard fast and affordable. ## From Jira to a Release Command Center - Previously, Release Managers depended on Jira tickets, multiple browser tabs, and Slack conversations. - This made it easy to miss details and required constant context switching. - The dashboard was designed to: - Prioritize the Release Manager’s workflow. - Remain understandable to anyone familiar with Spotify’s release process. - Reduce cognitive load. - Support fast, accurate decisions. ## Release Data Spotify treats each platform-and-version combination as a **track**. Android, iOS, and Desktop share some libraries but are released independently. - Track-specific information includes: - Current release state. - Release-blocking bugs. - Team regression-testing sign-offs. - Final release candidate build status. - Build verification test results. - App Store upload status. - Crash, ANR, and CPU-exception rates per song. - Daily active users. - The dashboard also highlights: - Blocking bugs without an assigned version. - Bugs without a priority. - Reports from internal users and alpha/beta testers. - Release management includes finding appropriate owners for unassigned issues, even when temporary team ownership is needed. ## React, TypeScript, and Backstage - The dashboard is a Backstage plugin built with React and TypeScript. - Spotify’s Backstage ecosystem already provides: - Software Catalog functionality for distributing builds to app stores. - App-build and crash plugins with deeper detail. - Shared UI components and data across developer tools. - The interface provides a quick health overview, with drill-down capabilities for investigating blockers. - Status colors communicate urgency: - **Green:** Ready for the next stage. - **Yellow:** Something still needs attention. - **Red:** An error requires corrective action. ## Backend Aggregation and Performance - A dedicated backend acts as an API gateway for approximately ten existing systems. - It consolidates their data into one consistent API for the dashboard. - The initial implementation queried large amounts of data on every reload, making it slow and expensive. - Caching and five-minute pre-aggregation reduced load time to about eight seconds while significantly lowering operating costs. ## Dashboard Sections ### Production - Shows the currently deployed Android, iOS, and Desktop versions. - Since these releases have completed the release process, only production metrics are displayed. - Metrics include: - Crash data. - Rolling daily active users over the previous 24 hours. - This helps Release Managers detect problems shortly after rollout. ### Current - Displays the branched version that has not yet reached production. - Tracks release blockers such as: - Open blocking bugs. - Incomplete regression testing. - Crash rates above release thresholds. - Builds that do not contain the latest release-branch commits. - Yellow indicators represent pending work, while red indicators call for direct investigation or action. - The ITGC section confirms that full production rollout is permitted only after: - ITGC tests pass. - Reporting is correct. - Data loss remains below the defined threshold. - A Release Status Ping link generates a Slack update with the release’s current state. ### Upcoming - Mirrors the Current release view for the next planned version. - Sections that are not yet relevant are shown in a grayed-out state. The dashboard illustrates how a specialized aggregation layer and focused UI can turn a fragmented release process into a clear operational workflow. For organizations managing complex, multi-platform releases, combining cached cross-system data with color-coded status and drill-down details can improve both speed and release safety.

ci-cdreacttypescriptjira+4

spotifyFeb 9, 2026

Our Multi-Agent Architecture for Smarter Advertising | Spotify Engineering (opens in new tab)

The post argues that fragmented advertising workflows, not backend infrastructure, are the core problem. Although buying channels share services and data, their planning and optimization logic is repeatedly reimplemented across channels and surfaces, causing drift and technical debt. The proposed solution is a shared agentic decision layer that interprets advertiser goals, orchestrates existing Ads APIs, and applies consistent reasoning across products. ## Fragmented Workflows Across a Shared Backend - Direct, Self-Serve, and Programmatic buying use largely consolidated infrastructure but retain different workflows and decision logic. - Spotify Ads Manager, Salesforce, Slack, and internal tools contain overlapping automation. - Budget allocation, inventory selection, reach, efficiency, and STR decisions are repeatedly implemented in different places. - Incremental workflow changes therefore create duplicated maintenance work and inconsistent behavior. ## Why Conventional Workflow Services Fall Short - Hard-coded state machines and REST services are poorly suited to combinatorial planning tasks. - Campaign planning depends on: - User and advertiser characteristics - Available inventory and audiences - Business priorities - Forecasts, performance, and optimization goals - A workflow optimized for one channel or “happy path” will not adapt well as requirements change. - Improvements to decision logic must be replicated across every product surface, increasing the risk of divergence. ## The Missing Intent Layer - Existing systems can perform individual actions such as creating line items, running forecasts, and retrieving insights. - They do not consistently translate high-level objectives into: - A sequence of tool calls - Explicit tradeoffs - Validation and safety checks - An objective such as maximizing reach in Brazil while protecting video inventory and meeting STR requires coordinated reasoning across multiple capabilities. ## A Modular Agentic Architecture - Campaign planning and management are modeled as cooperating specialized agents. - Agents use shared signals, including: - Inventory - Audiences - STR - Quality and risk - Historical performance - They jointly optimize advertiser goals and Spotify’s business constraints. - Existing Ads services become tools that agents orchestrate, rather than capabilities being rebuilt in each workflow. - A long-running orchestration layer delegates tasks while agents share context and evaluation logic. - The same decision engine can support every buying channel and surface. ## Engineering Implications - APIs need to be designed as agent tools, rather than only as CRUD interfaces. - Testing must include behavioral evaluation in addition to unit and integration tests. - Observability should explain what an agent decided and why, not merely track latency and errors. - Safety requires guardrails for semi-autonomous decisions, beyond ordinary input validation. - The approach avoids both duplicated deterministic workflows and a brittle, centralized rules engine for probabilistic, ML-heavy advertising logic. The overall recommendation is to centralize campaign decision-making in a reusable agentic platform while keeping existing services as specialized tools. This should reduce duplicated workflow logic, make improvements consistent across products, and allow advertising workflows to evolve without repeatedly rebuilding them.

machine-learningagentic-aiapi-designmulti-agent-systems+3