Encoding Your Domain Expert: The Context Layer Behind Spotify's Data Assistant | Spotify Engineering (opens in new tab)
Spotify’s data assistant, Vedder, relies less on model size than on carefully curated domain context. With more than 70,000 datasets, schemas alone cannot capture business definitions, data quality issues, or preferred query patterns. Spotify’s solution is a cluster-based context layer owned by domain experts, making AI-generated SQL more reliable, transparent, and maintainable. ## Why Schemas Alone Are Not Enough - Spotify has petabytes of data across more than 70,000 datasets, making it impossible to provide an LLM with the entire warehouse. - Even large context windows cannot represent all available schemas effectively. - Schema types and column names omit critical meaning, such as: - Which values represent test or legacy data - What “active user” means in a particular domain - Which tables or columns are authoritative - Without this context, an AI assistant may confidently choose the wrong dataset. ## Spotify’s Data Agent - Users ask questions in natural language, and the agent: - Selects the relevant context - Generates SQL - Executes it against the warehouse - Returns the answer, query, and sources - It uses a ReAct loop to reason, call tools, inspect results, and revise its approach. - Users can see how an answer was produced rather than receiving an opaque result. - The assistant is available through: - Slack - An MCP server for IDEs and AI tools - A dedicated web interface - Since August 2025, it has supported more than 2,100 users, 13,000 conversations, and 60,000 messages across 177 domain clusters. ## The Cluster Model Spotify organizes data domains into “clusters,” each owned by a named team of experts. A cluster contains: - **Datasets** - Relevant warehouse tables with schemas and profiling - Column cardinality, common values, and partition information - Details that help the model construct accurate filters and queries - **Pairs** - Expert-approved natural-language questions paired with SQL - Examples of both query patterns and domain semantics - **Docs** - Business terminology and definitions - Known data pitfalls - Guidance about which columns to use or avoid Clusters can represent organizations, initiatives, or specialized areas of interest. Domain experts decide what belongs in each cluster and which examples best represent correct practice. ## Why Human Curation Matters - Spotify considered automatically generating training pairs from historical query logs. - That approach produced unreliable results because query history contains: - Exploratory analysis - Debugging queries - One-off investigations - Incorrect table choices - Technically valid but misleading patterns - Cluster curators accepted only 12.5% of the proposed question-SQL pairs. - Experts therefore determine what is canonical and trustworthy, while the model uses that curated knowledge to answer more users. - The goal is not to replace data specialists, but to scale their judgment and expertise. ## Keeping Context Current - Data models and business logic change continuously. - Cluster health scores monitor signals such as: - Underlying data quality - Whether curated SQL still works after schema changes - Coverage of users’ real questions - Reproducibility of generated SQL - Renamed columns or deprecated tables can immediately reduce the validity of existing examples. - Cluster owners use health dashboards and recommended actions to prioritize maintenance. ## Learning from Every Conversation - Vedder records conversations, queries, answers, generated SQL, and user feedback. - Cluster owners use this information to identify missing documentation, weak examples, and emerging needs. - Each approved example or clarified definition improves future answers. - The system treats context as an ongoing product that requires ownership and maintenance, not a one-time upload of metadata. Spotify’s approach suggests that trustworthy enterprise AI depends on a maintained context layer: curated datasets, expert-approved examples, clear documentation, and continuous feedback. The model supplies reasoning and automation, but domain experts remain responsible for defining what the data means.