Dropbox/rag

4 posts

dropbox

Using LLMs to amplify human labeling and improve Dash search relevance (opens in new tab)

Dropbox Dash improves AI answers through retrieval-augmented generation (RAG): enterprise search retrieves relevant company documents, and an LLM uses a small subset of them to generate grounded responses. Because ranking determines which documents reach the LLM, search relevance depends heavily on high-quality query–document labels. Dash combines a small set of human judgments with large-scale LLM-generated labels to produce training data efficiently while retaining human oversight. ## How Dash search ranking works - Dash uses a trained ranking model, such as XGBoost, rather than manually configured rules. - The model learns from query–document pairs labeled on a 1–5 relevance scale: - **5:** Closely matches the user’s intent. - **1:** Not useful enough to display. - Relevance depends on the query, user context, and timing; it is not an intrinsic property of a document. - Ranking quality is especially important because enterprises may have millions or billions of indexed documents, while only a small selection can be sent to the answer-generating LLM. ## Sources of relevance labels - Labels can come from: - User behavior, such as clicks or skipped results. - Human evaluators assigning relevance scores. - LLMs directly judging query–document relevance. - Behavioral signals are useful but often sparse, biased by existing rankings, and unevenly distributed, so they work best as a supplement. - Human evaluators can provide comprehensive judgments across result sets, but labeling is expensive, difficult to scale, and vulnerable to inconsistency. - Humans also cannot directly review sensitive or proprietary customer data in this process, and different content types—such as Slack messages, Jira tickets, and Salesforce records—require different contextual expertise. ## LLM-assisted relevance evaluation - LLMs can evaluate far larger candidate sets at lower cost and with greater consistency than human annotators. - They can operate across languages and analyze customer content within established compliance boundaries. - Their judgments still depend on the model’s quality and the clarity of the evaluation prompt. - LLM-generated labels therefore require calibration and validation before being used for model training. ## Combining human review with LLM scale - Dropbox first creates a relatively small, high-quality dataset using human evaluators and limited, non-sensitive internal data. - These human labels are used to tune LLM prompts and model parameters. - Once the LLM meets quality thresholds, it generates hundreds of thousands or millions of relevance labels. - This approach multiplies human labeling effort by roughly 100 times, enabling broader and more representative training data. - LLMs are used offline rather than directly at query time because production-time use would introduce excessive latency and context-window limitations. - The LLM acts as a teacher for smaller, faster ranking models that can serve searches at scale. ## Evaluation as the foundation - Dash follows an iterative process: measure performance, change the model or instructions, and measure again. - The article compares this to chess engines, where the quality of the evaluation function determines which possible moves are preserved or discarded. - The same principle applies to ranking: poor relevance judgments can cause useful search-result patterns to be eliminated, while accurate judgments guide the model toward better rankings. Dash’s approach uses humans for quality control and contextual grounding, then uses LLMs to expand that expertise into large-scale training data. This hybrid strategy offers a practical way to improve enterprise search relevance without exposing customer data to human reviewers or imposing LLM latency on every search.

dropbox

Engineering VP Josh Clemm on how we use knowledge graphs, MCP, and DSPy in Dash (opens in new tab)

Dropbox VP Josh Clemm argues that useful workplace AI requires a unified context engine capable of securely understanding and retrieving information across many SaaS applications. Dropbox Dash combines custom connectors, multimodal content processing, knowledge graphs, hybrid search, and personalized access-control-aware ranking to make that possible. Clemm favors indexed retrieval over purely federated approaches because preprocessing enables richer context, faster search, and company-wide access, though it requires substantial engineering and infrastructure. ## Building Dash’s Context Engine - Custom connectors crawl third-party applications while handling: - Rate limits - API differences - Application-specific permissions and ACLs - Incoming content is normalized, often into Markdown, and enriched with: - Titles and metadata - Extracted links - Embeddings - Other key information - Different media types require different processing: - Documents can be text-extracted and indexed. - Images may require CLIP or multimodal models. - PDFs combine text, figures, and other elements. - Audio is transcribed. - Videos may require scene-by-scene multimodal analysis when dialogue is insufficient. - Dash models relationships between content as a knowledge graph: - Meetings can connect to documents, participants, transcripts, and previous notes. - Cross-application relationships provide richer context for search and agents. - Data is stored in secure systems using: - BM25 lexical search - Dense-vector storage - Hybrid retrieval - Multiple ranking stages personalize results and enforce user-specific permissions. ## Indexed Retrieval Versus Federated Retrieval - Federated retrieval queries external systems at runtime. - Its advantages include: - Fast initial implementation - Minimal storage requirements - Relatively fresh data - Easy addition of MCP servers and connectors - Its drawbacks include: - Inconsistent API speed, quality, and ranking - Limited access to company-wide content - Expensive post-processing and reranking - Large token usage when agents reason over returned results - Indexed retrieval preprocesses content during ingestion. - Its advantages include: - Access to shared company connectors - Enriched datasets created offline - Faster queries - More opportunities for recall and ranking experiments - Its costs include: - Significant custom connector development - Freshness challenges - High hosting costs - Difficult storage and architecture choices involving vector search, BM25, hybrid retrieval, or graph RAG. ## Making MCP Practical at Scale - MCP can simplify tool integration, but tool definitions consume substantial context-window space. - Large tool descriptions and retrieval results contribute to context rot and reduce agent effectiveness. - Dash aims to limit context usage to roughly 100,000 tokens. - MCP-based agents can also be slow: simple queries may take up to 45 seconds, while direct index retrieval returns results within seconds. - Dropbox’s approach is to wrap its index in a consolidated “super tool,” reducing the need to expose many separate tools to the agent. ## Broader AI Engineering Practices - The talk also covers Dropbox’s use of: - LLMs as evaluators or judges - Prompt optimization with DSPy - Tool calling and MCP design - These techniques complement the underlying context engine rather than replacing the indexing, enrichment, graph modeling, and permission systems required for reliable workplace AI. A practical takeaway is that organizations building AI over proprietary data should treat retrieval as a full data-platform problem. Start with robust connectors and permissions, enrich content before retrieval, model relationships across sources, and use MCP selectively where indexed retrieval can provide faster and more controlled results.

dropbox

How Dash uses context engineering for smarter AI (opens in new tab)

Dash evolved from a traditional RAG search system into an agentic AI that can interpret information, plan tasks, and act on users’ behalf. Dropbox’s experience shows that better agent performance comes not from adding more tools and data, but from carefully engineering context: limiting choices, filtering for relevance, and delegating complex work to specialized agents. The central conclusion is that precise, timely context improves reasoning speed, accuracy, and efficiency. ## From Search to Agentic AI - Dash initially combined semantic and keyword search to retrieve documents and generate concise answers. - Users began asking it to interpret, summarize, and act on retrieved information. - This required Dash to plan and execute multi-step tasks rather than simply search and summarize. - The resulting challenge was determining which information and tools the model actually needed at each stage. ## The Cost of Too Many Tools - Every tool adds descriptions and parameters to the model’s context window. - More tools expand the model’s decision space, potentially causing slower or less reliable choices. - Tool definitions also consume tokens, increasing cost and reducing room for reasoning. - Longer-running tasks suffered from “context rot,” where accumulated tool-call information degraded accuracy. - Model Context Protocol (MCP) standardizes tool descriptions, but does not eliminate the problem of excessive context. ## Limiting Tool Definitions - Dash found that exposing retrieval tools from many services—such as Confluence, Google Docs, and Jira—created confusion. - Instead of requiring the model to choose among numerous APIs, Dash consolidated retrieval into one purpose-built tool backed by its universal search index. - A single retrieval interface: - Simplifies planning - Reduces tool-selection errors - Keeps the context window focused - Provides consistent access across connected services - The same principle shaped Dash’s MCP server, which exposes retrieval through one lean tool to applications such as Claude, Cursor, and Goose. ## Filtering Context for Relevance - Retrieved information is not automatically useful for the task at hand. - Dash combines data from multiple sources in a unified index and uses a knowledge graph to connect people, activity, and content. - These relationships help rank results according to the query and the user’s context. - By filtering results before presenting them to the model, Dash ensures that each piece of supplied context is relevant. - Precomputing the index and graph allows runtime retrieval to remain fast and focused. ## Using Specialized Agents for Complex Tasks - Some tools require substantial instructions and examples to use correctly. - Dash Search became complex because query construction involves: - Understanding user intent - Mapping intent to index fields - Rewriting queries for semantic matching - Handling typos, synonyms, and implicit context - Adding these instructions directly to the main planning agent consumed context that could otherwise support broader reasoning. - Dash therefore moved search into a specialized agent: - The main agent decides when searching is necessary. - The search agent independently constructs the query using its dedicated prompt. - This division lets the main agent focus on the overall task while the specialist handles search details. Dash’s approach recommends treating context as a limited engineering resource. Use a small number of well-designed tools, pre-filter information for relevance, and delegate technically demanding subtasks to specialized agents rather than overwhelming one general-purpose model.

dropbox

A practical blueprint for evaluating conversational AI at scale (opens in new tab)

Conversational AI systems depend on many probabilistic stages, so even small changes can cause unexpected regressions. Dropbox Dash’s experience shows that evaluation should be treated like production engineering: systematic, repeatable, and required before changes are shipped. The approach combines curated datasets, actionable metrics, LLM-based judging, and human review. ## Evaluation as a Development Discipline - AI pipelines include intent classification, retrieval, ranking, prompt construction, inference, and safety filtering. - Changes to any stage can affect final answer quality in unpredictable ways. - Dropbox initially used ad-hoc testing, but shifted to a standardized process in which every model, prompt, or retrieval change had to pass evaluation before merging. - The evaluation framework covers datasets, metrics, tools, and workflows. - Future-proof evaluation must extend beyond text to images, video, and audio. ## Curating Public and Internal Datasets - Public datasets established baseline retrieval and question-answering performance: - **Natural Questions** tested retrieval from very large documents. - **MS MARCO** emphasized queries requiring multiple document hits. - **MuSiQue** tested multi-hop reasoning. - Internal datasets captured real-world usage from anonymized Dropbox employee queries and content. - Representative query sets reflected actual user behavior, using proxy labels or internal annotators. - Representative content sets focused on shared files, documentation, and connected data sources. - LLM-generated synthetic questions and answers covered tables, images, tutorials, and factual lookups. - These datasets became the foundation for automated pass/fail checks and experiment monitoring. ## Why Traditional Metrics Fall Short - Metrics such as BLEU, ROUGE, METEOR, BERTScore, and embedding similarity are fast and useful for detecting major regressions. - They often fail to measure production requirements, including: - Correct source citations - Factual accuracy - Valid file names and references - Reliable table parsing - Clear formatting - High ROUGE or BERTScore can coexist with hallucinations, missing citations, or buried factual errors. - Production evaluation therefore requires metrics tied directly to task requirements. ## Using LLMs as Evaluation Judges - LLM judges can assess dimensions traditional metrics miss, including: - Whether an answer addresses the query - Whether claims are supported by context - Citation correctness - Clarity, formatting, and tone - Judges receive the query, model answer, source context, and sometimes a hidden reference answer. - They return structured scores—scalar or categorical—alongside explanations. - Rubrics and judge models must themselves be tested, calibrated, versioned, and improved. - Specialized judges may be needed for particular languages or technical domains. ## Calibration and Human Review - Periodic manual labeling of sampled outputs created calibration sets for comparing human and judge-model decisions. - The team tracked agreement and judge drift over time. - Engineers manually reviewed 5–10% of each release’s regression suite. - Disagreements were investigated as either judge-prompt defects or model errors. - Recurring problems led to prompt revisions or more detailed scoring criteria. Dropbox’s evaluation-first approach treats AI changes like code changes: test them against realistic datasets, score them with task-specific rubrics, validate the evaluators, and retain human oversight. This makes conversational AI development more reliable as systems and modalities grow more complex.