dropboxJun 25, 2026

How we used DSPy to turn AI evaluations into better responses in Dash chat (opens in new tab)

Dropbox uses DSPy to turn AI evaluations into improvements for its Dash chat agent. The process first calibrates LLM judges against human-labeled conversations, then uses those judges to optimize the agent’s system prompt. This feedback loop reduced incomplete answers and token usage while maintaining answer quality. ## The Complexity of Evaluating AI Agents - Agent quality depends on more than the final response: - Understanding user intent - Selecting relevant context - Choosing and using tools - Synthesizing information across documents, messages, and meetings - Handling ambiguity and follow-up turns - Producing grounded, complete answers - Evaluations therefore inspect the full interaction trajectory, not just the output. - Separate evaluations for intent understanding, tool use, context selection, grounding, adaptation, and task completion help identify the source of failures. - Reliable judges were necessary before evaluation results could safely guide agent improvements. ## Calibrating LLM Judges with Human Labels - Dropbox sampled internal chats containing final answers and agent trace logs. - Human reviewers scored five dimensions: - User-intent following - Semantic relevance - Tool calling - Instruction following - Context selection - Reviewers followed a structured process: - Determine whether the agent understood the request. - Check whether it selected appropriate context. - Inspect searches, retrievals, and other tool actions. - Verify that final claims were supported by evidence. - Score relevance, grounding, completeness, and instruction adherence. - Many metrics used a 1–5 scale. - Reviewers also added: - Reasoning notes explaining their scores - Failure codes for issues such as stale evidence, missing context, unsupported claims, incomplete coverage, and poor personalization - These richer annotations helped improve judge prompts while also supporting debugging, error analysis, roadmap planning, and prioritization. ## Using DSPy to Improve Evaluation - DSPy was used to make LLM judges align more closely with human evaluations. - Judges were required to follow a retrospective workflow: - Infer the user’s intent - Inspect the conversation and agent trace - Review supporting evidence - Assess context selection and tool use - Produce scores, failure codes, and reasoning notes - GEPA and MIPROv2, optimization algorithms within DSPy, automatically proposed and tested prompt changes against human-labeled examples. - Optimization supported several scenarios: - Rewriting judge instructions entirely - Adapting a judge to another underlying model - Targeting specific failure modes while preserving the existing evaluation behavior The overall approach creates a scalable improvement loop: human labels calibrate the judges, calibrated judges provide consistent evaluation signals, and those signals guide improvements to the chat agent itself.

dspyllmagentic-aiprompt-optimization+4

dropboxMar 17, 2026

How we optimized Dash's relevance judge with DSPy (opens in new tab)

Dropbox Dash needed a relevance judge that could score query–document pairs accurately, cheaply, and reliably at scale. Its original judge used OpenAI’s o3, but the cost made it impractical for large-scale labeling, while its prompt performed poorly when moved to the cheaper gpt-oss-120b model. Dropbox used DSPy’s GEPA optimizer to turn prompt tuning into a measurable feedback loop, improving alignment with human judgments while preserving production-ready output formatting. ## Measuring Agreement with Human Reviewers - The judge rates each query–document pair on a 1–5 relevance scale: - **5** means a perfect match. - **1** means no meaningful connection to the query or user intent. - Human annotators provide both: - A relevance score. - A short explanation for their judgment. - Dropbox evaluates the model with normalized mean squared error (NMSE): - It measures the squared difference between model and human ratings. - Scores are normalized to a 0–100 scale. - **0** represents perfect agreement; higher values indicate worse performance. - Invalid JSON or incorrectly structured responses are treated as fully incorrect because they cannot be consumed reliably by downstream systems. - The optimization objective is therefore twofold: - Minimize disagreement with human ratings. - Ensure consistently parseable, production-ready outputs. ## Moving from o3 to a Lower-Cost Model - The original judge used OpenAI’s o3 because it delivered strong agreement with human ratings. - Running o3 across orders of magnitude more query–document pairs was too expensive. - Dropbox selected **gpt-oss-120b**, an open-weight model offering a better cost-performance balance. - The carefully tuned o3 prompt did not transfer directly: - Relevance quality declined under the NMSE metric. - Manual prompt rewriting would have required extensive iteration and regression testing. ## DSPy and GEPA-Based Prompt Optimization - Dropbox defined the optimization problem using: - A fixed relevance-rating task. - Human-annotated examples. - NMSE as the evaluation metric. - DSPy’s **GEPA optimizer** iteratively improves prompts for a specific target model. - Instead of relying only on an aggregate score, GEPA analyzes individual disagreements and generates structured feedback. - Feedback combines: - The difference and direction between predicted and human ratings. - The human annotator’s explanation. - The model’s reasoning. - DSPy then uses a reflection loop: - Evaluate the current prompt. - Identify recurring failure modes. - Revise the prompt with generalizable rules. - Repeat the process against the human-alignment metric. - This approach can address systematic errors such as: - Overvaluing keyword overlap. - Undervaluing document recency. - Misinterpreting user intent. - The feedback explicitly discourages overfitting to individual examples and preserves core task constraints, including the 1–5 rating range. Dropbox’s experience suggests that relevance judges should be optimized systematically rather than tuned manually. Defining a clear human-alignment metric, including structural validity, allows DSPy to adapt prompts across models while reducing cost and limiting regressions.

dspydatabase-designjsonopenai+3

dropboxJan 28, 2026

Engineering VP Josh Clemm on how we use knowledge graphs, MCP, and DSPy in Dash (opens in new tab)

Dropbox VP Josh Clemm argues that useful workplace AI requires a unified context engine capable of securely understanding and retrieving information across many SaaS applications. Dropbox Dash combines custom connectors, multimodal content processing, knowledge graphs, hybrid search, and personalized access-control-aware ranking to make that possible. Clemm favors indexed retrieval over purely federated approaches because preprocessing enables richer context, faster search, and company-wide access, though it requires substantial engineering and infrastructure. ## Building Dash’s Context Engine - Custom connectors crawl third-party applications while handling: - Rate limits - API differences - Application-specific permissions and ACLs - Incoming content is normalized, often into Markdown, and enriched with: - Titles and metadata - Extracted links - Embeddings - Other key information - Different media types require different processing: - Documents can be text-extracted and indexed. - Images may require CLIP or multimodal models. - PDFs combine text, figures, and other elements. - Audio is transcribed. - Videos may require scene-by-scene multimodal analysis when dialogue is insufficient. - Dash models relationships between content as a knowledge graph: - Meetings can connect to documents, participants, transcripts, and previous notes. - Cross-application relationships provide richer context for search and agents. - Data is stored in secure systems using: - BM25 lexical search - Dense-vector storage - Hybrid retrieval - Multiple ranking stages personalize results and enforce user-specific permissions. ## Indexed Retrieval Versus Federated Retrieval - Federated retrieval queries external systems at runtime. - Its advantages include: - Fast initial implementation - Minimal storage requirements - Relatively fresh data - Easy addition of MCP servers and connectors - Its drawbacks include: - Inconsistent API speed, quality, and ranking - Limited access to company-wide content - Expensive post-processing and reranking - Large token usage when agents reason over returned results - Indexed retrieval preprocesses content during ingestion. - Its advantages include: - Access to shared company connectors - Enriched datasets created offline - Faster queries - More opportunities for recall and ranking experiments - Its costs include: - Significant custom connector development - Freshness challenges - High hosting costs - Difficult storage and architecture choices involving vector search, BM25, hybrid retrieval, or graph RAG. ## Making MCP Practical at Scale - MCP can simplify tool integration, but tool definitions consume substantial context-window space. - Large tool descriptions and retrieval results contribute to context rot and reduce agent effectiveness. - Dash aims to limit context usage to roughly 100,000 tokens. - MCP-based agents can also be slow: simple queries may take up to 45 seconds, while direct index retrieval returns results within seconds. - Dropbox’s approach is to wrap its index in a consolidated “super tool,” reducing the need to expose many separate tools to the agent. ## Broader AI Engineering Practices - The talk also covers Dropbox’s use of: - LLMs as evaluators or judges - Prompt optimization with DSPy - Tool calling and MCP design - These techniques complement the underlying context engine rather than replacing the indexing, enrichment, graph modeling, and permission systems required for reliable workplace AI. A practical takeaway is that organizations building AI over proprietary data should treat retrieval as a full data-platform problem. Start with robust connectors and permissions, enrich content before retrieval, model relationships across sources, and use MCP selectively where indexed retrieval can provide faster and more controlled results.

dspymcpragmultimodal-ai+4