AI Agents - Dropbox | Techlist.io

dropboxJun 25, 20262 min readCurated summary

How we used DSPy to turn AI evaluations into better responses in Dash chat

Dropbox uses DSPy to turn AI evaluations into improvements for its Dash chat agent. The process first calibrates LLM judges against human-labeled conversations, then uses those judges to optimize the agent’s system prompt. This feedback loop reduced incomplete answers and token usage while maintaining answer quality. ## The Complexity of Evaluating AI Agents - Agent quality depends on more than the final response: - Understanding user intent - Selecting relevant context - Choosing and using tools - Synthesizing information across documents, messages, and meetings - Handling ambiguity and follow-up turns - Producing grounded, complete answers - Evaluations therefore inspect the full interaction trajectory, not just the output. - Separate evaluations for intent understanding, tool use, context selection, grounding, adaptation, and task completion help identify the source of failures. - Reliable judges were necessary before evaluation results could safely guide agent improvements. ## Calibrating LLM Judges with Human Labels - Dropbox sampled internal chats containing final answers and agent trace logs. - Human reviewers scored five dimensions: - User-intent following - Semantic relevance - Tool calling - Instruction following - Context selection - Reviewers followed a structured process: - Determine whether the agent understood the request. - Check whether it selected appropriate context. - Inspect searches, retrievals, and other tool actions. - Verify that final claims were supported by evidence. - Score relevance, grounding, completeness, and instruction adherence. - Many metrics used a 1–5 scale. - Reviewers also added: - Reasoning notes explaining their scores - Failure codes for issues such as stale evidence, missing context, unsupported claims, incomplete coverage, and poor personalization - These richer annotations helped improve judge prompts while also supporting debugging, error analysis, roadmap planning, and prioritization. ## Using DSPy to Improve Evaluation - DSPy was used to make LLM judges align more closely with human evaluations. - Judges were required to follow a retrospective workflow: - Infer the user’s intent - Inspect the conversation and agent trace - Review supporting evidence - Assess context selection and tool use - Produce scores, failure codes, and reasoning notes - GEPA and MIPROv2, optimization algorithms within DSPy, automatically proposed and tested prompt changes against human-labeled examples. - Optimization supported several scenarios: - Rewriting judge instructions entirely - Adapting a judge to another underlying model - Targeting specific failure modes while preserving the existing evaluation behavior The overall approach creates a scalable improvement loop: human labels calibrate the judges, calibrated judges provide consistent evaluation signals, and those signals guide improvements to the chat agent itself.

Read original

AI Agents Large Language Models Dspy Prompt Optimization+4

dropboxMay 28, 20263 min readCurated summary

Beyond code generation: rethinking engineering productivity in the age of AI agents

AI coding agents have increased Dropbox’s code production, but they have also exposed bottlenecks in review, testing, release coordination, and operations. The central argument is that engineering productivity must be measured by end-to-end product velocity and customer impact—not code or pull-request volume alone. Dropbox is responding with agent platforms, stronger workflow infrastructure, broader quality metrics, and training that helps engineers adapt to new responsibilities. ## From Copilots to Agents - Copilots assist with explanations, snippets, and questions within existing workflows. - Agents can take scoped tasks, inspect repositories, edit files, run tests, fix failures, and return changes for human review. - Engineers remain responsible for intent, architecture, quality, and release decisions. - Increased parallel work and code output create new pressure on: - Code review systems - CI and testing infrastructure - Validation workflows - Release processes - Production operations - More code and pull requests do not necessarily create more customer value. ## Nova as Dropbox’s Agent Platform - Nova runs AI coding agents in controlled environments with relevant codebase context and internal engineering practices. - Its value comes from the surrounding platform—safe execution, workflow integration, guardrails, and human review—not only from the underlying model. - Nova generates approximately one in twelve Dropbox pull requests. - It supports both feature work and maintenance tasks, including: - Migrations - Flaky-test remediation - Bug investigation - Dependency updates - Other high-effort engineering work - The intended workflow is structured: define the task, let the agent work within constraints, validate the result, and require human approval before production. ## Measuring Product Velocity and Impact - Pull-request throughput was useful when implementation speed was the main constraint, but it is insufficient as AI increases output. - Dropbox evaluates whether the broader engineering system can absorb additional work efficiently. - Its four-stage measurement model tracks: - **Fuel:** Whether engineers use AI tools - **Adoption:** How teams change their workflows - **Output:** Whether AI contributes to production work - **Impact:** Whether products reach customers faster and create greater value - Quality and trust metrics include review turnaround time, first-run test pass rate, defect ratio, and rework rate. - Productivity improvements must not come at the expense of reliability or customer trust. ## Evolving Engineering Workflows - Engineers increasingly focus on defining intent, framing problems, reviewing changes, and making architectural and quality decisions. - Adoption requires more than tools; Dropbox uses hands-on learning, hackathons, bootcamps, workflow examples, and peer-led training. - Teams should adopt agents at different speeds based on risk, context, and readiness. - The goal is not to automate every workflow, but to make agentic development safe, useful, measurable, and repeatable. ## Broader Lessons - AI does not remove bottlenecks; it shifts them downstream. - Organizations must invest in validation, orchestration, governance, workflow integration, and measurement alongside code generation. - Competitive advantage will come less from access to common foundation models and more from the systems built around them: internal context, safeguards, quality controls, and integrated engineering workflows. Dropbox’s experience suggests that companies should treat AI agents as a change to the entire software delivery system, not merely as faster coding tools. The strongest results will come from improving the full path from idea to validated production impact.

Read original

AI Agents CI/CD Automation Code Review+3

dropboxNov 17, 20253 min readCurated summary

How Dash uses context engineering for smarter AI

Dash evolved from a traditional RAG search system into an agentic AI that can interpret information, plan tasks, and act on users’ behalf. Dropbox’s experience shows that better agent performance comes not from adding more tools and data, but from carefully engineering context: limiting choices, filtering for relevance, and delegating complex work to specialized agents. The central conclusion is that precise, timely context improves reasoning speed, accuracy, and efficiency. ## From Search to Agentic AI - Dash initially combined semantic and keyword search to retrieve documents and generate concise answers. - Users began asking it to interpret, summarize, and act on retrieved information. - This required Dash to plan and execute multi-step tasks rather than simply search and summarize. - The resulting challenge was determining which information and tools the model actually needed at each stage. ## The Cost of Too Many Tools - Every tool adds descriptions and parameters to the model’s context window. - More tools expand the model’s decision space, potentially causing slower or less reliable choices. - Tool definitions also consume tokens, increasing cost and reducing room for reasoning. - Longer-running tasks suffered from “context rot,” where accumulated tool-call information degraded accuracy. - Model Context Protocol (MCP) standardizes tool descriptions, but does not eliminate the problem of excessive context. ## Limiting Tool Definitions - Dash found that exposing retrieval tools from many services—such as Confluence, Google Docs, and Jira—created confusion. - Instead of requiring the model to choose among numerous APIs, Dash consolidated retrieval into one purpose-built tool backed by its universal search index. - A single retrieval interface: - Simplifies planning - Reduces tool-selection errors - Keeps the context window focused - Provides consistent access across connected services - The same principle shaped Dash’s MCP server, which exposes retrieval through one lean tool to applications such as Claude, Cursor, and Goose. ## Filtering Context for Relevance - Retrieved information is not automatically useful for the task at hand. - Dash combines data from multiple sources in a unified index and uses a knowledge graph to connect people, activity, and content. - These relationships help rank results according to the query and the user’s context. - By filtering results before presenting them to the model, Dash ensures that each piece of supplied context is relevant. - Precomputing the index and graph allows runtime retrieval to remain fast and focused. ## Using Specialized Agents for Complex Tasks - Some tools require substantial instructions and examples to use correctly. - Dash Search became complex because query construction involves: - Understanding user intent - Mapping intent to index fields - Rewriting queries for semantic matching - Handling typos, synonyms, and implicit context - Adding these instructions directly to the main planning agent consumed context that could otherwise support broader reasoning. - Dash therefore moved search into a specialized agent: - The main agent decides when searching is necessary. - The search agent independently constructs the query using its dedicated prompt. - This division lets the main agent focus on the overall task while the specialist handles search details. Dash’s approach recommends treating context as a limited engineering resource. Use a small number of well-designed tools, pre-filter information for relevance, and delegate technically demanding subtasks to specialized agents rather than overwhelming one general-purpose model.

Read original

AI Agents Model Context Protocol RAG Context Engineering+3