A practical blueprint for evaluating conversational AI at scale

Conversational AI systems depend on many probabilistic stages, so even small changes can cause unexpected regressions. Dropbox Dash’s experience shows that evaluation should be treated like production engineering: systematic, repeatable, and required before changes are shipped. The approach combines curated datasets, actionable metrics, LLM-based judging, and human review.

Evaluation as a Development Discipline

• AI pipelines include intent classification, retrieval, ranking, prompt construction, inference, and safety filtering.
• Changes to any stage can affect final answer quality in unpredictable ways.
• Dropbox initially used ad-hoc testing, but shifted to a standardized process in which every model, prompt, or retrieval change had to pass evaluation before merging.
• The evaluation framework covers datasets, metrics, tools, and workflows.
• Future-proof evaluation must extend beyond text to images, video, and audio.

Curating Public and Internal Datasets

• Public datasets established baseline retrieval and question-answering performance:
  • Natural Questions tested retrieval from very large documents.
  • MS MARCO emphasized queries requiring multiple document hits.
  • MuSiQue tested multi-hop reasoning.
• Internal datasets captured real-world usage from anonymized Dropbox employee queries and content.
• Representative query sets reflected actual user behavior, using proxy labels or internal annotators.
• Representative content sets focused on shared files, documentation, and connected data sources.
• LLM-generated synthetic questions and answers covered tables, images, tutorials, and factual lookups.
• These datasets became the foundation for automated pass/fail checks and experiment monitoring.

Why Traditional Metrics Fall Short

• Metrics such as BLEU, ROUGE, METEOR, BERTScore, and embedding similarity are fast and useful for detecting major regressions.
• They often fail to measure production requirements, including:
  • Correct source citations
  • Factual accuracy
  • Valid file names and references
  • Reliable table parsing
  • Clear formatting
• High ROUGE or BERTScore can coexist with hallucinations, missing citations, or buried factual errors.
• Production evaluation therefore requires metrics tied directly to task requirements.

Using LLMs as Evaluation Judges

• LLM judges can assess dimensions traditional metrics miss, including:
  • Whether an answer addresses the query
  • Whether claims are supported by context
  • Citation correctness
  • Clarity, formatting, and tone
• Judges receive the query, model answer, source context, and sometimes a hidden reference answer.
• They return structured scores—scalar or categorical—alongside explanations.
• Rubrics and judge models must themselves be tested, calibrated, versioned, and improved.
• Specialized judges may be needed for particular languages or technical domains.

Calibration and Human Review

• Periodic manual labeling of sampled outputs created calibration sets for comparing human and judge-model decisions.
• The team tracked agreement and judge drift over time.
• Engineers manually reviewed 5–10% of each release’s regression suite.
• Disagreements were investigated as either judge-prompt defects or model errors.
• Recurring problems led to prompt revisions or more detailed scoring criteria.

Dropbox’s evaluation-first approach treats AI changes like code changes: test them against realistic datasets, score them with task-specific rubrics, validate the evaluators, and retain human oversight. This makes conversational AI development more reliable as systems and modalities grow more complex.