Evaluating Netflix Show Synopses with LLM-as-a-Judge
Netflix developed an LLM-as-a-Judge system to evaluate show synopses at the scale of its extensive catalog. The system assesses creative quality against expert-defined standards while also examining whether scores predict member behavior. With calibrated prompts, extended reasoning, and consensus scoring, the approach achieves more than 85% agreement with creative writers and can identify potentially impactful synopsis problems before a title launches. ## Defining a Good Synopsis - Synopsis quality is measured in two ways: - **Creative quality:** how well a synopsis follows Netflix’s editorial standards. - **Member feedback:** how the synopsis affects viewing decisions and early engagement. - Strong synopses help members quickly understand and choose titles. - Weak or misleading synopses can cause frustration, abandonment, and reduced viewing. ## Building Expert-Labeled Evaluation Data - Creative experts initially labeled roughly 1,000 diverse synopses. - Three writers scored each synopsis and explained their decisions. - Because the task was subjective, Netflix used eight calibration rounds to improve consistency. - Techniques that increased agreement included: - Replacing 1–4 ratings with binary scores. - Allowing writers to consult previous examples. - Maintaining a searchable taxonomy of recurring errors. - A model-in-the-loop process helped resolve disagreements: - Multiple writers supplied scores. - An LLM aggregated the judgments. - Writers reviewed cases with significant disagreement. - The resulting “golden set” contains about 600 synopses with criterion-level labels and explanations. ## Measuring Member Impact - Netflix uses two behavioral metrics: - **Take fraction:** how often members who see a synopsis start watching the title. - **Abandonment rate:** how often viewers stop shortly after beginning. - These metrics act as short-term proxies for long-term retention and have been validated through A/B testing. - Netflix evaluates whether LLM-generated quality scores can predict these engagement outcomes. ## Criterion-Specific LLM Judges - Initial prompts provide: - Relevant show metadata. - A summary of the applicable quality guidelines. - A request for an explanation followed by a binary score. - A single prompt covering every criterion performed poorly because it overloaded the model. - Separate judges for individual criteria performed better. - Binary outputs make evaluation straightforward using accuracy against the expert-labeled golden set. ## Improving Prompts and Reasoning - Netflix applies Automatic Prompt Optimization to a development set of about 300 examples. - Prompts are then manually refined with LLM assistance. - Performance varies significantly by criterion: prompts work well for areas such as precision but less well for subjective criteria such as clarity. - Inference-time scaling improves difficult judgments through: - **Longer rationales**, which give the model more room to reason. - **Consensus scoring**, which samples multiple judgments and combines their results. ## Tiered Rationales - Longer explanations generally improve accuracy, but they become harder for creative experts to read and audit. - Netflix therefore uses tiered rationales: - The model may reason at length internally. - It produces a concise explanation before the final score. - This approach preserves the benefits of extended reasoning while improving interpretability. - For example, the tone evaluator’s accuracy increased from 86.55% to 87.85% with tiered rationales. Netflix’s approach combines expert standards, calibrated evaluation data, specialized prompts, and inference-time reasoning to scale synopsis-quality review. The practical recommendation is to use LLM judges as carefully aligned evaluators—not generic critics—while validating their scores against both human judgment and real member behavior.
Read original(opens in new tab)