LLM-Powered Relevance Assessment for Pinterest Search

Pinterest Search uses fine-tuned multilingual LLMs to assess search-result relevance at a much larger scale than human labeling allows. The approach combines five-level relevance classification, stratified query sampling, and paired A/B-test evaluation to detect smaller overall effects and differences across query types. XLM-RoBERTa-large provides a practical balance of accuracy and cost, achieving strong agreement with human judgments while enabling substantially faster labeling.

Relevance Measurement Challenges

• Search relevance measures how well Pins satisfy a user’s query, rather than merely reflecting past engagement.
• Human annotations are expensive and limited in volume.
• Previous sampling designs could detect only relatively large topline changes, with minimum detectable effects (MDEs) around 1.3%–1.5%.
• Limited labels also made it difficult to measure heterogeneous effects across query interests or popularity segments.

Fine-Tuned LLM Relevance Model

• Pinterest defines relevance using five labels:
  • L5: Highly Relevant
  • L4: Relevant
  • L3: Marginally Relevant
  • L2: Irrelevant
  • L1: Highly Irrelevant
• A cross-encoder model predicts the relevance of each Pin for a query.
• Open-source multilingual models are fine-tuned on human-annotated examples using multiclass cross-entropy loss.
• Pin representations include:
  • Titles and descriptions
  • BLIP-generated image captions
  • Linked-page titles and descriptions
  • Board titles where Pins were saved
  • Highly engaged query tokens associated with the Pin
• Models tested included multilingual BERT, T5, mDeBERTa, XLM-RoBERTa, and Llama 3.
• The final relevance label is selected from the model’s five output scores using argmax.

Stratified Query Sampling

• Lower LLM labeling costs allow Pinterest to use much larger and more detailed samples.
• Queries are stratified using:
  • A DistilBERT-based query-to-interest model
  • Query popularity, based on how many users issue each query
• Stratification improves representativeness and reduces variance by grouping similar queries.
• Pinterest moved from simple random sampling to stratified sampling with optimal allocation across strata.
• Most of the MDE improvement came from variance reduction through stratification.
• The redesigned process reduced MDEs from approximately 1.3%–1.5% to 0.25% or less.

LLM-Based A/B-Test Measurement

• Pinterest samples paired queries from control and treatment groups.
• Pairing controls for differences between queries, which are a major source of relevance variance.
• For each query, the top 25 results are retained and labeled by the LLM.
• Query-level relevance is measured using sDCG@25, a variant of nDCG that assumes an unlimited supply of highly relevant L5 results.
• Results are aggregated into topline experiment metrics.
• Heterogeneous effects are analyzed by query popularity and interest categories such as beauty, fashion, and art.
• The Benjamini–Hochberg procedure controls the false discovery rate when testing multiple segments.

Model Choice and Validation

• XLM-RoBERTa-large was selected for its balance of quality and efficiency.
• On a single A10G GPU, it can label 150,000 rows in about 30 minutes.
• Llama 3–8B produced slightly better accuracy but required roughly six times the inference time and cost.
• LLM labels matched human labels exactly for 73.7% of Pins.
• A total of 91.7% of predictions differed from human ratings by no more than one relevance point.

Pinterest’s approach makes relevance evaluation cheaper, faster, and more statistically sensitive. Fine-tuned LLMs paired with stratified sampling are recommended for search experimentation when human labeling cannot provide enough coverage to detect small or heterogeneous ranking effects.