pinterestFeb 27, 2026

Bridging the Gap: Diagnosing Online–Offline Discrepancy in Pinterest’s L1 Conversion Models (opens in new tab)

Pinterest found that strong offline gains in L1 conversion-rate models did not translate into online improvements because training and serving environments were not aligned. Although experimental models reduced LogMAE by roughly 20–45% and improved calibration, online A/B tests showed neutral or worse CPA and unexpected oCPM mix shifts. The investigation identified feature coverage gaps and embedding version skew as structural causes rather than problems with offline evaluation or serving reliability. ## How L1 Models Are Evaluated - L1 filters and prioritizes ads under strict latency limits before downstream ranking and auction stages. - Offline evaluation focused on: - LogMAE and calibration - Performance across candidate pools and pCVR percentiles - Multiple data sources, including auction winners and candidates - Online evaluation focused on: - CPA and other business metrics - Candidate counts and recall across funnel stages - Differences among optimization types, especially oCPM traffic ## Hypotheses That Were Ruled Out - **Offline evaluation errors** - The experimental model consistently outperformed production across three log sources. - Gains remained across pCVR buckets, including after outlier handling. - **Exposure bias** - Increasing treatment traffic from approximately 20% to 70% did not resolve the online over-calibration issue. - **Serving failures** - Control and treatment had comparable success rates and p50/p90/p99 latency. - Timeouts and tail latency were therefore unlikely to explain the discrepancy. ## Missing Features in L1 Serving - Offline training used rich logged features, while online L1 embeddings only included features explicitly onboarded into the embedding pipeline. - Important feature families were absent online, including: - Targeting specification flags - Offsite conversion visit counts over 1-, 7-, 30-, and 90-day windows - Annotations and MediaSage image embeddings - Models learned to depend on these signals during training, but received a substantially thinner feature set when serving many oCPM and performance-oriented ads. - Pinterest updated UFR configurations to add the missing features to L1 embeddings. - Online feature coverage recovered, and online loss improved for CVR and engagement models, particularly on shopping traffic. - UFR tooling was also changed so features onboarded for L2 are automatically considered for L1 embedding usage. ## Query–Pin Embedding Version Skew - Pinterest’s two-tower architecture requires query and Pin embeddings to be generated from compatible model checkpoints. - Offline evaluation generally uses one fixed checkpoint for both towers. - Online pipelines could instead serve query and Pin embeddings produced from different model versions, creating a mismatch between training assumptions and production behavior. - This version skew was identified as a second structural source of online–offline inconsistency. ## Practical Conclusion Offline model quality is not sufficient for launching L1 improvements. Teams must verify feature coverage in serving artifacts such as ANN indices, enforce synchronized query and Pin embedding versions, and monitor funnel behavior and online feature coverage alongside standard offline metrics.

feature-storemachine-learningranking-modelstwo-tower-architecture+4

pinterestJan 28, 2026

Ads Candidate Generation using Behavioral Sequence Modeling (opens in new tab)

Pinterest’s Ads team uses behavioral sequence modeling to improve ad candidate generation by predicting what users are likely to convert on next. Transformer-based two-tower models first predict relevant advertisers and then specific products, using offsite activity such as views, purchases, and add-to-cart events. The advertiser model is already in production, while item-level modeling addresses Pinterest’s rapidly growing catalog and enables more precise, scalable personalization. ## Predicting Advertiser Interaction - A bidirectional Transformer encodes each user’s behavioral event sequence. - An MLP-based advertiser tower represents candidate advertisers. - Training uses: - In-batch negative samples - Sampled softmax loss - Positive events consisting of checkout, add-to-cart, or signup conversions within a future K-day window - Log-Q bias correction to avoid excessively penalizing popular advertisers - The model is evaluated with Recall@K by comparing user and advertiser embedding similarity against an indexed set of roughly 2 million advertisers. - An offline batch job generates each user’s top 100 advertisers and publishes them to the online feature store. - During ad serving, eligible ads from those advertisers are passed to the L1 ranker, blended with other candidate sources, and scored by heavier downstream models and the marketplace auction. - Online experiments produced higher conversion volume and lower cost per action. - The advertiser-level model has served production traffic for Standard ads since Spring 2024. ## Moving from Advertisers to Products - Pinterest next sought to predict the specific products a user would interact with, rather than only the likely advertiser. - Item-level prediction better matches the item-based ad delivery funnel and avoids forcing downstream models to score an impractically large set of products from selected advertisers. - The approach aims to capture both immediate intent and longer-term interests. ## Item-Level Model Architecture - The model retains the two-tower design: - A user tower encodes behavioral sequences. - An item tower represents individual shopping product Pins. - Item representations combine: - Internal Pin embeddings learned from Pinterest’s engagement graph - Product metadata from the merchant catalog - Because the catalog exceeds 1 billion items, training uses both in-batch negatives and a randomly sampled negative set of 20 million Pins. - The model uses the same conversion labels as the advertiser model. - Label weights and log-Q parameters are tuned to balance retrieval quality with diversity across both products and advertisers. - Daily inference updates user embeddings only for users with new activity, appending them to a previous feature-store snapshot to reduce computation. - The trained item tower indexes hundreds of millions of ad items. ## Evaluation and Diversity - Item retrieval is evaluated using cosine similarity and hit rates at different K values. - Final model selection considers both: - Item-level Recall@K - Advertiser-level Recall@K - Qualitative review is also important because offsite activity is sparse and noisy. - The model is compared with max-pooling and mean-pooling baselines that use aggregated embeddings without Transformer-based sequence modeling. - The evaluation emphasizes that strong retrieval must also produce semantically relevant and sufficiently diverse recommendations. Pinterest’s progression from advertiser prediction to item prediction shows how behavioral sequence models can make ad retrieval more personalized while remaining scalable. A practical system should combine sequence-aware user representations, large-scale approximate retrieval, and explicit controls for popularity, diversity, and computational efficiency.

feature-storetransformerembeddingscontrastive-learning+4