netflixJul 30, 2026

GenRec: Towards LLM-Native Recommendation at Netflix (opens in new tab)

GenRec is Netflix’s LLM-backed recommendation ranker, designed to reduce dependence on thousands of hand-engineered features. It verbalizes user history, item metadata, and context, then post-trains a Netflix-adapted foundation model for catalog-aware ranking and long-term member value. In large-scale A/B testing, it reportedly improved both short- and long-term metrics while using far fewer labeled examples and input signals than an established production ranker. ## Motivation for an LLM-Native Recommender - Netflix’s existing recommendation stack supports many content types and product surfaces but is costly to extend. - New use cases can require substantial feature engineering, architectural changes, infrastructure work, and experimentation. - LLMs offer: - Shared semantic representations for users, items, and interactions - Natural-language steering through prompts - Rich understanding of content and user histories - General-purpose LLMs are not production-ready on their own because they may: - Over-recommend globally popular titles - Hallucinate items outside the catalog - Ignore business constraints - Provide limited personalization ## Ranking Problem and Long-Term Utility - GenRec ranks the full Netflix catalog, or a provided candidate set. - It uses the user, interaction history, current context, and time to produce a personalized ranking. - The optimization target is expected long-term member utility—a proxy for satisfaction and retention—rather than only immediate clicks or plays. ## Two-Phase Training ### Netflix-Adapted Foundation Model - Netflix first adapts an open-source LLM using proprietary Netflix data. - The model learns: - Netflix content and metadata - Member behavior and preference patterns - General language understanding and generation - This relatively stable foundation can support multiple Netflix applications. ### GenRec Post-Training - A second training phase specializes the foundation model for recommendation. - It focuses on ranking quality, steering, reward alignment, and serving-cost constraints. - This phase is refreshed more frequently to reflect new content and changing member preferences. ## Interaction Data as Conversations - Hundreds of billions of events—including views, play duration, feedback, add-to-list actions, and abandons—are converted into single- or multi-turn recommender conversations. - Each user message includes verbalized: - Context and profile - Interaction history - Item metadata - The recommendation task - Assistant messages represent actual member behavior, such as titles played, viewing duration, or feedback. - During inference, GenRec uses the verbalized request and a catalog-aware scoring head; it does not generate conversational responses. - The conversational format mainly supports language-model training and comprehension of rich textual inputs. ## Context Engineering Instead of Feature Engineering - GenRec represents histories and context as natural language rather than relying primarily on dense, manually designed features. - The token budget becomes the new feature budget, so histories are selectively compressed: - Retain detailed, high-signal events such as long plays and thumbs-up - Remove low-signal events such as brief plays and hovers - Summarize repetitive behavior such as binge-watching - Add detail for important or cold-start items, including new releases - Recent and high-value interactions receive priority, while older information is compressed or dropped. - Prompt structure is also optimized for shared prefixes and prefix caching, reducing serving costs. ## Ranking, Language, and Reward Objectives - GenRec combines several training objectives: - **Catalog-aware ranking:** Uses high-value engagements as positive labels, applies denoising and thresholds, and trains the model with cross-entropy over the catalog or candidate set. - **Language modeling:** Preserves understanding of natural-language histories and metadata and supports potential future features such as recommendation explanations. - **Reward-weighted alignment:** Incorporates business requirements and longer-term satisfaction into training. - Reward signals can help balance content categories such as movies, series, games, live content, and podcasts instead of optimizing only immediate engagement. ## Serving and Results - GenRec runs in prefill-only mode on Netflix’s LLM serving infrastructure, avoiding the cost of decoding generated text. - A catalog-aware scoring head converts the model’s representations into item rankings. - Compared with a mature production ranker, GenRec achieved statistically significant improvements in short- and long-term online metrics. - It did so with a small fraction of the labeled data and input signals used by the existing system. GenRec suggests that recommendation systems can shift from extensive manual feature construction toward careful context engineering, LLM post-training, and reward alignment. The approach is most promising when paired with catalog constraints, efficient serving, and objectives that reflect long-term member value rather than raw engagement alone.

ranking-modelsllmnlprecommendation-systems+2

pinterestFeb 27, 2026

Bridging the Gap: Diagnosing Online–Offline Discrepancy in Pinterest’s L1 Conversion Models (opens in new tab)

Pinterest found that strong offline gains in L1 conversion-rate models did not translate into online improvements because training and serving environments were not aligned. Although experimental models reduced LogMAE by roughly 20–45% and improved calibration, online A/B tests showed neutral or worse CPA and unexpected oCPM mix shifts. The investigation identified feature coverage gaps and embedding version skew as structural causes rather than problems with offline evaluation or serving reliability. ## How L1 Models Are Evaluated - L1 filters and prioritizes ads under strict latency limits before downstream ranking and auction stages. - Offline evaluation focused on: - LogMAE and calibration - Performance across candidate pools and pCVR percentiles - Multiple data sources, including auction winners and candidates - Online evaluation focused on: - CPA and other business metrics - Candidate counts and recall across funnel stages - Differences among optimization types, especially oCPM traffic ## Hypotheses That Were Ruled Out - **Offline evaluation errors** - The experimental model consistently outperformed production across three log sources. - Gains remained across pCVR buckets, including after outlier handling. - **Exposure bias** - Increasing treatment traffic from approximately 20% to 70% did not resolve the online over-calibration issue. - **Serving failures** - Control and treatment had comparable success rates and p50/p90/p99 latency. - Timeouts and tail latency were therefore unlikely to explain the discrepancy. ## Missing Features in L1 Serving - Offline training used rich logged features, while online L1 embeddings only included features explicitly onboarded into the embedding pipeline. - Important feature families were absent online, including: - Targeting specification flags - Offsite conversion visit counts over 1-, 7-, 30-, and 90-day windows - Annotations and MediaSage image embeddings - Models learned to depend on these signals during training, but received a substantially thinner feature set when serving many oCPM and performance-oriented ads. - Pinterest updated UFR configurations to add the missing features to L1 embeddings. - Online feature coverage recovered, and online loss improved for CVR and engagement models, particularly on shopping traffic. - UFR tooling was also changed so features onboarded for L2 are automatically considered for L1 embedding usage. ## Query–Pin Embedding Version Skew - Pinterest’s two-tower architecture requires query and Pin embeddings to be generated from compatible model checkpoints. - Offline evaluation generally uses one fixed checkpoint for both towers. - Online pipelines could instead serve query and Pin embeddings produced from different model versions, creating a mismatch between training assumptions and production behavior. - This version skew was identified as a second structural source of online–offline inconsistency. ## Practical Conclusion Offline model quality is not sufficient for launching L1 improvements. Teams must verify feature coverage in serving artifacts such as ANN indices, enforce synchronized query and Pin embedding versions, and monitor funnel behavior and online feature coverage alongside standard offline metrics.

ranking-modelsmachine-learningfeature-storetwo-tower-architecture+4

pinterestFeb 2, 2026

Beyond Two Towers: Re-architecting the Serving Stack for Next-Gen Ads Lightweight Ranking Models… (opens in new tab)

Two-Tower models make retrieval and lightweight ranking highly efficient by scoring user and item embeddings with a dot product, but they cannot represent rich user-item interactions or deep feature crossings. This post describes an ads-serving redesign that introduces general-purpose GPU models while preserving end-to-end latency. The main strategy is to reduce data movement, move filtering logic onto the GPU, and optimize inference from an initial 4-second p90 latency to about 20 milliseconds. ## Why Move Beyond Two-Tower Models - Two-Tower architectures independently encode users and items, enabling fast scoring across millions of candidates. - Their decoupled structure limits: - User-item interaction features - Target attention - Early feature crossing - Deep architectures requiring simultaneous access to user and candidate data - More expressive models require GPU-based general-purpose inference rather than specialized dot-product or ANN retrieval. - The existing retrieval stack was not designed to transfer large candidate and feature sets to a GPU, creating a major latency challenge. ## Restructuring the Serving Funnel The traditional funnel consisted of: - Feature expansion for thousands of candidates - Retrieval and Two-Tower lightweight ranking - Heavy ranking and auction processing for the top documents Adding GPU inference directly to this flow would require fetching, serializing, transferring, and returning features for tens of thousands of documents. The authors therefore redesigned the entire early-stage serving pipeline instead of optimizing the model alone. ## Segmenting the Inventory for Feature Fetching Feature retrieval was a major latency source, often taking longer than model inference for workloads ranging from 10,000 to 100,000 documents. - **High-value inventory:** Roughly 1 million documents responsible for a substantial share of revenue have their features embedded in the PyTorch model as registered buffers. - Features become part of the model state, similar to weights. - They remain in GPU high-bandwidth memory. - Requests avoid remote feature-service calls and host-to-device transfers. - The model file must be periodically updated to refresh features. - Future work may include GPU-based caching. - **Long-tail inventory:** The remaining roughly 1 billion documents use a high-performance key-value store with in-host caching. - The post focuses on the first strategy, which is already running in production. ## Moving Business Logic onto the GPU Previously, the model returned scores for approximately 100,000 candidates, while CPU-side code handled utility calculation, filtering, diversity, deduplication, and top-k selection. - The new PyTorch model performs these operations directly: - Combines pCTR, pCVR, bid, and other signals into utility scores. - Applies diversity and filtering rules. - Performs top-k selection. - The GPU returns only the final winners—typically around 1,000 documents—instead of all candidate scores. - This reduces device-to-host data transfer and takes advantage of GPU parallelism. - The approach works because lightweight-ranking business rules are sufficiently simple to express with tensor operations. ## Reducing GPU Inference Latency Initial GPU inference measured roughly 4,000 ms at p90, far too slow for real-time serving. Several systems optimizations reduced this to approximately 20 ms: - **Multiple CUDA streams:** Separate streams for workers allow host-to-device transfers, computation, and device-to-host transfers to overlap. - **Worker alignment:** Worker threads are matched and pinned to physical CPU cores to reduce context switching and lock contention. - **Kernel fusion:** Triton kernels combine operations such as linear layers and activations, reducing memory traffic. - **BF16 computation:** Brain Floating Point 16 lowers memory usage and accelerates arithmetic compared with FP32. - **Profiling tools:** PyTorch Profiler and NVIDIA Nsight Systems were used to identify bottlenecks. ## Practical Recommendation Deploying more expressive ranking models requires rethinking the serving architecture around data movement and execution placement. Embedding frequently used features, executing business logic on the GPU, and applying low-level CUDA and kernel optimizations can make complex neural ranking feasible without increasing end-to-end latency.

ranking-modelsmachine-learningrecommendation-systemspytorch+4