Transformer - Netflix

netflixJun 29, 20263 min readCurated summary

GenPage: Towards End-to-End Generative Homepage Construction at Netflix

GenPage is Netflix’s end-to-end generative approach to building personalized homepages. Instead of separately ranking rows and items, one transformer autoregressively generates the entire page—including rows, entities, and layout—from user and request context. In production, it outperformed Netflix’s mature multi-stage recommender on a core engagement metric while reducing serving latency by 20%. ## Reframing Homepage Recommendation - Netflix’s homepage is a personalized two-dimensional structure, not a single ranked list. - Traditional systems use separate candidate-generation and ranking stages for rows and entities. - GenPage treats homepage construction as a prompt-response task: - The prompt contains user history, profile information, and request context. - The response is the complete homepage generated autoregressively. - The approach aims to: - Replace complex multi-stage pipelines with one end-to-end model. - Optimize the whole page using reinforcement learning. - Capture interactions such as diversity and the trade-off between high-value rows and continued browsing. - Scale more predictably with additional data, compute, and model capacity. - Support new content types, layouts, UI components, and personalized artwork with fewer architectural changes. ## Production Challenges and Results - Real-time generation makes serving latency a major constraint. - The system must address: - Cold-start entities in a constantly changing catalog. - Shifting user interests and cultural trends. - Product and business rules that constrain generated pages. - An online A/B test against Netflix’s optimized production recommender produced: - Statistically significant improvement on Netflix’s primary launch engagement metric. - A 20% reduction in end-to-end serving latency. - Offline experiments found that: - Improving the prompt helped more than increasing model capacity in the tested regime. - Reinforcement-learning post-training improved homepage diversity, even though diversity was not an explicit objective. ## Tokenizing Context and Pages - Each training example contains: - **Context:** user history, profile attributes, and request information. - **Page:** displayed rows and entities in layout order. - **Feedback:** interactions such as plays, thumbs-up, and abandonment. - Context and page are tokenized as model inputs and outputs. - Feedback is used to derive reward and supervision signals rather than being directly generated. ## Domain-Specific Tokenization - GenPage uses a custom recommender-system tokenizer instead of a general-purpose text tokenizer. - This reduces sequence length and improves inference cost and latency. - For example, an action such as watching *Orange Is the New Black* can be represented with four tokens: - Entity ID - Action type - Time bucket - Duration bucket - Direct token mappings to product concepts, such as rows and entities, also make it easier to enforce generation rules and business constraints. ## Context Representation - User-history tokens encode: - Action type - Entity ID - Timestamp - Duration - The history includes explicit signals, such as playback, adding titles to My List, and thumbs-up, as well as implicit signals such as trailer views and detail-page visits. - Profile tokens represent attributes including language and profile type. - Request-context tokens include time of day, day of week, and device. - Long data sources, such as complete impression histories, are summarized to control sequence length and cost. - These summaries improve practicality but introduce handcrafted prompt engineering; learning to compress such information end to end remains a future direction. - Special segment markers help the model distinguish between different context sources.

Read original

Transformer Generative AI Reinforcement Learning Personalization+2

netflixJun 23, 20263 min readCurated summary

Toward More Controllable AI Video Editing: An Early Research Exploration at Netflix

Netflix explores AI video-editing tools designed to preserve artists’ creative control rather than regenerate entire clips indiscriminately. The research addresses two major problems: unintended changes to untouched footage and physically implausible results when objects are removed. Its proposed systems, Vera and VOID, generate targeted edits while preserving scene identity, performance, and continuity. ## Challenges in Generative Video Editing - Full-video regeneration can unintentionally change: - Actors’ identities and performances - Backgrounds and objects - Important scene details - Object removal often produces unnatural results because models erase the target without reconstructing realistic motion and physical interactions. - Professional editors need precise control over what changes and what remains untouched. ## Vera: Layered Video Diffusion - Vera generates: - An edit layer containing the requested visual change - An alpha matte defining where that change should appear - These layers are composited with the original footage, leaving pixels outside the edited region intact. - The approach supports tasks such as: - Adding objects - Changing backgrounds - This layered design helps preserve original identities, performances, and details. ## Training Dataset - Netflix created a custom dataset because existing public datasets lacked high-quality layered video data. - The dataset contains 486,000 frames at 832×480 resolution. - It includes: - **Synthetic composites:** Foreground objects with alpha mattes placed over generated backgrounds. - **Realistic single-object videos:** Real footage processed with segmentation, matting, background generation, and human review. - **Realistic multi-object videos with effects:** Objects isolated along with shadows, reflections, and other scene effects. ## Vera’s Model Architecture - Vera uses a Mixture-of-Transformers design with three specialized DiTs for: - The edit layer - The alpha matte - The composite video - Each branch has its own attention projections and feed-forward weights, allowing specialization while joint attention enables communication between layers. - The model is initialized from a pretrained text-to-video model. - Additional embeddings and input layers help distinguish source-video, mask, alpha, and composite information. ## Evaluation and Results - Netflix tested Vera on: - 72 object-addition video-prompt pairs - 69 background-change pairs - The benchmark included varied motion speeds, camera movements, object counts, and scene complexity. - Evaluation measured: - Preservation of untouched content - Compliance with text instructions - Temporal and per-frame video quality - Vera-1.3B and Vera-14B substantially outperformed existing methods on content preservation while achieving comparable instruction-following and visual quality. Netflix’s research favors localized, layered editing over unrestricted video regeneration. Vera demonstrates how separating edits from original footage can make generative tools safer and more controllable for professional workflows; the accompanying VOID research aims to apply similar principles to physically plausible object and interaction removal.

Read original

Transformer Generative AI Computer Vision Diffusion Models+4

netflixFeb 24, 20263 min readCurated summary

MediaFM: The Multimodal AI Foundation for Media Understanding at Netflix

Netflix’s Media Foundational Model (MediaFM) is a tri-modal AI system that combines video, audio, and timed text to understand long-form entertainment. It represents sequences of shots while using title-level metadata and temporal context to produce richer content embeddings. Netflix concludes that these contextual embeddings improve many downstream tasks, including advertising relevance, clip selection, tone classification, and popularity prediction. ## Motivation for MediaFM - Netflix needs machine-readable understanding of its expanding catalog, including films, series, live events, and podcasts. - Long-form media requires recognizing narrative dependencies, emotional arcs, scene transitions, and subtle tones across entire episodes or films. - Combining visual, audio, and textual signals provides a more complete understanding than relying on video alone. - The resulting embeddings support applications such as: - Cold-start recommendations for new titles - Promotional art and trailer optimization - Advertising relevance - Clip tagging and internal content analysis ## Multimodal Input Representation - The model uses a shot as its fundamental unit, with titles segmented using shot-boundary detection. - Each shot receives three modality-specific embeddings: - **Video:** Frames sampled from the shot are encoded with SeqCLIP, Netflix’s video-retrieval model. - **Audio:** Sound is encoded using Meta FAIR’s wav2vec2. - **Timed text:** Captions, subtitles, or audio descriptions are encoded with OpenAI’s `text-embedding-3-large`. - The three embeddings are concatenated and unit-normalized into a 2,304-dimensional fused vector. - Training examples consist of temporally ordered shot sequences from a movie or episode, with up to 512 shots. - Title metadata, such as synopses and tags, is also embedded and supplied as global context. ## Transformer Architecture - MediaFM uses a BERT-like Transformer encoder. - Fused shot embeddings are first projected into the model’s hidden dimension. - Two special tokens are prepended: - `[CLS]`, a learnable sequence-level embedding - `[GLOBAL]`, containing projected title-level metadata - Positional embeddings and self-attention allow each shot representation to incorporate surrounding narrative context. - A final projection maps contextualized representations back into the original 2,304-dimensional embedding space. ## Masked Shot Modeling - The model masks 20% of shot embeddings in each training sequence. - Masked inputs are replaced with a learnable `[MASK]` embedding. - The Transformer must reconstruct the original fused embedding for each masked shot. - Training minimizes cosine distance between predicted and ground-truth embeddings. - Hidden parameters are optimized with Muon, while other parameters use AdamW; Netflix reports noticeable gains after adopting Muon. ## Evaluation Through Linear Probes - Netflix evaluates MediaFM by freezing its representations and training task-specific linear layers on top. - Most evaluation tasks involve short clips extracted from larger titles. - Embedding a clip within the context of its surrounding episode or film performs better than embedding the clip in isolation, demonstrating the value of long-range contextualization. ## Downstream Applications - **Ad relevancy:** Multilabel classification identifies clips suitable for relevant advertising; MediaFM helps retrieve candidate clips before ad-serving optimization. - **Clip popularity ranking:** The model predicts relative clip performance and click-through rate within a title, evaluated using Kendall’s tau. - **Clip tone:** Clips are classified into 100 categories, such as creepy, scary, or humorous. - **Clip genre:** Clips are assigned to core genres including Action, Comedy, Documentary, Drama, Horror, Romance, and Thriller. - **Clip retrieval:** The system distinguishes “clip-worthy” content from unsuitable clips based on human annotations, using Average Precision. MediaFM’s main practical lesson is that effective media understanding depends on fusing all available modalities and preserving long-form temporal context. Netflix’s approach provides a reusable embedding foundation for recommendation, promotion, advertising, and content-analysis systems rather than building a separate representation for every task.

Read original

Transformer Database Design Multimodal Video Embedding+4

netflixOct 25, 2025Original article

Post-Training Generative Recommenders with Advantage-Weighted Supervised Finetuning | by Netflix Technology Blog | Netflix TechBlog (opens in new tab)

Netflix is evolving its recommendation systems by moving beyond simple behavior imitation toward generative recommenders that better align with true user preferences. While generative models like HSTU and OneRec effectively capture sequential user patterns, they often struggle to distinguish between habitual clicks and genuine satisfaction. To bridge this gap, Netflix developed Advantage-Weighted Supervised Fine-tuning (A-SFT), a post-training method that leverages noisy reward signals to refine model performance without the need for complex counterfactual data. ### The Shift to Generative Recommenders * Modern generative recommenders (GRs), such as HSTU and OneRec, utilize transformer architectures to treat recommendation as a sequential transduction task. * The models are typically trained using next-item prediction, where the system learns to imitate the chronological sequence of a user’s activities. * A significant drawback of this "behavior cloning" approach is that it captures external trends and noise rather than long-term user satisfaction, potentially recommending content the user finished but did not actually enjoy. ### Barriers to Reinforcement Learning in RecSys * Traditional post-training methods used in Large Language Models, such as Proximal Policy Optimization (PPO) or Direct Preference Optimization (DPO), require counterfactual feedback that is difficult to obtain in recommendation contexts. * Because user sequences span weeks or years, it is impractical to generate and test hypothetical, counterfactual experiences for real-time user validation. * Reward signals in recommendation systems are inherently noisy; for instance, high watch time might indicate interest, but it can also be a result of external circumstances, making it an unreliable metric for optimization. ### Advantage-Weighted Supervised Fine-tuning (A-SFT) * A-SFT is a hybrid approach that sits between offline reinforcement learning and standard supervised fine-tuning. * The algorithm incorporates an advantage function to weight training examples, allowing the model to prioritize actions that lead to higher rewards while filtering out noise from the reward model. * This method is specifically designed to handle high-variance reward signals, using them as directional guides rather than absolute truth, which prevents the model from over-exploiting inaccurate data. * Benchmarks against other representative methods show that A-SFT achieves superior alignment between the generative recommendation policy and the underlying reward model. For organizations managing large-scale recommendation engines, A-SFT offers a practical path to implementing post-training improvements. By focusing on advantage-weighted signals, developers can improve recommendation quality using existing implicit feedback—like watch time and clicks—without the infrastructure hurdles of online reinforcement learning.

Transformer AI Machine Learning Recommendation Systems+4