MediaFM: The Multimodal AI Foundation for Media Understanding at Netflix
Netflix’s Media Foundational Model (MediaFM) is a tri-modal AI system that combines video, audio, and timed text to understand long-form entertainment. It represents sequences of shots while using title-level metadata and temporal context to produce richer content embeddings. Netflix concludes that these contextual embeddings improve many downstream tasks, including advertising relevance, clip selection, tone classification, and popularity prediction. ## Motivation for MediaFM - Netflix needs machine-readable understanding of its expanding catalog, including films, series, live events, and podcasts. - Long-form media requires recognizing narrative dependencies, emotional arcs, scene transitions, and subtle tones across entire episodes or films. - Combining visual, audio, and textual signals provides a more complete understanding than relying on video alone. - The resulting embeddings support applications such as: - Cold-start recommendations for new titles - Promotional art and trailer optimization - Advertising relevance - Clip tagging and internal content analysis ## Multimodal Input Representation - The model uses a shot as its fundamental unit, with titles segmented using shot-boundary detection. - Each shot receives three modality-specific embeddings: - **Video:** Frames sampled from the shot are encoded with SeqCLIP, Netflix’s video-retrieval model. - **Audio:** Sound is encoded using Meta FAIR’s wav2vec2. - **Timed text:** Captions, subtitles, or audio descriptions are encoded with OpenAI’s `text-embedding-3-large`. - The three embeddings are concatenated and unit-normalized into a 2,304-dimensional fused vector. - Training examples consist of temporally ordered shot sequences from a movie or episode, with up to 512 shots. - Title metadata, such as synopses and tags, is also embedded and supplied as global context. ## Transformer Architecture - MediaFM uses a BERT-like Transformer encoder. - Fused shot embeddings are first projected into the model’s hidden dimension. - Two special tokens are prepended: - `[CLS]`, a learnable sequence-level embedding - `[GLOBAL]`, containing projected title-level metadata - Positional embeddings and self-attention allow each shot representation to incorporate surrounding narrative context. - A final projection maps contextualized representations back into the original 2,304-dimensional embedding space. ## Masked Shot Modeling - The model masks 20% of shot embeddings in each training sequence. - Masked inputs are replaced with a learnable `[MASK]` embedding. - The Transformer must reconstruct the original fused embedding for each masked shot. - Training minimizes cosine distance between predicted and ground-truth embeddings. - Hidden parameters are optimized with Muon, while other parameters use AdamW; Netflix reports noticeable gains after adopting Muon. ## Evaluation Through Linear Probes - Netflix evaluates MediaFM by freezing its representations and training task-specific linear layers on top. - Most evaluation tasks involve short clips extracted from larger titles. - Embedding a clip within the context of its surrounding episode or film performs better than embedding the clip in isolation, demonstrating the value of long-range contextualization. ## Downstream Applications - **Ad relevancy:** Multilabel classification identifies clips suitable for relevant advertising; MediaFM helps retrieve candidate clips before ad-serving optimization. - **Clip popularity ranking:** The model predicts relative clip performance and click-through rate within a title, evaluated using Kendall’s tau. - **Clip tone:** Clips are classified into 100 categories, such as creepy, scary, or humorous. - **Clip genre:** Clips are assigned to core genres including Action, Comedy, Documentary, Drama, Horror, Romance, and Thriller. - **Clip retrieval:** The system distinguishes “clip-worthy” content from unsuitable clips based on human annotations, using Average Precision. MediaFM’s main practical lesson is that effective media understanding depends on fusing all available modalities and preserving long-form temporal context. Netflix’s approach provides a reusable embedding foundation for recommendation, promotion, advertising, and content-analysis systems rather than building a separate representation for every task.
Read original(opens in new tab)