Machine Learning - Netflix

netflixJun 19, 20263 min readCurated summary

Predicting Risk in Content Launches: How Data-Driven Insights can Transform Launch Planning

Netflix developed boosted-tree models to predict when in-progress productions will deliver Locked Cuts and final IMF media. The models address gaps and inaccuracies in manually maintained schedules, improving delivery-date accuracy and providing earlier warnings of risk. Backtesting shows that predictive dates reduce error and Accumulated Error Days (AED), a measure strongly associated with launch delays. ## Launch Preparation and Schedule Risk - After production, titles move through post-production and launch preparation. - Final IMF assets trigger work on: - Artwork and trailers - Subtitles - Maturity ratings - Quality control - Teams can begin earlier with a non-final Locked Cut, but later changes may require conformance work. - Waiting for the IMF risks compressing the launch timeline if delivery is late. ## Problems with Manual Schedules - Delivery estimates are manually supplied by content partners. - Schedules often contain missing dates and inaccurate estimates. - Dynamic production conditions—schedule changes, conflicts, and unforeseen obstacles—frequently cause delays. - Predictive modeling can fill missing ETAs and improve existing ones. ## Accumulated Error Days and Launch Misses - Accumulated Error Days (AED) measures the cumulative difference between estimated and actual delivery dates. - Titles with launch misses have significantly higher mean AED than titles without misses. - Inaccuracies close to delivery are more strongly associated with launch misses than errors accumulated over longer periods. - Improving schedule accuracy near launch is therefore especially valuable. ## Predicting Time to Delivery - Netflix uses boosted-tree regression models to predict the number of days until Locked Cut or IMF delivery. - Models use: - Production progress signals - Title metadata - Seasonal indicators - Daily snapshots of production data - Snapshot-based modeling keeps predictions current and supports changing features throughout all production phases. ## Evaluating Predictive Performance - Netflix compares predicted and scheduled dates using: - Mean and median absolute error - Mean and median bias - Error standard deviation - Rates of large errors beyond specified day thresholds - Predictive dates offer full coverage, unlike schedules that may lack estimates at some horizons. - Backtesting showed lower errors and fewer outliers for predicted IMF and Locked Cut dates. ## Earlier Accuracy Signals - Predictions can become reliable earlier than manual schedules. - Six months before Locked Cut delivery, predictions were more accurate than scheduled dates for 76% of titles. - Their 6.1-week mean absolute error matched the accuracy of scheduled dates only 11 weeks later. - Across six months before delivery, predicted dates reduced AED for most buying organizations and content types. ## Integrating Predictions into Existing Workflows - Because delivery dates already support stakeholder workflows, predictive estimates can be introduced without redesigning those processes. - The remaining challenge is deciding when to trust scheduled dates versus predictions. - Although predictions are generally more accurate, manual schedules can still outperform them in some situations, requiring a way to select the more reliable estimate. Netflix’s modeling approach turns production data into an ongoing risk signal rather than relying solely on static partner schedules. Using predictive dates alongside existing workflows can give teams earlier, more accurate information for launch planning and help reduce avoidable launch delays.

Read original

Machine Learning Predictive Modeling Analytics Engineering Time Series Forecasting+3

netflixMay 4, 20263 min readCurated summary

Democratizing Machine Learning at Netflix: Building the Model Lifecycle Graph

Netflix’s growing use of machine learning across personalization, Studio, payments, advertising, and other domains has created a fragmented ecosystem of tools and metadata. The Metadata Service (MDS) addresses this problem by building a Model Lifecycle Graph that connects models, features, pipelines, experiments, datasets, and ownership information. Its goal is to make ML assets discoverable, understandable, and reusable across organizational boundaries. ## A Fragmented Machine Learning Landscape - Netflix ML has expanded from personalization into areas such as: - Studio production and post-production - Fraud detection and payment optimization - Advertising and real-time targeting - Each domain uses different technologies, metrics, and organizational structures. - Valuable assets often remain isolated in specialized systems. - For example, Studio-generated content embeddings could support: - Contextual ad matching - Episodic merchandising - Recommendations based on tone, topic, or mood - Practitioners struggle to answer basic questions because relevant information is split across: - Model registries - Pipeline orchestrators - Experimentation platforms - Feature stores - Dataset systems - This fragmentation makes discovery, lineage tracking, impact analysis, and ownership difficult. ## The Challenge of Connecting ML Infrastructure - MDS must unify metadata from many independent systems, including: - Pipeline execution and transformation data - Model versions, artifacts, deployments, and staleness - A/B test configurations - Feature definitions and usage - Dataset creation and discovery - User, team, and organization information - These systems use different identifiers, formats, and conceptual models. - The core challenge is transforming heterogeneous metadata into a common entity model and connected graph—not merely creating a consolidated user interface. ## The Model Lifecycle Graph - Netflix’s Metadata Service indexes ML-related assets and materializes relationships between them. - It supports real-time metadata ingestion and cross-domain questions such as: - Which experiments use a particular model? - Which models depend on a feature? - What data sources feed a model? - Who owns each part of the workflow? - The graph is intended to make every ML asset discoverable and reusable regardless of its originating team or business domain. ## Core Concepts and Vocabulary - **Component:** Any uniquely addressable object identified by an AIP URI, such as: - `aip://model/registry/ranking-v5` - `aip://user/identity/alice` - `aip://pipeline/orchestrator/weekly-training` - **Entity:** A component enriched with properties such as name, description, creation date, and ownership. - **Entity type:** A group of entities sharing the same data shape and required properties. - **Domain:** An abstract interface for a category of ML assets, such as Models or Pipelines. - **Provider:** A concrete backend implementation of a domain, such as Netflix’s internal model registry. - Separating domains from providers allows multiple systems to implement the same interface without changing how consumers interact with MDS. - URI-based addressing gives services a consistent way to reference assets and resolve them to connected metadata. ## From Events to a Queryable Graph - MDS receives metadata events through Kafka and AWS SNS/SQS. - Source systems emit lightweight events containing an event type and resource identifier. - For example, a model registry might emit a `model_instance_created` event with the new instance’s ID. - This keeps event producers simple while allowing MDS to enrich events, construct entities, and infer relationships such as connections between models and A/B tests. The Model Lifecycle Graph provides Netflix with a common layer for connecting previously isolated ML systems. By standardizing identifiers, entities, domains, and providers, MDS can support cross-domain discovery, lineage, impact analysis, and collaboration at scale.

Read original

Machine Learning Apache Kafka Feature Store Graph Database+3

netflixMay 1, 20263 min readCurated summary

State of Routing in Model Serving

Netflix’s centralized ML serving platform provides a single, domain-independent API for model inference across personalized experiences and other use cases. Rather than exposing individual scoring functions, Netflix packages feature computation, preprocessing, inference, and postprocessing into self-contained model workflows. The core routing challenge is directing each request to the correct model version and serving cluster while keeping client services independent from model changes and infrastructure topology. ## Models as End-to-End Workflows - Netflix distinguishes **model serving** from traditional model inference: - Inference typically means `infer(features) -> score`. - Serving includes preprocessing, feature computation, optional trained components, and postprocessing. - Example workflows include: - Ranking titles for a personalized Continue Watching row using user, country, and device context. - Predicting payment fraud using user, country, and transaction details. - Models declare the facts they need, while the serving platform retrieves those facts from other microservices. - During offline training, Netflix’s ML fact store provides snapshots for bulk feature computation. - Calling services provide standard request context and domain-specific inputs, while the platform handles feature generation, model selection, and execution. ## Platform Design Principles - **Model innovation without client changes** - Client applications integrate with the platform once. - Model versions, A/B tests, additional experimental data, logging, and model selection remain hidden behind the platform API. - **Clients decoupled from model sharding** - Models run across multiple serving cluster shards, each with its own Virtual IP address. - Shard assignments can change based on traffic, SLAs, model architecture, and resource availability. - Clients should not need to track these VIP changes. - **Flexible traffic routing** - Routing must support A/B allocations, gradual traffic shifts, new model versions, new VIPs, and client-specific overrides. - Safe lifecycle management requires support for shadow deployments, canaries, rollbacks, and migrations. ## Switchboard: Context-Aware Routing - Generic API gateways and service-mesh proxies did not satisfy Netflix’s requirements. - Netflix needed: - Native integration with its experimentation platform. - gRPC support. - Routing based on rich, domain-specific request context. - Model-specific rollout and migration controls. - Netflix built **Switchboard**, a custom proxy layer handling more than one million requests per second. - Switchboard is the mandatory entry point for clients and: - Routes requests to the appropriate model based on request context. - Applies configured context enrichment before invoking the model. - Hides model locations and infrastructure changes from client services. ## Objective Abstraction - Every request must provide an **Objective**, an enumeration defined by the serving platform. - The excerpt introduces Objectives as a central abstraction for identifying the business purpose of a serving request, but the supplied text ends before describing its full roles. Netflix’s approach is to centralize routing, experimentation, and model execution behind one stable API. This allows client applications to evolve independently while researchers can iterate on models and safely manage large-scale production rollouts.

Read original

Machine Learning Database Design PostgreSQL Microservices+4

netflixNov 4, 2025Original article

Netflix's Metaflow Spin: Faster ML Development | Netflix TechBlog (opens in new tab)

Netflix has introduced Spin, a new functionality within the Metaflow framework designed to significantly accelerate the iterative development cycle for ML and AI workflows. By bridging the gap between the interactive speed of notebooks and the production-grade reliability of versioned workflows, Spin allows developers to experiment with stateful increments without the latency of full restarts. This enhancement ensures that the "prototype to production" pipeline remains fluid while maintaining the deterministic execution and explicit state management that Metaflow provides at scale. ### The Nature of ML and AI Iteration * ML and AI development is distinct from traditional software engineering because it involves large, mutable datasets and computationally expensive, stochastic processes. * State management is a primary concern in this domain, as reloading data or recomputing transformations for every minor code change creates a prohibitively slow feedback loop. * While notebooks like Jupyter or Marimo excel at preserving in-memory state for fast exploration, they often lead to "hidden state" problems and non-deterministic results due to out-of-order cell execution. ### Metaflow as a State-Aware Framework * Metaflow uses the `@step` decorator to define checkpoint boundaries where the framework automatically persists all instance variables as versioned artifacts. * The framework’s `resume` command allows developers to restart execution from a specific step, cloning previous state to avoid recomputing successful upstream tasks. * This architecture addresses notebook limitations by ensuring execution order is explicit and deterministic while making the state fully discoverable and versioned. ### Introducing Spin for Rapid Development * Spin is a new feature introduced in Metaflow 2.19 that further reduces the friction of the iterative development loop. * It aims to provide the near-instant feedback of a notebook environment while operating within the structure of a production-ready Metaflow workflow. * The tool helps developers manage the stateful nature of ML development, allowing for quick, incremental experimentation without losing continuity between code iterations. To improve data science productivity and reduce "waiting time" during the development phase, engineering teams should look to adopt Metaflow 2.19 and integrate Spin into their experimentation workflows.

Machine Learning AI Data Science Workflow Orchestration+3

netflixOct 25, 2025Original article

Post-Training Generative Recommenders with Advantage-Weighted Supervised Finetuning | by Netflix Technology Blog | Netflix TechBlog (opens in new tab)

Netflix is evolving its recommendation systems by moving beyond simple behavior imitation toward generative recommenders that better align with true user preferences. While generative models like HSTU and OneRec effectively capture sequential user patterns, they often struggle to distinguish between habitual clicks and genuine satisfaction. To bridge this gap, Netflix developed Advantage-Weighted Supervised Fine-tuning (A-SFT), a post-training method that leverages noisy reward signals to refine model performance without the need for complex counterfactual data. ### The Shift to Generative Recommenders * Modern generative recommenders (GRs), such as HSTU and OneRec, utilize transformer architectures to treat recommendation as a sequential transduction task. * The models are typically trained using next-item prediction, where the system learns to imitate the chronological sequence of a user’s activities. * A significant drawback of this "behavior cloning" approach is that it captures external trends and noise rather than long-term user satisfaction, potentially recommending content the user finished but did not actually enjoy. ### Barriers to Reinforcement Learning in RecSys * Traditional post-training methods used in Large Language Models, such as Proximal Policy Optimization (PPO) or Direct Preference Optimization (DPO), require counterfactual feedback that is difficult to obtain in recommendation contexts. * Because user sequences span weeks or years, it is impractical to generate and test hypothetical, counterfactual experiences for real-time user validation. * Reward signals in recommendation systems are inherently noisy; for instance, high watch time might indicate interest, but it can also be a result of external circumstances, making it an unreliable metric for optimization. ### Advantage-Weighted Supervised Fine-tuning (A-SFT) * A-SFT is a hybrid approach that sits between offline reinforcement learning and standard supervised fine-tuning. * The algorithm incorporates an advantage function to weight training examples, allowing the model to prioritize actions that lead to higher rewards while filtering out noise from the reward model. * This method is specifically designed to handle high-variance reward signals, using them as directional guides rather than absolute truth, which prevents the model from over-exploiting inaccurate data. * Benchmarks against other representative methods show that A-SFT achieves superior alignment between the generative recommendation policy and the underlying reward model. For organizations managing large-scale recommendation engines, A-SFT offers a practical path to implementing post-training improvements. By focusing on advantage-weighted signals, developers can improve recommendation quality using existing implicit feedback—like watch time and clicks—without the infrastructure hurdles of online reinforcement learning.

Machine Learning AI Transformer Recommendation Systems+4