State of Routing in Model Serving (opens in new tab)
Netflix’s centralized ML serving platform provides a single, domain-independent API for model inference across personalized experiences and other use cases. Rather than exposing individual scoring functions, Netflix packages feature computation, preprocessing, inference, and postprocessing into self-contained model workflows. The core routing challenge is directing each request to the correct model version and serving cluster while keeping client services independent from model changes and infrastructure topology. ## Models as End-to-End Workflows - Netflix distinguishes **model serving** from traditional model inference: - Inference typically means `infer(features) -> score`. - Serving includes preprocessing, feature computation, optional trained components, and postprocessing. - Example workflows include: - Ranking titles for a personalized Continue Watching row using user, country, and device context. - Predicting payment fraud using user, country, and transaction details. - Models declare the facts they need, while the serving platform retrieves those facts from other microservices. - During offline training, Netflix’s ML fact store provides snapshots for bulk feature computation. - Calling services provide standard request context and domain-specific inputs, while the platform handles feature generation, model selection, and execution. ## Platform Design Principles - **Model innovation without client changes** - Client applications integrate with the platform once. - Model versions, A/B tests, additional experimental data, logging, and model selection remain hidden behind the platform API. - **Clients decoupled from model sharding** - Models run across multiple serving cluster shards, each with its own Virtual IP address. - Shard assignments can change based on traffic, SLAs, model architecture, and resource availability. - Clients should not need to track these VIP changes. - **Flexible traffic routing** - Routing must support A/B allocations, gradual traffic shifts, new model versions, new VIPs, and client-specific overrides. - Safe lifecycle management requires support for shadow deployments, canaries, rollbacks, and migrations. ## Switchboard: Context-Aware Routing - Generic API gateways and service-mesh proxies did not satisfy Netflix’s requirements. - Netflix needed: - Native integration with its experimentation platform. - gRPC support. - Routing based on rich, domain-specific request context. - Model-specific rollout and migration controls. - Netflix built **Switchboard**, a custom proxy layer handling more than one million requests per second. - Switchboard is the mandatory entry point for clients and: - Routes requests to the appropriate model based on request context. - Applies configured context enrichment before invoking the model. - Hides model locations and infrastructure changes from client services. ## Objective Abstraction - Every request must provide an **Objective**, an enumeration defined by the serving platform. - The excerpt introduces Objectives as a central abstraction for identifying the business purpose of a serving request, but the supplied text ends before describing its full roles. Netflix’s approach is to centralize routing, experimentation, and model execution behind one stable API. This allows client applications to evolve independently while researchers can iterate on models and safely manage large-scale production rollouts.
