How Netflix Simplified Batch Compute with Kueue
Netflix replaced much of its custom Compute Managed Batch (CMB) queuing and scheduling logic with Kubernetes-native Kueue. The migration preserved the existing user experience while enabling features such as preemption, fair sharing, all-or-nothing scheduling, and topology-aware placement. Kueue now manages millions of batch workloads across Netflix’s Titus-based infrastructure. ## CMB and Titus Architecture - CMB manages workloads that run to completion using: - Hierarchical tenants - Priority-based ordering - Per-tenant capacity management - Workloads ultimately run on Titus, Netflix’s container platform. - Titus provides federation across multiple Kubernetes cells and shared capacity reservations, allowing CMB to interact with a unified endpoint. - CMB tenants are either: - **Internal tenants**, which organize child tenants but do not accept jobs - **Leaf tenants**, which accept jobs through associated queues - Capacity includes: - **Reserved capacity**, providing predictable resources within a tenant hierarchy - **Shared capacity**, a global pool that tenants can burst into - CMB enforced fair sharing only at admission time because it lacked preemption; admitted jobs ran to completion even when demand changed. ## Why Netflix Chose Kueue - CMB was developed before many Kubernetes batch features became available in open source. - Kueue provided capabilities Netflix had previously built or wanted to build, including: - Fair sharing - Hierarchical tenancy - Capacity management - Priority queues - Preemption - Unlike schedulers such as YuniKorn and Volcano, Kueue works with the existing Kubernetes scheduler rather than replacing it. - This allowed Netflix to retain Titus scheduling profiles and avoid inefficient job placement. - Kueue also supports: - Multi-tenant quotas across heterogeneous hardware - Native Kubernetes objects such as `Pod` and `Job` - Higher-level workloads such as `RayJob` and `RayCluster` - All-or-nothing admission and topology-aware scheduling ## Migrating CMB Workloads - The migration, called **Netflix Batch**, was designed to: - Require no changes from CMB users - Avoid regressions in launch rates and maximum throughput - Move queuing and scheduling responsibilities to Kueue - Kueue runs in enabled Titus cells, while a custom router and Titus federation direct workloads to the appropriate cell. - Tenant enrollment was exposed as a simple operator action in Netflix’s UI, making rollout and rollback straightforward. - Internally, the migration mapped: - CMB internal tenants to Kueue **Cohorts** - Leaf tenants to **ClusterQueues** and **LocalQueues** - Capacity configurations to Kueue **resource flavors** and **nominal quotas** ## Lessons from the Rollout - Maintaining API compatibility reduced customer disruption and allowed Netflix to replace backend components incrementally. - Migrating the largest and most complex customer early exposed problems sooner and increased confidence in the broader rollout. - The production migration took approximately four weeks. - Kueue required substantially higher QPS, burst, and `groupKindConcurrency` settings than its defaults. - Netflix validated these settings early through load tests in an environment modeled on Titus. ## Kueue in Production - Kueue is fully deployed at Netflix and manages millions of batch workloads. - Netflix is extending its use to additional Titus batch workloads. - Fair sharing and preemption are being expanded to improve utilization of reserved capacity. - Netflix’s experience is also informing other internal Kubernetes-native systems, including training infrastructure. Netflix’s migration demonstrates that a batch platform can adopt Kubernetes-native scheduling incrementally without forcing users to change APIs or abandoning existing placement infrastructure. For organizations with mature custom systems, preserving the external contract while delegating queueing and admission to Kueue offers a lower-risk path to modern features and simpler long-term operations.
Read original(opens in new tab)