Modernizing the Meta Ads Service With an Open-Source Kernel Scheduler
At Meta’s scale, small latency regressions can materially affect ad relevance, ranking, and revenue. When Linux kernel 6.9’s EEVDF scheduler reduced ad-serving performance, Meta used sched_ext to deploy a workload-specific BPF scheduler without modifying the kernel. The solution reduced p99 ads-retrieval latency by 28%, saved 3.28 MW of power, increased ads ranked by 1.1%, and enabled further improvements through rapid user-space updates. ## Why Ads Latency Matters - Meta’s ads platform processes more than 5 million requests per second, or over 400 billion requests daily. - Lower p99 latency allows more relevant ads to be retrieved and ranked within each request. - General-purpose schedulers such as CFS and EEVDF balance CPU usage without understanding which threads are most important to ad delivery. - Ads-specific scheduling can prioritize work on the critical request path while deferring less-sensitive tasks. ## The Kernel Upgrade Problem - During a move from Linux 6.4 to 6.9, Meta found that EEVDF introduced a latency regression. - The regression reduced the number of ads ranked in responses. - Some servers had to remain on Linux 6.4, creating operational fragmentation and technical debt. - sched_ext provided a way to address the regression without waiting for another kernel release. ## How sched_ext Customizes Scheduling - sched_ext is an upstream, BPF-based framework that entered Linux 6.12. - It lets developers implement scheduling policies in BPF programs responding to events such as: - Thread wake-ups - Run-queue insertion - Dispatching the next thread - CPU idle-state transitions - Meta’s policy divides CPUs into two dynamically sized pools: - Latency-critical request-path threads - Less latency-sensitive background work - Keeping related work on the same CPUs improves L3 cache locality and reduces DRAM access. - The scheduler is loaded by a user-space binary, so new policies can be deployed by restarting the scheduler process rather than rebuilding the kernel. ## Performance and Operational Results The initial deployment on the largest ads-serving server type achieved: - 28% lower p99 latency on the ads retrieval path - 1.1% more weighted ads ranked - 3.28 MW of fleet-wide power savings Two subsequent policy updates produced additional gains: - A further 60% reduction in service p99 latency - 18% fewer timeout errors on the critical path - Delivery in days instead of the months typically required for kernel changes ## From Fix to Optimization Platform - sched_ext gives Meta an independent scheduling-development path alongside upstream Linux evolution. - BPF updates support rapid experimentation with: - Cache-aware thread placement - ROI-based executor routing - NUMA-aware scheduling - Because sched_ext is upstream, other organizations can implement workload-specific policies without maintaining a Linux kernel fork. - Meta plans to use application-level hints, such as request importance, to adjust scheduling slices and queue priority dynamically. sched_ext demonstrates that application-aware scheduling can produce measurable business, latency, and energy benefits. For workloads with priorities that general-purpose schedulers cannot see, an extensible BPF-based scheduler offers a practical way to optimize continuously without coupling improvements to kernel release cycles.
Read original(opens in new tab)