KernelEvolve: How Meta’s Ranking Engineer Agent Optimizes AI Infrastructure (opens in new tab)
KernelEvolve is Meta’s agentic system for automating the creation and optimization of low-level AI kernels across diverse hardware. It treats kernel tuning as a search problem rather than one-shot code generation, evaluating hundreds of alternatives with profiling and diagnostics. The system reduces optimization work from weeks to hours and has delivered over 60% higher inference throughput for an Ads model on NVIDIA GPUs and over 25% higher training throughput on Meta’s MTIA chips. ## Kernel Optimization at Meta - AI models rely on optimized kernels that translate high-level operations into hardware-specific instructions. - Meta runs models across NVIDIA GPUs, AMD GPUs, custom MTIA accelerators, and CPUs. - Production workloads require many custom operators beyond standard GEMMs and convolutions available in vendor libraries. - Kernels must be developed and tuned for each combination of: - Hardware type and generation - Model architecture - Operator type ## The Challenge of Hardware Heterogeneity - NVIDIA, AMD, MTIA, and CPU platforms differ in: - Memory architectures and hierarchies - Instruction sets - Execution models - Supported numeric data types - A kernel optimized for one platform may perform poorly or fail on another. - Hardware generations also require new optimization strategies. Meta’s MTIA roadmap includes four generations, from MTIA 300 through MTIA 500, in two years. - Manual tuning by kernel specialists cannot keep pace with these changes. ## Increasing Model and Operator Complexity - Meta’s recommendation systems have evolved from embedding-based models to sequence models with attention, GEM, and LLM-scale models such as Meta Adaptive Ranking Model. - Each new model generation introduces operators that earlier systems did not require. - Multiple model families may be involved in a single ads-serving request. - As model architectures and operator inventories grow, the number of kernel configurations expands rapidly into the thousands. ## How KernelEvolve Works - KernelEvolve generates candidate implementations in languages and DSLs including: - Triton, Cute DSL, and FlyDSL - CUDA, HIP, and MTIA C++ - A dedicated job harness compiles, runs, profiles, and evaluates each candidate. - Performance results, correctness checks, and diagnostic information are fed back to the LLM. - The system continuously searches through hundreds of alternatives instead of stopping at the first plausible implementation. - Its automated workflow includes profiling, optimization, testing, and cross-hardware debugging. ## Results and Broader Impact - KernelEvolve improved Andromeda Ads inference throughput by more than 60% on NVIDIA GPUs. - It improved training throughput for an ads model by more than 25% on Meta’s MTIA silicon. - The system operates across both public and proprietary hardware. - In production, it optimizes code supporting trillions of daily inference requests. - By automating kernel development, Meta can enable new hardware and adapt to changing model architectures with substantially less engineering effort. KernelEvolve turns kernel development from a manual, expert-driven bottleneck into a continuous automated process. Its search-based approach is particularly valuable as Meta’s hardware portfolio and model architectures continue to diversify.