metaApr 2, 2026

KernelEvolve: How Meta’s Ranking Engineer Agent Optimizes AI Infrastructure (opens in new tab)

KernelEvolve is Meta’s agentic system for automating the creation and optimization of low-level AI kernels across diverse hardware. It treats kernel tuning as a search problem rather than one-shot code generation, evaluating hundreds of alternatives with profiling and diagnostics. The system reduces optimization work from weeks to hours and has delivered over 60% higher inference throughput for an Ads model on NVIDIA GPUs and over 25% higher training throughput on Meta’s MTIA chips. ## Kernel Optimization at Meta - AI models rely on optimized kernels that translate high-level operations into hardware-specific instructions. - Meta runs models across NVIDIA GPUs, AMD GPUs, custom MTIA accelerators, and CPUs. - Production workloads require many custom operators beyond standard GEMMs and convolutions available in vendor libraries. - Kernels must be developed and tuned for each combination of: - Hardware type and generation - Model architecture - Operator type ## The Challenge of Hardware Heterogeneity - NVIDIA, AMD, MTIA, and CPU platforms differ in: - Memory architectures and hierarchies - Instruction sets - Execution models - Supported numeric data types - A kernel optimized for one platform may perform poorly or fail on another. - Hardware generations also require new optimization strategies. Meta’s MTIA roadmap includes four generations, from MTIA 300 through MTIA 500, in two years. - Manual tuning by kernel specialists cannot keep pace with these changes. ## Increasing Model and Operator Complexity - Meta’s recommendation systems have evolved from embedding-based models to sequence models with attention, GEM, and LLM-scale models such as Meta Adaptive Ranking Model. - Each new model generation introduces operators that earlier systems did not require. - Multiple model families may be involved in a single ads-serving request. - As model architectures and operator inventories grow, the number of kernel configurations expands rapidly into the thousands. ## How KernelEvolve Works - KernelEvolve generates candidate implementations in languages and DSLs including: - Triton, Cute DSL, and FlyDSL - CUDA, HIP, and MTIA C++ - A dedicated job harness compiles, runs, profiles, and evaluates each candidate. - Performance results, correctness checks, and diagnostic information are fed back to the LLM. - The system continuously searches through hundreds of alternatives instead of stopping at the first plausible implementation. - Its automated workflow includes profiling, optimization, testing, and cross-hardware debugging. ## Results and Broader Impact - KernelEvolve improved Andromeda Ads inference throughput by more than 60% on NVIDIA GPUs. - It improved training throughput for an ads model by more than 25% on Meta’s MTIA silicon. - The system operates across both public and proprietary hardware. - In production, it optimizes code supporting trillions of daily inference requests. - By automating kernel development, Meta can enable new hardware and adapt to changing model architectures with substantially less engineering effort. KernelEvolve turns kernel development from a manual, expert-driven bottleneck into a continuous automated process. Its search-based approach is particularly valuable as Meta’s hardware portfolio and model architectures continue to diversify.

profilingmachine-learningdeep-learningcuda+4

datadogApr 4, 2024

.NET Continuous Profiler: Exception and lock contention (opens in new tab)

Datadog’s .NET continuous profiler can diagnose performance problems that CPU and wall-time profiling may miss: excessive exceptions and lock contention. Exceptions consume significant CPU and latency, while locks increase request latency through waiting rather than active computation. By collecting exception details and measuring contention duration with low overhead, the profiler helps developers identify the code and runtime conditions responsible. ## Exception Profiling - The CLR notifies the profiler through `ICorProfilerCallback::ExceptionThrown`, providing the exception’s `ObjectID`. - `ExceptionProvider::OnExceptionThrown` extracts details such as: - Exception type - Thread ID - Source location - The profiler maps the exception object to its `ClassID` using `ICorProfilerInfo::GetClassFromObject`. - Type names are resolved and cached by the `FrameStore`. - Exception messages require reading the private `System.Exception._message` field: - The profiler locates `System.Exception` in `mscorlib` or `System.Private.CoreLib`. - `GetModuleMetaData` provides access to assembly metadata. - `FindTypeDefByName` locates the type definition. - `GetClassFromTokenAndTypeArgs` obtains its `ClassID`. - `GetClassLayout` identifies field offsets. - `FindField` locates `_message`. - `GetStringLayout2` provides the string buffer and length needed to read the message. - Collecting exception counts by type, message, and call site makes it possible to replace expensive exception-driven control flow with cheaper checks such as `TryParse`. ## Lock Contention Monitoring - Standard .NET monitoring exposes contention counts, but generally not how long threads waited or where the contention originated. - The CLR emits: - `ContentionStart` when a thread begins waiting - `ContentionStop` when it acquires the lock - On .NET Framework, contention duration is calculated from timestamps recorded for each thread because `ContentionStop` does not include the duration. - Since .NET 8, `ContentionStart` includes the lock’s `ObjectID` and the ID of the thread holding it, allowing the profiler to identify the blocking thread. - .NET Framework exposes counters such as `Contention Rate / Sec` and `Total # of Contentions`; .NET Core provides `monitor-lock-contention-count` through `dotnet-counters`. - These counters alone do not reveal the duration or cause of waits. ## Consuming CLR Events - Since .NET 5, profilers can synchronously receive CLR events through `ICorProfilerCallback10::EventPipeEventDelivered`. - Datadog’s `ClrEventParser` interprets event payloads based on event IDs and keywords. - The parsed duration is passed to `ContentionProvider::OnContention`. - Runtime differences require version-specific handling because event payloads are not identical across .NET Framework and .NET Core. The practical recommendation is to profile both exception frequency and lock-wait duration, rather than relying only on CPU usage or contention counters. This reveals inefficient exception-based logic and identifies locks—and, on newer runtimes, the threads holding them—that materially affect application latency.

profilingdotnetexception-handlinglock-contention+4

datadogFeb 28, 2022

Profiling improvements in Go 1.18 (opens in new tab)

Go 1.18 introduced major profiling improvements alongside features such as generics and fuzzing. Its Linux CPU profiler became substantially more accurate on multicore systems by addressing dropped `SIGPROF` signals, and profiler labels received an important correctness fix. These changes strengthened Go’s ability to connect continuous profiling data with tracing systems such as Datadog. ## More Accurate Linux CPU Profiling - Earlier Go versions used `setitimer(2)` to request a `SIGPROF` signal every 10 ms of CPU time. - On busy multicore systems, Linux could generate multiple signals within a single kernel “jiffy” window, but standard POSIX signals do not queue. - As a result, many signals were dropped: - A service using 20 CPU cores might generate roughly 2,000 profiling signals per second. - Its Go profile could contain only about 240 samples per second. - Linux’s software clock, commonly operating at 250 Hz, could only measure CPU time in roughly 4 ms intervals. This caused signal bursts and undercounted CPU usage. - `setitimer(2)` also distributed process-directed signals unevenly across threads, creating additional profiling bias. ## Combining `timer_create` and `setitimer` - `timer_create(2)` provided more reliable per-thread signal accounting and avoided most of the signal-dropping and thread-bias problems. - Its drawback was that the profiler needed awareness of every thread, including threads created independently by cgo code. - The Go 1.18 fix combined both timer mechanisms: - The signal handler identifies the signal source. - Signals from inferior sources are discarded. - The implementation accounts for short-lived threads and cgo edge cases. - The work originated from investigations into Go issues GH 35057 and GH 14434 and was developed through collaboration between contributors and Go maintainers. ## Profiler Label Correctness - Profiler labels, also called pprof labels or tags, associate key/value metadata with goroutines. - Labels are inherited by child goroutines and appear in CPU and goroutine profiles, enabling profiles to be filtered by request, service, or trace metadata. - Testing at Datadog revealed that some stack samples were missing labels they should have carried. - The cause was a CPU profiler lookup using the wrong goroutine reference. - The fix changed the profiler to use `gp.m.curg`, the thread’s actual current goroutine, rather than relying on `gp`, which can differ in certain runtime situations. Go 1.18’s profiling changes made CPU measurements more trustworthy on Linux and improved the accuracy of metadata attached to profile samples. Together, they provided a stronger foundation for correlating Go profiling with distributed tracing.

profilinggolinuxcpu-profiling+4

datadogSep 30, 2021

How we optimized our Akka application using Datadog’s Continuous Profiler (opens in new tab)

Datadog discovered that an unexpected 20–30% CPU overhead came from Akka’s use of `ForkJoinPool`, not from the log-processing code they initially suspected. Profiling showed that an actor handling intermittent latency metrics repeatedly caused worker threads to park and unpark. Moving that actor to a busier, more stable dispatcher reduced CPU usage by about 30%. ## How Profiling Revealed the Problem - Datadog used Akka to parallelize log-event processing through actors and dispatchers. - An optimization to log parsing produced little improvement, despite reducing parsing CPU time. - Continuous Profiler flame graphs showed increased CPU time in: - `ForkJoinPool.scan()` - `Unsafe.park()` - Thread-level analysis revealed that the default Akka dispatcher—not the expected dedicated work pool—was responsible. - Many of the affected threads were executing a latency-reporting actor. ## Why `ForkJoinPool` Was Consuming CPU - `ForkJoinPool` dynamically manages worker threads: - It creates threads when work increases. - It suspends idle threads with `Unsafe.park()`. - It resumes them with `Unsafe.unpark()`. - It terminates idle workers after a default period. - The latency actor received a few hundred events per second, processed them within milliseconds, and then remained idle until the next batch. - Because the pool allowed up to 32 threads—matching the number of processor cores—it repeatedly activated and suspended many workers. - These frequent parking and unparking operations created short CPU spikes and excessive time in `ForkJoinPool.scan()`. ## The Dispatcher Change - The team moved the latency actor from Akka’s default dispatcher to the main `work-dispatcher`. - The work dispatcher already handled a steadier stream of log-processing tasks, keeping its worker threads active. - This required only a configuration change assigning the actor to `work-dispatcher`. - CPU usage fell by roughly 30% across services. - The default dispatcher also shrank from 32 threads to 2, confirming that unnecessary thread activation was the cause. ## Recommendations - Monitor CPU time spent in `ForkJoinPool.scan()`, especially when it exceeds roughly 10–15%. - Limit the number of Akka actor instances. - Set a suitable maximum thread count for each pool. - Reduce the number of separate thread pools where practical. - Use task queues to absorb frequent, short-lived workload spikes. - Aim to keep the number of active `ForkJoinPool` workers relatively stable and avoid repeated parking and unparking.

profilingdatabase-designjavascala+4

datadogOct 7, 2020

How we wrote a Python profiler (opens in new tab)

Datadog built a Python continuous profiler because Python lacked Java-style, always-on production profiling tools. The post argues that deterministic profilers such as `cProfile` impose too much overhead for continuous use, while statistical profiling can provide representative performance data with minimal disruption. Datadog’s profiler addresses this through modular collectors, recording, scheduling, and data export. ## Profiling Versus Tracing - **Profiling** measures resource consumption such as CPU time and memory allocation to reveal performance problems. - **Tracing** records individual operations—such as SQL queries or HTTP requests—within a request timeline. - Tracing explains request latency, but profiling provides deeper insight into code-level execution and operating-system resource usage. ## Limitations of Deterministic Python Profilers - Python’s `cProfile`, available since CPython 2.5, records every function call and the time spent in each call. - It can provide a complete execution flow, but its usefulness depends heavily on code structure: - A program built around a few large functions produces little actionable detail. - A program containing thousands of functions can incur two- or three-times runtime overhead. - This overhead makes deterministic profiling unsuitable for always-on production environments. ## Why Profile in Production? - Optimizing without profiling is essentially guessing; real workloads often differ from development environments. - Production systems vary from developer machines in hardware, concurrency, input data, and workload behavior. - Continuous profiling captures how an application actually consumes resources under authentic conditions. - These requirements lead to statistical rather than deterministic profiling. ## Statistical Profiling in Python - Statistical profilers sample program activity periodically instead of recording every function call. - Individual short-lived calls may be missed, but repeated sampling over hours produces a reliable picture of resource consumption. - Lower overhead allows the application to run closer to its normal, unprofiled behavior. - Datadog evaluated numerous open-source Python profilers but found limitations involving platform support, collected data, or presentation-focused designs. - The team therefore developed its own statistical profiler, incorporating ideas from the tools it studied. ## Datadog Python Profiler Design The profiler was designed around three constraints: - Keep runtime overhead as low as possible. - Make deployment simple. - Support common operating systems and environments. Its architecture, inspired by the JDK Flight Recorder, consists of: - **Collectors:** Gather data such as CPU usage and memory allocation. - **Recorder:** Stores events produced by collectors. - **Exporter:** Sends profiling data outside the application. - **Scheduler:** Invokes components at appropriate intervals, such as exporting data every 60 seconds. - **Profiler:** Provides the high-level interface used by applications. ## Stack Collection - The stack collector is the primary built-in collector. - It wakes 100 times per second and captures the execution stack of every Python thread. - For each thread, it gathers information including: - The currently executing function - CPU time consumed - Exceptions being handled - The collector monitors the time required to inspect the application so it can control and limit its own CPU overhead. A statistical profiler with low overhead is the appropriate foundation for continuous production profiling, giving teams evidence about real application behavior without substantially changing that behavior.

profilingdatadogpythonjava+4