datadogJul 22, 2026

Unbiased Java CPU profiling with JFR in JDK 25 (opens in new tab)

Java Flight Recorder (JFR) provides low-overhead, production-safe diagnostics, but its `ExecutionSample` event can produce biased CPU profiles because it samples JVM-observed runnable threads rather than strictly measuring CPU time. For CPU-bound workloads—particularly reactive applications—this may obscure the real hotspots. Modern profilers therefore combine JFR with JVMTI, `SIGPROF`, `AsyncGetCallTrace`, and JVM-internal techniques, while the Java ecosystem works toward a supported CPU-sampling mechanism. ## How Sampling Profilers Work - Continuous profilers repeatedly capture stack traces and aggregate them to reveal recurring behavior. - CPU profilers commonly sample at fixed intervals, such as every 20 milliseconds. - **CPU-time sampling** highlights code actively consuming processor cycles. - **Wall-clock sampling** reveals latency sources, including I/O waits, lock contention, and blocked threads. - Other profilers trigger on events such as allocations, garbage collection, thread parking, or lock contention. - Regardless of the trigger, profilers capture a stack, associate it with an event, and aggregate the results. ## Limitations of JFR’s `ExecutionSample` - JFR is integrated into the JVM and designed for low-overhead, always-on production use. - Its `ExecutionSample` event captures stacks from a rotating subset of runnable threads. - CPU-heavy threads tend to appear more often, but samples are not strictly proportional to actual CPU consumption. - This can lead to incomplete or biased results on CPU-saturated systems. - Reactive applications are a notable example: their scheduling behavior can cause thread CPU usage and hotspots to be underrepresented. ## CPU Sampling with `AsyncGetCallTrace` - JVMTI agents can use operating-system signals such as `SIGPROF` to sample threads according to CPU time. - The signal handler invokes HotSpot’s `AsyncGetCallTrace` to walk Java stacks asynchronously. - This approach avoids safepoint bias and can capture stacks during arbitrary execution states. - Tools such as `async-profiler` use this technique to produce profiles that more closely match actual CPU usage. - The drawback is that `AsyncGetCallTrace` is an unsupported internal JVM API. - Under heavy load, it can occasionally fault, requiring profilers to add extensive safeguards. - Datadog also uses **vmstructs walking**, which reads internal JVM metadata to recover stack and runtime information unavailable through standard APIs. ## The Safety–Accuracy Tradeoff - JFR offers stability, structured runtime telemetry, and low overhead. - `AsyncGetCallTrace` and vmstructs walking offer more accurate CPU sampling. - Relying on JVM internals creates maintenance and reliability risks because those interfaces are not officially stable. - Consequently, modern profilers combine JFR with unsupported sampling mechanisms rather than choosing only one approach. ## Toward a Supported CPU Profiling Event - Datadog, SAP, Amazon, and OpenJDK contributors recognized that this limitation affected the broader profiling ecosystem. - JFR was already the natural foundation for safe, continuous profiling. - The missing capability was a first-class CPU sampling event that could provide accurate CPU-based results without depending on unsupported JVM internals. - Datadog participated in OpenJDK discussions to explain why existing sampling was insufficient and to help improve the platform’s profiling foundation. Ultimately, accurate production CPU profiling requires both JFR’s safety and CPU-time-based sampling. A supported JFR CPU profiling event would remove the ecosystem’s dependence on fragile JVM internals while preserving the low-overhead behavior needed for continuous use.

cpu-profilingjavajvmjava-flight-recorder+3

datadogFeb 13, 2024

.NET Continuous Profiler: CPU and wall time profiling (opens in new tab)

Datadog’s .NET profiler uses low-overhead thread sampling to collect CPU and wall-time profiles suitable for production. CPU profiling measures time spent running on a processor, while wall-time profiling captures delays caused by I/O, locks, or scheduling. The implementation tracks managed threads, walks their stacks, aggregates samples, and periodically uploads profiles, with platform-specific optimizations for Windows and Linux. ## CPU and Wall-Time Profiling - CPU profiling identifies code consuming processor cycles. - Wall-time profiling finds slow methods regardless of whether threads are running, blocked, or waiting for I/O. - ETW, Linux `perf`, and .NET enter/leave hooks provide profiling data but impose too much overhead or require elevated privileges. - Datadog instead samples threads at intervals, captures their call stacks, and assigns durations to the samples. ## Managed Thread Sampling - The profiler tracks managed threads created through: - The thread pool - `Task` and `async/await` - Explicit `Thread` instances - `ICorProfilerCallback` notifications such as `ThreadCreated` and `ThreadDestroyed` maintain the `ManagedThreadList`. - `StackSamplerLoop` captures stacks and sends raw samples to `CpuTimeProvider` and `WallTimeProvider`. - `SamplesCollector` aggregates provider data, while a shared Rust exporter uploads profiles to Datadog every minute. - An early C# implementation caused native worker threads entering managed code to appear as application threads. Removing that implementation eliminated accidental self-sampling. ## Native Runtime Threads - Server-mode garbage collection uses native threads that can compete with application threads for CPU. - Because the .NET profiling API does not expose these threads directly, the profiler identifies them by their .NET 5 names: - `.NET Server GC` - `.NET BGC` - Their CPU usage is displayed under a Garbage Collector frame in the flame graph. ## CPU Sampling and Performance Optimizations - Every 10 milliseconds, the profiler searches for runnable managed threads, skipping up to 64 non-runnable threads. - `IsRunning` determines whether a thread is executing on a CPU: - Windows uses `NtQueryInformationThread`. - Linux reads `/proc/self/task/<tid>/stat`. - CPU sample duration is calculated from the difference between the thread’s current and previously recorded CPU consumption. - The initial Linux implementation used `std::ifstream` and `std::getline`, allocating an 8 KB buffer for each sample. - Replacing them with lower-level C file-reading code eliminated allocations and reduced CPU usage. The original approach consumed nearly 500 MB of allocations and about 2% of total CPU in testing. ## Wall-Time Profiling and Code Hotspots - Wall-time profiles work with Datadog tracing to explain why requests are slow. - The tracer provides the profiler with a span ID when a thread handles a request. - To avoid repeated expensive P/Invoke calls, the profiler exposes a memory location that the tracer can update directly. - Because short requests may finish before their thread is sampled, the profiler samples an additional group of ten span-associated threads beyond the initial five. - The duration between consecutive samples of a thread is attributed to the later sample, so heavily threaded applications produce longer individual wall-time sample durations.

cpu-profilingdatadogrustlinux+4

datadogJan 9, 2024

.NET Continuous Profiler: Under the hood (opens in new tab)

Datadog’s .NET profiler is designed for continuous, low-overhead production monitoring rather than occasional diagnostic runs. It collects CPU, wall time, exceptions, lock contention, and allocation data, aggregates it into compact `.pprof` files, and links profiles to traces and services through runtime metadata. The post introduces the architecture and emphasizes preserving application performance as a central design requirement. ## What a Continuous Profiler Does - Profiling analyzes runtime performance and method call stacks. - It complements APM, which focuses on request latency, throughput, and errors. - The profiler also measures: - CPU usage - Wall time and method duration - Exceptions - Lock contention - Memory allocations and potential leaks - Unlike tools such as PerfView, dotTrace, dotMemory, and Visual Studio profilers, Datadog’s profiler is intended to run continuously in production with negligible overhead. - Continuous profiling avoids the need to recreate production traffic, security settings, hardware, and load in a separate environment. ## Datadog’s .NET Profiler Architecture - The profiler is composed of specialized profilers for different resource types. - Each profiler includes: - A sampler that collects raw data - A provider that exposes the collected samples - An aggregator combines samples from all profilers. - An exporter serializes the data into Google’s `.pprof` format and uploads it through the Datadog Agent. - Datadog’s backend processes the profiles for visualization and analysis. ## Sample Aggregation and Storage Each sample contains: - A call stack made up of method frames - Key-value labels, such as thread identifiers - A numeric value vector representing measurements like CPU consumption or wall time Samples with identical call stacks and labels are merged, and their numeric values are added together. This reduces duplication and produces smaller profile files—for example, repeated exceptions from the same code path and thread can be stored as one aggregated sample. The aggregation and `.pprof` serialization code is implemented in Rust and shared across Datadog’s Ruby, PHP, and other runtime profilers. ## Connecting Profiles to Traces and Services - Each uploaded profile includes process ID, host name, and runtime ID metadata. - The runtime ID uniquely identifies a .NET service running within a process. - This is important because a single .NET process can host multiple services, such as separate IIS applications running in different AppDomains. - The tracer communicates the mapping between runtime IDs, AppDomains, and service names. - Service names come from `DD_SERVICE`; if it is unset, the process name is used. - Datadog sends one profile per runtime ID every minute, so multiple profiles from one process may share a timestamp while representing different services. - Runtime IDs allow the backend to associate profiles with the correct traces and spans. ## Making .NET Call Stacks Easier to Read The .NET profiling API can expose compiler- and runtime-generated names that differ from the original source code. Datadog rewrites these frames to make visualized call stacks more understandable. - Constructors named `.ctor` are displayed using the class name. - Compiler-generated anonymous methods are rendered as the enclosing method followed by `_AnonymousMethod`. - Lambdas and local methods use an enclosing-method name with the `_Lambda` suffix. - Nested named methods such as `<DefiningMethodName>g__InnerMethodName|yyy_zzz` are displayed as `DefiningMethodName.InnerMethodName`. - Compiler-generated state-machine methods such as `MoveNext` are mapped back to the original source-level type and method names. ## Native and Managed Implementation Considerations - The team considered using Microsoft’s `TraceEvent` NuGet package to receive and parse CLR events in C#. - That approach would execute managed profiling code on the same CLR as the application being profiled. - Allocations made by the profiler could therefore increase garbage-collector pressure. - The post begins discussing how this performance concern influenced the implementation, but the provided excerpt ends before that design is explained. A production profiler must not only collect useful data but also minimize the memory and CPU costs of collecting it. Datadog’s architecture addresses this through specialized samplers, aggregation, compact serialization, runtime-aware trace association, and source-oriented call-stack cleanup.

cpu-profilingdatabase-designdatadogdotnet+4

datadogJan 31, 2023

Performance improvements in the Datadog Agent metrics pipeline | Datadog (opens in new tab)

Datadog engineers recently optimized the Datadog Agent's metric processing pipeline to achieve higher throughput and lower CPU overhead. By identifying that metric context generation—the process of creating unique keys for metrics—was a primary bottleneck, they implemented a series of algorithmic changes and Go runtime optimizations. These improvements allow the Agent to process significantly more metrics using the same computational resources. ### Identifying Bottlenecks via CPU Profiling * Developers utilized Go’s native profiling tools to capture CPU usage during high-volume metric ingestion via DogStatsD. * Flamegraph analysis revealed that the `addSample` and `trackContext` functions were the most CPU-intensive components of the pipeline. * The profiling data specifically pointed to tag sorting and deduplication as the underlying operations consuming the most processing time. ### The Challenges of Metric Context Generation * The Agent must generate a unique hash (context) for every metric received to address it within a hash table in RAM. * To ensure the same metric always generates the same key, the original algorithm required sorting all tags and ensuring their uniqueness. * The computational cost of sorting lists repeatedly for every incoming message created a performance ceiling for the entire metrics pipeline. ### Specialization and Runtime Optimization * **Algorithmic Specialization:** The team implemented specialized sorting logic that adjusts based on the number of tags, optimizing the "hot path" for the most common metric structures. * **Hashing Efficiency:** Micro-benchmarks identified Murmur3 as the most efficient hash implementation for balancing speed and collision resistance in this use case. * **Leveraging Go Runtime:** The team transitioned from 128-bit hashes to 64-bit metric contexts. This change allowed the Agent to utilize Go's internal `mapassign_fast64` and `mapaccess2_fast64` functions, which provide optimized map operations for 64-bit keys. ### Redesigning for Performance * The original design followed a rigid "hash metric name -> sort tags -> deduplicate tags -> iterative hash" workflow. * Recognizing that sorting was the primary architectural bottleneck, the team moved toward a new design intended to minimize or eliminate the overhead of traditional list sorting during context generation. To achieve similar performance gains in high-throughput Go applications, developers should profile their applications under realistic load and look for opportunities to leverage runtime-specific optimizations, such as using 64-bit map keys to trigger specialized compiler paths.

cpu-profilingdatabase-designgodatadog-agent+4

datadogJan 31, 2023

Performance improvements in the Datadog Agent metrics pipeline (opens in new tab)

The Datadog Agent needed to process more metrics without increasing CPU usage. Profiling showed that generating unique metric contexts—especially sorting and deduplicating tags—was a major bottleneck. Datadog improved throughput through specialized sorting paths, faster hashing, and a more efficient context-storage design. ## Identifying the Bottleneck - Datadog uses Go’s CPU and memory profiling tools to optimize the Agent’s metrics pipeline. - Profiles were captured while Agents processed large volumes of DogStatsD metrics, ensuring the results reflected real workload pressure. - Flamegraphs showed that `addSample` and `trackContext` consumed the most CPU. - Sorting-related functions, including `util.SortUniqInPlace` and `sort`, were significant contributors to that cost. ## How Metric Contexts Work - Each received metric is assigned a metric context that uniquely identifies it in an in-memory hash table. - The context must incorporate: - The metric name - Tags included in the DogStatsD message - Container-generated tags - The context is computed as a hash, so it must be fast while minimizing collisions. - Tags must be consistently ordered so the same metric always produces the same context. - The original implementation sorted tags and removed duplicates, making sorting a recurring CPU expense. ## Specialized Sorting - Performance varied according to the number of tags attached to a metric. - Datadog introduced specialized sorting paths based on tag count. - This allowed common cases to use more efficient algorithms while retaining correct ordering and deduplication. ## Faster Hashing and Map Access - Micro-benchmarks compared hash functions according to speed and uniqueness. - Murmur3 performed best for Datadog’s requirements. - Datadog also changed metric contexts from 128-bit to 64-bit hashes. - A 64-bit hash still provided sufficient collision resistance for the use case and enabled Go runtime optimizations: - `runtime.mapassign_fast64` - `runtime.mapaccess2_fast64` - These optimized map operations improved both context storage and metric sampling performance. ## Redesigning the Algorithm - Sorting served two purposes: producing an ordered tag list and helping deduplicate tags. - Because sorting was the largest bottleneck, Datadog began exploring a design that could address these responsibilities more efficiently rather than relying on a single general-purpose sort. The practical lesson is to profile under realistic load, optimize the hottest paths, and combine targeted specialization, benchmark-driven implementation choices, and data-structure redesign to increase throughput without adding CPU capacity.

cpu-profilingperformance-optimizationgodatadog-agent+3

datadogFeb 28, 2022

Profiling improvements in Go 1.18 (opens in new tab)

Go 1.18 introduced major profiling improvements alongside features such as generics and fuzzing. Its Linux CPU profiler became substantially more accurate on multicore systems by addressing dropped `SIGPROF` signals, and profiler labels received an important correctness fix. These changes strengthened Go’s ability to connect continuous profiling data with tracing systems such as Datadog. ## More Accurate Linux CPU Profiling - Earlier Go versions used `setitimer(2)` to request a `SIGPROF` signal every 10 ms of CPU time. - On busy multicore systems, Linux could generate multiple signals within a single kernel “jiffy” window, but standard POSIX signals do not queue. - As a result, many signals were dropped: - A service using 20 CPU cores might generate roughly 2,000 profiling signals per second. - Its Go profile could contain only about 240 samples per second. - Linux’s software clock, commonly operating at 250 Hz, could only measure CPU time in roughly 4 ms intervals. This caused signal bursts and undercounted CPU usage. - `setitimer(2)` also distributed process-directed signals unevenly across threads, creating additional profiling bias. ## Combining `timer_create` and `setitimer` - `timer_create(2)` provided more reliable per-thread signal accounting and avoided most of the signal-dropping and thread-bias problems. - Its drawback was that the profiler needed awareness of every thread, including threads created independently by cgo code. - The Go 1.18 fix combined both timer mechanisms: - The signal handler identifies the signal source. - Signals from inferior sources are discarded. - The implementation accounts for short-lived threads and cgo edge cases. - The work originated from investigations into Go issues GH 35057 and GH 14434 and was developed through collaboration between contributors and Go maintainers. ## Profiler Label Correctness - Profiler labels, also called pprof labels or tags, associate key/value metadata with goroutines. - Labels are inherited by child goroutines and appear in CPU and goroutine profiles, enabling profiles to be filtered by request, service, or trace metadata. - Testing at Datadog revealed that some stack samples were missing labels they should have carried. - The cause was a CPU profiler lookup using the wrong goroutine reference. - The fix changed the profiler to use `gp.m.curg`, the thread’s actual current goroutine, rather than relying on `gp`, which can differ in certain runtime situations. Go 1.18’s profiling changes made CPU measurements more trustworthy on Linux and improved the accuracy of metadata attached to profile samples. Together, they provided a stronger foundation for correlating Go profiling with distributed tracing.

cpu-profilinggolinuxprofiling+4