.NET Continuous Profiler: Under the hood

Datadog’s .NET profiler is designed for continuous, low-overhead production monitoring rather than occasional diagnostic runs. It collects CPU, wall time, exceptions, lock contention, and allocation data, aggregates it into compact .pprof files, and links profiles to traces and services through runtime metadata. The post introduces the architecture and emphasizes preserving application performance as a central design requirement.

What a Continuous Profiler Does

• Profiling analyzes runtime performance and method call stacks.
• It complements APM, which focuses on request latency, throughput, and errors.
• The profiler also measures:
  • CPU usage
  • Wall time and method duration
  • Exceptions
  • Lock contention
  • Memory allocations and potential leaks
• Unlike tools such as PerfView, dotTrace, dotMemory, and Visual Studio profilers, Datadog’s profiler is intended to run continuously in production with negligible overhead.
• Continuous profiling avoids the need to recreate production traffic, security settings, hardware, and load in a separate environment.

Datadog’s .NET Profiler Architecture

• The profiler is composed of specialized profilers for different resource types.
• Each profiler includes:
  • A sampler that collects raw data
  • A provider that exposes the collected samples
• An aggregator combines samples from all profilers.
• An exporter serializes the data into Google’s .pprof format and uploads it through the Datadog Agent.
• Datadog’s backend processes the profiles for visualization and analysis.

Sample Aggregation and Storage

Each sample contains:

• A call stack made up of method frames
• Key-value labels, such as thread identifiers
• A numeric value vector representing measurements like CPU consumption or wall time

Samples with identical call stacks and labels are merged, and their numeric values are added together. This reduces duplication and produces smaller profile files—for example, repeated exceptions from the same code path and thread can be stored as one aggregated sample. The aggregation and .pprof serialization code is implemented in Rust and shared across Datadog’s Ruby, PHP, and other runtime profilers.

Connecting Profiles to Traces and Services

• Each uploaded profile includes process ID, host name, and runtime ID metadata.
• The runtime ID uniquely identifies a .NET service running within a process.
• This is important because a single .NET process can host multiple services, such as separate IIS applications running in different AppDomains.
• The tracer communicates the mapping between runtime IDs, AppDomains, and service names.
• Service names come from DD_SERVICE; if it is unset, the process name is used.
• Datadog sends one profile per runtime ID every minute, so multiple profiles from one process may share a timestamp while representing different services.
• Runtime IDs allow the backend to associate profiles with the correct traces and spans.

Making .NET Call Stacks Easier to Read

The .NET profiling API can expose compiler- and runtime-generated names that differ from the original source code. Datadog rewrites these frames to make visualized call stacks more understandable.

• Constructors named .ctor are displayed using the class name.
• Compiler-generated anonymous methods are rendered as the enclosing method followed by _AnonymousMethod.
• Lambdas and local methods use an enclosing-method name with the _Lambda suffix.
• Nested named methods such as <DefiningMethodName>g__InnerMethodName|yyy_zzz are displayed as DefiningMethodName.InnerMethodName.
• Compiler-generated state-machine methods such as MoveNext are mapped back to the original source-level type and method names.

Native and Managed Implementation Considerations

• The team considered using Microsoft’s TraceEvent NuGet package to receive and parse CLR events in C#.
• That approach would execute managed profiling code on the same CLR as the application being profiled.
• Allocations made by the profiler could therefore increase garbage-collector pressure.
• The post begins discussing how this performance concern influenced the implementation, but the provided excerpt ends before that design is explained.

A production profiler must not only collect useful data but also minimize the memory and CPU costs of collecting it. Datadog’s architecture addresses this through specialized samplers, aggregation, compact serialization, runtime-aware trace association, and source-oriented call-stack cleanup.