Squeezing every millisecond: How we rebuilt the Datadog Lambda Extension in Rust

Datadog rewrote its AWS Lambda extension from Go into Rust to overcome the performance limits of adapting its large, host-oriented Datadog Agent to Lambda’s constrained environment. The redesign reduced cold-start latency by 82%, memory usage by 40%, and binary size from 55 MB to 7 MB. The project succeeded by narrowing the problem, enforcing performance budgets from the beginning, and designing specifically for Lambda’s execution model.

Why the Original Extension Needed to Change

• The Lambda extension runs as a sidecar process, collecting logs, metrics, traces, profiles, and process data asynchronously.
• It was originally based on the Datadog Agent, which is designed for hosts, containers, and clusters.
• The Agent’s fairness, buffering, caching, and high-throughput features introduced unnecessary overhead in Lambda.
• Optimization attempts included:
  • Removing dependencies with build tags
  • Compressing binaries with UPX
  • Eliminating unnecessary init methods
  • Exploring Go plugins for lazy loading
• These changes could not reduce additional cold-start latency below roughly 450–500 milliseconds.

Why a Rewrite—and Why Rust

• Rewrites are risky because they can lose undocumented invariants, reproduce subtle bugs, and create the burden of supporting two systems.
• The team concluded that Lambda represented a fundamentally different scale and workload from the general-purpose Datadog Agent.
• Rust was well suited because:
  • Memory safety reduces the risk of crashes and data races.
  • Extension crashes also terminate the Lambda function and trigger another cold start.
  • Rust produces small binaries with limited runtime overhead.
  • Lambda targets a narrow platform set: Amazon Linux on x86 and Arm.
  • Compile-time concurrency guarantees support reliable multithreaded code.
• A hackathon prototype demonstrated enough potential to begin the full rewrite, named Project Bottlecap.

Project Bottlecap’s Design Constraints

• The extension had to minimize interference with the function handler, especially because many Lambda functions serve latency-sensitive APIs.
• Telemetry work should occur after the handler returns whenever possible.
• The team also minimized post-runtime duration—the CPU time added after normal function execution.
• Performance was monitored from the start:
  • Dashboards and alerts tracked cold-start overhead.
  • Every pull request was benchmarked.
  • Regressions were investigated before merging.
• The team accepted targeted tradeoffs for speed, including manually implementing AWS API calls and request signing instead of using SDKs that added too much overhead.
• The design emphasized optionality because Lambda workloads range from small API functions to large asynchronous batch jobs.
• Planned flush strategies included:
  • Flushing at the end of an invocation for infrequently called or CPU-constrained functions
  • Periodic or in-invocation flushing for workloads needing different latency and resource tradeoffs

The practical lesson is that software optimized for large, long-running systems may be fundamentally unsuitable for serverless runtimes. When optimization reaches a hard performance floor, a focused rewrite—constrained by the target environment and measured continuously—can deliver major gains.