Hardening eBPF for runtime security: Lessons from Datadog Workload Protection

eBPF gives security tools broad, efficient, and relatively safe access to Linux kernel activity, making it well suited for runtime threat detection. Datadog chose it for Workload Protection after comparing kernel modules, tracing interfaces, ptrace, seccomp, Linux Audit, and other approaches. However, five years of production use across diverse kernels showed that eBPF’s safety and performance benefits are not automatic; reliability, compatibility, observability, and operational discipline are essential at scale.

Why Runtime Workload Protection Is Needed

• Static analysis and vulnerability scanning cannot catch every threat.
• Zero-days and vulnerable third-party dependencies can remain active while patches are being prepared or deployed.
• Workload Protection is intended to:
  • Monitor known-vulnerable workloads until they can be patched.
  • Continuously observe all workloads.
  • Detect and help mitigate previously unknown vulnerabilities during incident response.

Alternatives Evaluated

Datadog evaluated a broad range of Linux monitoring and instrumentation mechanisms:

• Linux kernel modules
  • Offer deep access and can hook or replace almost any kernel function.
  • Are invasive and often considered too risky for production infrastructure.
• Traditional tracing interfaces
  • Include inotify, fanotify, kprobes, tracepoints, and perf events.
  • Provide useful visibility but generally need to be combined for comprehensive coverage.
• ptrace and seccomp-bpf
  • Can provide detailed user-space process visibility.
  • Are less suitable as a unified solution for monitoring the whole system.
• Linux Audit
  • Produces configurable streams for process execution, file access, and network activity.
  • Is widely used by security tooling but has its own performance and operational tradeoffs.
• Other mechanisms
  • Netlink, LD_PRELOAD, and binfmt_misc were also considered.
  • Each involves compromises in reliability, visibility, or system impact.

Why eBPF Stood Out

• Safety checks
  • The kernel statically verifies eBPF bytecode before loading it.
  • Verification detects issues such as infinite loops and unsafe memory access.
  • This is safer than deploying custom kernel modules, though eBPF can still cause harm or performance problems.
• Performance
  • eBPF generally has lower overhead than approaches such as Linux Audit or ptrace.
  • Actual impact depends heavily on implementation and workload.
• Unified visibility
  • A single mechanism can observe process, filesystem, and network activity.
  • This avoids assembling multiple specialized tracing systems.
• Container and namespace coverage
  • eBPF provides consistent visibility across namespaces, cgroups, and containers.
  • CO-RE (Compile Once–Run Everywhere) improves portability across Linux distributions and kernel versions.
• Enforcement capabilities
  • BPF LSM programs support mandatory access controls.
  • This gives eBPF enforcement power beyond ordinary tracing mechanisms, which is important for runtime security.

Lessons from Operating eBPF at Scale

After five years of operating an agent that hooks process scheduling, filesystem, and networking internals, Datadog emphasizes that production eBPF is more complicated than its reputation suggests.

The six areas of operational experience are:

• Ensuring programs load, attach, and continue firing across kernel versions.
• Capturing and enriching event data accurately.
• Monitoring and auditing eBPF usage to reduce the attack surface.
• Coexisting with other eBPF-based tools on the same host.
• Measuring and controlling performance overhead.
• Shipping changes safely through disciplined rollout practices.

The practical recommendation is to treat eBPF as powerful infrastructure rather than a maintenance-free kernel feature: validate behavior across kernels and workloads, monitor its own operation, measure overhead continuously, and use cautious deployment practices.