Hardening eBPF for runtime security: Lessons from Datadog Workload Protection | Datadog (opens in new tab)

Scaling real-time file monitoring across high-traffic environments requires a strategy to process billions of kernel events without exhausting system resources. By leveraging eBPF, organizations can move filtering logic directly into the Linux kernel, drastically reducing the overhead associated with traditional userspace monitoring tools. This approach enables precise observability of file system activity while maintaining the performance necessary for large-scale production workloads.

Limitations of Traditional Monitoring Tools

• Conventional tools like auditd often struggle with performance bottlenecks because they require every event to be copied from the kernel to userspace for evaluation.
• Standard APIs like fanotify and inotify lack the granularity needed for complex filtering, often resulting in "event storms" during high I/O operations.
• The high frequency of context switching between kernel and userspace when processing billions of events per minute can lead to significant CPU spikes and system instability.

Architecture of eBPF-Based File Monitoring

• The system hooks into the Virtual File System (VFS) layer using kprobes and tracepoints to capture actions such as vfs_read, vfs_write, and vfs_open.
• LSM (Linux Security Module) hooks are utilized for security-focused monitoring, providing a stable interface that is less prone to kernel version changes than raw kprobes.
• By executing C-like code within the kernel’s sandboxed environment, the system can inspect file paths and process IDs (PIDs) instantly upon event creation.

In-Kernel Filtering and Data Management

• High-performance eBPF maps, specifically BPF_MAP_TYPE_HASH and BPF_MAP_TYPE_LPM_TRIE, are used to store allowlists and denylists for specific directories and file extensions.
• The system implements prefix matching to ignore high-volume, low-value paths like /proc, /sys, or temporary build directories, discarding these events before they ever leave the kernel.
• To minimize memory contention, per-CPU maps are employed, allowing the eBPF programs to aggregate data locally on each core without the need for expensive global locks.

Efficient Data Transmission with Ring Buffers

• The implementation utilizes BPF_RINGBUF rather than the older BPF_PERF_EVENT_ARRAY to handle data transfer to userspace.
• Ring buffers provide a shared memory space between the kernel and userspace, offering better memory efficiency and guaranteeing event ordering.
• By only pushing "filtered" events—representing a tiny fraction of the billions of raw kernel events—the system prevents userspace consumers from becoming overwhelmed.

For organizations operating at massive scale, moving from reactive userspace logging to proactive kernel-level filtering is essential. Implementing an eBPF-based monitoring stack allows for deep visibility into file system changes with minimal performance impact, making it the recommended standard for modern, high-throughput cloud environments.