netflixJul 14, 2026

Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned (opens in new tab)

The post explains how Netflix built a real-time service topology system capable of processing millions of network-flow records per second at production scale. Its core design combines streaming ingestion, reactive backpressure, physically separate data layers, and a distributed aggregation pipeline that resolves network intermediaries into meaningful service dependencies. The system favors slightly delayed but complete updates over stale batch data or incomplete results caused by dropping records. ## The Need for Real-Time Topology - Traditional topology tools rely on hourly or daily batch processing, making their data outdated during incidents. - Netflix combines: - eBPF network flows - IPC metrics delivered through Server-Sent Events - Distributed tracing data - These sources are stored in separate graph or columnar storage layers and can be queried independently or merged. - The goal is near-real-time freshness, faster incident response, blast-radius analysis, and immediate change validation. ## Backpressure for Reliable Streaming - Processing millions of flow records per second creates a risk that downstream systems will become overwhelmed. - Common alternatives are inadequate: - Unbounded queues eventually exhaust memory. - Dropping records produces incomplete topology. - Batch processing introduces unacceptable delays. - Reactive streams propagate slowdown upstream: - A graph database signals Stage 2. - Stage 2 slows Stage 1. - Stage 1 pauses Kafka consumption. - Kafka retains the data until capacity returns. - This allows the system to degrade gracefully during traffic spikes, garbage-collection pauses, or temporary storage slowdowns. - Updates may be delayed by seconds or minutes, but the data remains substantially more complete than a dropped or hourly-processed stream. ## Physically Separate Topology Layers Netflix keeps each data source in storage optimized for its characteristics: - **Network layer:** eBPF flow logs provide broad coverage but limited application context. - **IPC layer:** Application metrics offer detailed endpoint information but cover only instrumented services. - **Tracing layer:** Parquet-based distributed traces show actual request paths but are sampled. - Separate storage enables each layer to evolve and scale independently. - Queries can run in parallel and merge results while preserving sub-second response times. ## Three-Stage Distributed Aggregation The network layer uses a distributed pipeline to transform individual network hops into logical service dependencies. - Cloud traffic commonly passes through load balancers, NAT gateways, API gateways, and proxies. - Flow logs therefore show relationships such as: - `App A → Load Balancer` - `Load Balancer → App B` - The useful topology must infer the logical dependency: `App A → App B`. ### Stage 1: Initial Flow Aggregation - Consumes flow logs from Kafka across four regions. - Filters invalid records. - Groups data into five-minute windows. - Creates initial aggregators for each window. - Uses consistent hashing to distribute aggregators. - Streams the results to Stage 2 through SSE. ### Stage 2: Intermediary Resolution - Receives the initial aggregators from Stage 1. - Groups flows by intermediary components. - Resolves multi-hop network paths into application-level relationships. - This prevents infrastructure components from dominating the resulting service graph. ## Engineering Trade-offs - Streaming provides much fresher data than batch processing but introduces greater operational and conceptual complexity. - Backpressure is essential for stability at Netflix’s scale, even though reactive pipelines are harder to reason about than synchronous systems. - The architecture prioritizes reliable, complete topology updates over perfectly immediate processing. - Production behavior differed substantially from local testing: consumers lagged, memory was exhausted, traffic became unevenly distributed, and garbage collection consumed significant resources. Netflix’s approach demonstrates that large-scale real-time topology requires streaming ingestion, end-to-end backpressure, specialized storage, and staged aggregation. For similar distributed systems, the practical recommendation is to design explicitly for overload and partial slowdown rather than relying on unbounded buffering, dropped data, or stale batch snapshots.

ebpfkafkagraph-databasedistributed-tracing+4

metaJul 13, 2026

Modernizing the Meta Ads Service With an Open-Source Kernel Scheduler (opens in new tab)

At Meta’s scale, small latency regressions can materially affect ad relevance, ranking, and revenue. When Linux kernel 6.9’s EEVDF scheduler reduced ad-serving performance, Meta used sched_ext to deploy a workload-specific BPF scheduler without modifying the kernel. The solution reduced p99 ads-retrieval latency by 28%, saved 3.28 MW of power, increased ads ranked by 1.1%, and enabled further improvements through rapid user-space updates. ## Why Ads Latency Matters - Meta’s ads platform processes more than 5 million requests per second, or over 400 billion requests daily. - Lower p99 latency allows more relevant ads to be retrieved and ranked within each request. - General-purpose schedulers such as CFS and EEVDF balance CPU usage without understanding which threads are most important to ad delivery. - Ads-specific scheduling can prioritize work on the critical request path while deferring less-sensitive tasks. ## The Kernel Upgrade Problem - During a move from Linux 6.4 to 6.9, Meta found that EEVDF introduced a latency regression. - The regression reduced the number of ads ranked in responses. - Some servers had to remain on Linux 6.4, creating operational fragmentation and technical debt. - sched_ext provided a way to address the regression without waiting for another kernel release. ## How sched_ext Customizes Scheduling - sched_ext is an upstream, BPF-based framework that entered Linux 6.12. - It lets developers implement scheduling policies in BPF programs responding to events such as: - Thread wake-ups - Run-queue insertion - Dispatching the next thread - CPU idle-state transitions - Meta’s policy divides CPUs into two dynamically sized pools: - Latency-critical request-path threads - Less latency-sensitive background work - Keeping related work on the same CPUs improves L3 cache locality and reduces DRAM access. - The scheduler is loaded by a user-space binary, so new policies can be deployed by restarting the scheduler process rather than rebuilding the kernel. ## Performance and Operational Results The initial deployment on the largest ads-serving server type achieved: - 28% lower p99 latency on the ads retrieval path - 1.1% more weighted ads ranked - 3.28 MW of fleet-wide power savings Two subsequent policy updates produced additional gains: - A further 60% reduction in service p99 latency - 18% fewer timeout errors on the critical path - Delivery in days instead of the months typically required for kernel changes ## From Fix to Optimization Platform - sched_ext gives Meta an independent scheduling-development path alongside upstream Linux evolution. - BPF updates support rapid experimentation with: - Cache-aware thread placement - ROI-based executor routing - NUMA-aware scheduling - Because sched_ext is upstream, other organizations can implement workload-specific policies without maintaining a Linux kernel fork. - Meta plans to use application-level hints, such as request importance, to adjust scheduling slices and queue priority dynamically. sched_ext demonstrates that application-aware scheduling can produce measurable business, latency, and energy benefits. For workloads with priorities that general-purpose schedulers cannot see, an extensible BPF-based scheduler offers a practical way to optimize continuously without coupling improvements to kernel release cycles.

ebpflinuxlinux-kernelnuma+4

netflixMay 29, 2026

From Silos to Service Topology: Why Netflix Built a Real-Time Service Map (opens in new tab)

Netflix built Service Topology to give engineers a real-time, unified view of dependencies across its thousands of microservices. Traditional metrics, logs, and traces provide isolated signals but do not reveal the broader service relationships needed to diagnose failures or assess blast radius. The system combines multiple dependency sources into a living map that supports fast, context-rich troubleshooting. ## The Observability Problem - Netflix’s distributed architecture involves thousands of services and complex chains of calls for actions such as playback, authentication, recommendations, and optimization. - During incidents, engineers need to determine: - Which services depend on one another - What the potential blast radius is - Whether a failure originates locally or upstream - Existing observability tools show symptoms, logs, or individual request paths, but not the complete steady-state topology. - Manually combining information from different tools is slow and error-prone, especially during urgent incidents. ## Why Real-Time Service Mapping Matters - Frequent deployments and changing traffic patterns make static architecture diagrams quickly obsolete. - Netflix’s Live programming and advertising-supported plans increase the need for rapid diagnosis and operational awareness. - Engineers repeatedly asked about dependencies, failures, maintenance impact, unknown metrics, and recent call-path changes. - These recurring questions demonstrated the need for accurate, near-real-time dependency information. ## Lessons from Earlier Approaches - Netflix evaluated vendor platforms, graph databases, and internal prototypes before developing Service Topology. - Key lessons included: - Dependency data must update in near real time. - Storage and query systems must operate at Netflix’s scale. - The solution should integrate with existing observability workflows. - Incorrect or incomplete topology data can mislead engineers during incidents. - No single data source captures every aspect of service relationships. ## Requirements for a Living Map Service Topology was designed to provide: - Real-time updates as services deploy and dependencies change - Sub-second queries for traversing service call graphs - Both network-level and application-level views - Context such as health, availability tiers, ownership, and business domains - A visual interface for engineers and programmatic APIs for automation, resilience systems, and blast-radius analysis ## Combining Multiple Sources of Truth Netflix separates dependency information into physically distinct graphs so each layer can evolve and be queried independently. When a unified view is requested, the system traverses the layers in parallel and merges the results to maintain fast response times. ### eBPF Network Flows - eBPF captures network activity at the kernel level, recording which services communicate over the network. - This provides broad coverage, including services that lack application instrumentation. - It supports both cluster-level and application-level topology. - Its limitation is that network traffic alone does not provide application-specific context, such as the APIs or endpoints involved. Netflix’s approach is to combine complementary perspectives rather than rely on a single imperfect dependency source, producing a more complete and actionable service map.

ebpfobservabilitydistributed-systemsmicroservices+2

githubApr 16, 2026

How GitHub uses eBPF to improve deployment safety (opens in new tab)

GitHub uses eBPF to prevent deployment scripts from depending on GitHub or other services that may be unavailable during an outage. The approach applies network restrictions only to deployment processes, preserving normal traffic for stateful production hosts. By combining Linux cgroups with eBPF’s `BPF_PROG_TYPE_CGROUP_SKB` hooks, GitHub can detect or block unsafe outbound calls before they create circular deployment dependencies. ## The Deployment Circular Dependency - GitHub hosts its own source code on `github.com`, creating a basic dependency: GitHub may need GitHub to deploy a fix. - GitHub mitigates this with: - A code mirror used for “fix-forward” deployments. - Prebuilt assets used to roll back changes. - Additional circular dependencies can still be introduced by deployment scripts, internal services, or tools that download binaries dynamically. ## Three Types of Circular Dependencies The post uses a hypothetical MySQL outage to illustrate how deployment recovery can fail: - **Direct dependencies** - A deployment script downloads the latest release of an open-source tool from GitHub. - If GitHub cannot serve release data, the deployment cannot complete. - **Hidden dependencies** - A required tool is already installed locally but checks GitHub for updates when it runs. - The tool may fail or hang if that update check cannot connect. - **Transient dependencies** - The deployment calls another internal service, such as a migrations service. - That service then attempts to download a binary from GitHub, causing the failure to propagate back to the deployment. ## Why Manual Review Is Insufficient - Teams responsible for stateful hosts traditionally review deployment scripts for circular dependencies. - Many indirect or unexpected dependencies are discovered only during incidents, when they can delay recovery. - Blocking `github.com` at the host level would be too disruptive because stateful hosts continue serving customer traffic during deploys, drains, and restarts. ## Per-Process Network Filtering with eBPF - eBPF allows custom programs to run inside the Linux kernel and attach to low-level operations such as networking. - GitHub focused on `BPF_PROG_TYPE_CGROUP_SKB`, which can inspect network egress for a specific cgroup. - Linux cgroups provide process grouping, isolation, and resource controls without requiring Docker. - The proposed design: - Create a dedicated cgroup. - Place only the deployment script and its processes inside it. - Restrict or monitor outbound network access for that group. - Leave the host’s ordinary production traffic unaffected. ## Proof of Concept with Go and eBPF - The proof of concept uses Go and the `cilium/ebpf` library. - The library simplifies: - Compiling and loading eBPF programs. - Attaching programs to kernel hooks. - Reading and updating eBPF maps. - The example attaches an egress program to `/sys/fs/cgroup/system.slice`. - An eBPF array map tracks the number of egress packets, while the Go program periodically reads and reports the counter. - The same mechanism can be extended from packet counting to selectively allowing or blocking network traffic. GitHub’s approach moves dependency validation from manual inspection into enforcement at runtime. Restricting only deployment processes with cgroups and eBPF provides a practical way to make emergency deployments independent of services that may be down, without blocking the production workloads sharing the same host.

ebpfgolinux-kernelcgroups+2

cloudflareApr 10, 2026

500 Tbps of capacity: 16 years of scaling our global network (opens in new tab)

Cloudflare’s network has grown from a single transit provider in 2010 to more than 500 Tbps of provisioned external capacity across 330+ cities. The company argues that this scale is not merely about bandwidth: it enables security decisions, application execution, and routing validation to happen locally on every server. Its distributed architecture can absorb massive attacks automatically while supporting edge computing and emerging Internet protocols. ## From Transit Provider to Global Network - Cloudflare began with nLayer Communications as its first transit provider. - Expansion required city-by-city work: colocation contracts, fiber installation, hardware deployment, and Internet exchange peering. - In 2018, Cloudflare opened 31 cities in 24 days, despite logistical challenges such as customs delays and missing equipment. - The network now spans more than 330 cities and protects over 20% of the web. - The 500 Tbps figure represents provisioned interconnection capacity across transit, private peering, Internet exchanges, and Cloudflare Network Interconnect ports—not peak traffic. ## Turning the Network into a Security Layer - Cloudflare expanded from caching websites to securing employees and enterprise networks. - Its systems establish secure tunnels to private subnets and advertise customer IP space through BGP. - In 2025, Cloudflare mitigated a 31.4 Tbps DDoS attack lasting 35 seconds. - The attack was part of more than 5,000 attacks blocked that day, without paging an engineer. - Distributed automation allows attacks that once required nation-state resources to be handled in seconds. ## Packet-Level DDoS Mitigation - Incoming packets enter an XDP program chain in driver mode immediately after reaching the network interface card. - The `l4drop` eBPF program applies mitigation rules generated by `dosd`, Cloudflare’s denial-of-service daemon. - Each server identifies heavy traffic sources and shares the information across its colocation facility. - Mitigation rules spread globally through Quicksilver, Cloudflare’s distributed key-value store. - Only legitimate traffic reaches Unimog, the Layer 4 load balancer; Magic Transit traffic receives additional stateful inspection through `flowtrackd`. - The 31.4 Tbps attack was stopped at line rate without centralized scrubbing or human intervention. - Sufficient physical port capacity remains essential: software defenses cannot work if the network cannot first absorb the traffic. ## A Developer Platform at the Edge - Because Cloudflare already runs software on every server for packet filtering, it extended the same infrastructure to customer code through Workers. - Workers, KV, and Durable Objects run across Cloudflare’s global footprint rather than in a small number of cloud regions. - Workers Containers, introduced in 2025, support heavier workloads at the edge. - V8 isolates and custom filesystem layers reduce cold-start times. - Applications run on the same servers that discard malicious traffic before it reaches the network stack. ## Securing Routing with RPKI and ASPA - Cloudflare uses IPv6 and RPKI to reduce the risk of BGP hijacks. - It signs Route Origin Authorizations and rejects routes that fail Route Origin Validation, even when misconfigured networks become temporarily unreachable. - ASPA will extend protection by validating the network path, not just the organization authorized to originate a prefix. - The post compares RPKI to checking a destination passport and ASPA to verifying the entire flight manifest. - Cloudflare says 867,000 prefixes now have valid RPKI certificates, compared with nearly none a decade ago. - The company promotes early adoption of routing security standards because delays leave the Internet exposed to hijacks and route leaks. ## AI Agents and Internet Traffic - AI crawlers, training systems, and autonomous agents now generate more than 4% of HTML requests on Cloudflare’s network. - Human-initiated “user action” crawling increased more than 15-fold in 2025. - Unlike browsers, crawlers may retrieve every linked resource at maximum speed, making legitimate activity difficult to distinguish from attacks. - Cloudflare uses verified bot IP ranges, TLS fingerprints, behavioral analysis, and robots.txt signals to classify AI crawlers. - These signals help site owners decide which automated agents to permit. Cloudflare’s central lesson is that a global network must combine abundant capacity with intelligence distributed across every server. Its continued investment in automated mitigation, edge execution, routing security, and traffic classification is intended to make the Internet faster, safer, and more resilient as traffic patterns evolve.

ebpfcloudflaredistributed-systemsnetwork-security+4

cloudflareApr 8, 2026

From bytecode to bytes- automated magic packet generation (opens in new tab)

Classic BPF filters can hide Linux malware until a precisely crafted “magic” packet arrives, but manually reverse-engineering large filters is slow and error-prone. The post presents a symbolic-execution approach using the Z3 theorem prover to model BPF instructions as packet constraints and automatically generate triggering packets. This reduces analysis from hours of manual work to seconds, even for filters exceeding 100 instructions. ## Why BPF Filters Are Difficult to Analyze - Classic BPF is a small, efficient virtual machine used to filter network traffic inside the Linux kernel. - Unlike eBPF, classic BPF has a simple two-register design but can still contain many conditional jumps and packet-offset calculations. - Malware authors exploit BPF because kernel-level filtering can hide traffic from ordinary user-space monitoring tools. - Short programs are manageable manually, but complexity grows rapidly as filters reach 100 or more instructions. - The core problem is determining which packet bytes satisfy the conditions along an accepting execution path. ## BPFDoor as a Practical Example - BPFDoor is a stealthy Linux backdoor associated with cyberespionage campaigns and groups including Red Menshen/Earth Bluecrow. - It uses BPF to inspect incoming traffic without listening on a dedicated open port. - The example filter checks: - IPv6 or IPv4 EtherType. - UDP protocol. - DNS destination port 53. - Fragmentation status for IPv4 packets. - The IPv4 header length when locating the UDP destination port. - The filter contains two paths leading to acceptance: - An IPv6 UDP packet destined for port 53. - A non-fragmented IPv4 UDP packet destined for port 53. - These paths expose the byte offsets and values that a generated packet must satisfy. ## Finding the Shortest Accepting Path - The tool explores the BPF control-flow graph using a queue. - Each queued item records: - The next instruction pointer. - The sequence of instructions already traversed. - Conditional jumps are explored in both directions. - Paths ending in a drop result are discarded, while paths reaching a nonzero return value are recorded as accepting paths. - Breadth-first traversal prioritizes paths with fewer conditions, helping identify the shortest route to acceptance. - Unconditional jumps are followed directly, while conditional branches enqueue true and false destinations in order of path length. ## Turning Paths into Packets - Once accepting paths are identified, each branch condition becomes a constraint on packet contents. - The required byte offsets, widths, and values can be collected from the executed instructions. - Symbolic execution represents these checks as constraints instead of requiring analysts to reason through every instruction manually. - Z3 can then solve the resulting constraint set and produce packet bytes that satisfy the selected accepting path. - This approach is especially useful for large or heavily branched BPF programs where manual packet construction becomes impractical. The recommended workflow is to combine control-flow exploration with symbolic constraint solving: first identify viable accepting paths, then use Z3 to generate packets satisfying their byte-level requirements. This automates a formerly labor-intensive part of malware analysis and makes complex BPF-based backdoors much faster to investigate.

ebpfnetwork-securityberkeley-packet-filterbpf+4

cloudflareMar 31, 2026

Introducing Programmable Flow Protection: custom DDoS mitigation logic for Magic Transit customers (opens in new tab)

Programmable Flow Protection lets Magic Transit Enterprise customers define custom DDoS mitigation logic for proprietary UDP protocols. Customers write eBPF programs that identify valid and malicious packets, then deploy them across Cloudflare’s global network to pass, drop, or challenge traffic. Currently in beta for an additional cost, the system addresses the limitations of generic blocking and rate limiting. ## Custom Protection for Proprietary UDP - Existing Cloudflare protections understand established protocols such as TCP, DNS, NTP, RDP, and SIP. - Proprietary UDP protocols are harder to protect because Cloudflare cannot interpret their application-level payloads. - Customers can now define what constitutes “good” and “bad” traffic using their own protocol knowledge. - Programs can drop or challenge invalid packets before they reach the customer’s origin. ## Why Generic UDP Mitigation Falls Short - UDP is connectionless and optimized for speed, making it useful for gaming, VoIP, and streaming. - When traffic does not match a known protocol, mitigation typically falls back to: - Blocking a destination IP and port - Applying a generic rate limit - These approaches cannot distinguish legitimate packets from attack traffic, potentially causing lag or connection loss for real users. - Fixed rate limits may also be inappropriate: - A network expecting 1 Gbps may need stricter limits. - A network expecting 25 Gbps may require more permissive thresholds. ## How Programmable Flow Protection Works - Customers upload custom eBPF programs that run on every packet destined for their network. - Programs execute in userspace rather than kernel space, providing isolation and flexibility across different customers and use cases. - Execution occurs after Cloudflare’s existing DDoS protections, preserving baseline security coverage. - Like kernel-based XDP eBPF programs, these programs: - Compile to BPF bytecode - Pass safety and termination verification - Run inside a lightweight, isolated virtual machine - Cloudflare provides specialized helpers for: - Maintaining client state between packet executions - Performing cryptographic validation - Sending challenge packets to clients ## Example: Protecting a Proprietary Game Protocol - A gaming provider running on UDP port 207 could inspect its proprietary application header. - If the header contains a protocol-specific token, the customer’s eBPF program can: - Parse the packet - Extract part of the token, such as its final byte - Pass packets with the expected value - Drop packets that fail validation - This allows legitimate players’ traffic through even when attacks use randomized source addresses, ports, and payloads. Programmable Flow Protection is best suited to Magic Transit customers whose custom UDP protocols cannot be protected effectively by standard protocol-aware controls. By combining customer-specific packet logic with Cloudflare’s global network and stateful challenge mechanisms, it enables more precise mitigation than blanket blocking or generic rate limiting.

ebpfcloudflarenetwork-securitytcp+4

datadogJan 7, 2026

Hardening eBPF for runtime security: Lessons from Datadog Workload Protection (opens in new tab)

eBPF gives security tools broad, efficient, and relatively safe access to Linux kernel activity, making it well suited for runtime threat detection. Datadog chose it for Workload Protection after comparing kernel modules, tracing interfaces, ptrace, seccomp, Linux Audit, and other approaches. However, five years of production use across diverse kernels showed that eBPF’s safety and performance benefits are not automatic; reliability, compatibility, observability, and operational discipline are essential at scale. ## Why Runtime Workload Protection Is Needed - Static analysis and vulnerability scanning cannot catch every threat. - Zero-days and vulnerable third-party dependencies can remain active while patches are being prepared or deployed. - Workload Protection is intended to: - Monitor known-vulnerable workloads until they can be patched. - Continuously observe all workloads. - Detect and help mitigate previously unknown vulnerabilities during incident response. ## Alternatives Evaluated Datadog evaluated a broad range of Linux monitoring and instrumentation mechanisms: - **Linux kernel modules** - Offer deep access and can hook or replace almost any kernel function. - Are invasive and often considered too risky for production infrastructure. - **Traditional tracing interfaces** - Include inotify, fanotify, kprobes, tracepoints, and perf events. - Provide useful visibility but generally need to be combined for comprehensive coverage. - **ptrace and seccomp-bpf** - Can provide detailed user-space process visibility. - Are less suitable as a unified solution for monitoring the whole system. - **Linux Audit** - Produces configurable streams for process execution, file access, and network activity. - Is widely used by security tooling but has its own performance and operational tradeoffs. - **Other mechanisms** - Netlink, LD_PRELOAD, and binfmt_misc were also considered. - Each involves compromises in reliability, visibility, or system impact. ## Why eBPF Stood Out - **Safety checks** - The kernel statically verifies eBPF bytecode before loading it. - Verification detects issues such as infinite loops and unsafe memory access. - This is safer than deploying custom kernel modules, though eBPF can still cause harm or performance problems. - **Performance** - eBPF generally has lower overhead than approaches such as Linux Audit or ptrace. - Actual impact depends heavily on implementation and workload. - **Unified visibility** - A single mechanism can observe process, filesystem, and network activity. - This avoids assembling multiple specialized tracing systems. - **Container and namespace coverage** - eBPF provides consistent visibility across namespaces, cgroups, and containers. - CO-RE (Compile Once–Run Everywhere) improves portability across Linux distributions and kernel versions. - **Enforcement capabilities** - BPF LSM programs support mandatory access controls. - This gives eBPF enforcement power beyond ordinary tracing mechanisms, which is important for runtime security. ## Lessons from Operating eBPF at Scale After five years of operating an agent that hooks process scheduling, filesystem, and networking internals, Datadog emphasizes that production eBPF is more complicated than its reputation suggests. The six areas of operational experience are: - Ensuring programs load, attach, and continue firing across kernel versions. - Capturing and enriching event data accurately. - Monitoring and auditing eBPF usage to reduce the attack surface. - Coexisting with other eBPF-based tools on the same host. - Measuring and controlling performance overhead. - Shipping changes safely through disciplined rollout practices. The practical recommendation is to treat eBPF as powerful infrastructure rather than a maintenance-free kernel feature: validate behavior across kernels and workloads, monitor its own operation, measure overhead continuously, and use cautious deployment practices.

ebpfdatabase-designobservabilitylinux-kernel+4

datadogNov 18, 2025

Scaling real-time file monitoring with eBPF: How we filtered billions of kernel events per minute | Datadog (opens in new tab)

Datadog was named a Leader in the 2026 Gartner® Magic Quadrant™ for Observability Platforms. The announcement positions Datadog as a provider of broad, integrated monitoring across infrastructure, applications, logs, security, digital experiences, software delivery, and AI. The supplied content does not include Gartner’s evaluation details or the blog post’s supporting arguments. ## Recognition and Positioning - Datadog highlights its designation as a Leader in the Gartner Magic Quadrant for Observability Platforms. - The announcement emphasizes Datadog’s unified observability platform rather than a single monitoring product. ## Breadth of the Platform - **Infrastructure:** infrastructure, container, network, serverless, GPU, storage, and cloud-cost monitoring. - **Applications and data:** APM, database monitoring, continuous profiling, data-stream monitoring, and job monitoring. - **Logs and observability operations:** log management, sensitive-data scanning, audit trails, and observability pipelines. - **Security:** cloud security, SIEM, workload protection, code security, vulnerability management, and application/API protection. - **Digital experience:** browser and mobile RUM, session replay, synthetic monitoring, product analytics, and error tracking. - **Software delivery and service management:** CI visibility, testing, feature flags, incident response, SLOs, workflow automation, and case management. - **AI capabilities:** agent observability, GPU monitoring, AI integrations, Bits AI agents, and investigation tools. Overall, the available material presents Datadog’s Gartner Leader recognition and extensive product coverage, but it does not provide enough article text to summarize the specific reasoning behind the designation.

ebpfk8sdatadogobservability+2

datadogNov 18, 2025

Scaling real-time file monitoring with eBPF: How we filtered billions of kernel events per minute (opens in new tab)

File integrity monitoring must provide more than proof that a file changed: security teams need to know how, why, and by whom it changed. Datadog found that filesystem scans, inotify, and auditd could not provide sufficient context, reliability, or scalability. An eBPF-based approach delivered kernel-level visibility into processes and containers, but required extensive filtering and edge processing to handle more than 10 billion events per minute. ## Why Traditional Monitoring Falls Short - Periodic scans can miss changes that are made and reverted between scans. - Scans show that a file changed, but not the process, container, or mechanism responsible. - `inotify` lacks the system-level context needed to correlate file events with processes and containers. - `auditd` offers richer information but can impose significant performance overhead and struggle under heavy load. ## eBPF for Context-Rich File Monitoring - eBPF observes file activity directly in the Linux kernel in real time. - Events can include: - The modified file - The process that triggered the change - The container in which the process ran - Additional security-relevant metadata - This context makes events more useful for investigations than simple “file changed” notifications. ## Scaling at the Agent and Backend - Datadog observed more than 10 billion file-related events per minute across its infrastructure. - Each serialized event was approximately 5 KB, making unrestricted transmission infeasible—potentially several terabytes per second. - Sending every event would also overload Agents through excessive CPU, memory, serialization, and network usage. - Agent-side rules filter events locally, discarding noise before transmission. - This reduced the stream to roughly one million events per minute while preserving detection coverage. ## Filtering Events in the Kernel - A basic architecture loads eBPF programs into the Agent, observes system activity, writes events to a ring buffer, and evaluates them in user space. - Sensitive workloads can generate up to 5,000 relevant syscalls per second. - Initial implementations risked ring-buffer backlogs and dropped events, creating security blind spots. - Datadog moved as much evaluation as possible into eBPF programs to reduce the number of events reaching user space. - The Agent could then perform a deeper second-stage evaluation before forwarding events to the backend. ## Two-Stage Evaluation: Approvers and Discarders - eBPF’s safety constraints limit computation, especially on older Linux kernels. - The system therefore separates evaluation into: - **In-kernel filtering:** Lightweight decisions that quickly approve or discard events. - **User-space evaluation:** More complex analysis using richer context, correlations, and logic unsuitable for the kernel. - This design balances kernel safety and performance with the need for detailed security detection. Datadog’s approach shows that scalable FIM requires combining eBPF’s deep visibility with aggressive filtering at the edge and in the kernel. The practical recommendation is to keep expensive analysis in user space while rejecting irrelevant events as early as possible.

ebpfdatabase-designlinuxsecurity-monitoring+4