How we measure data completeness at scale
Datadog built a real-time data-completeness system to ensure that every customer’s telemetry is available for dashboards, alerts, queries, and AI-driven decisions. Because ingestion spans hundreds of distributed paths and customers may send delayed or retried data, global or watermark-based tracking is unreliable. The system instead tracks payloads segment by segment, using idempotent create and acknowledgment events to identify losses and calculate end-to-end completeness. ## Defining Completeness at Datadog’s Scale - Completeness means every ingested payload—metrics, logs, spans, or other telemetry—is ultimately available to customers. - The system must measure completeness: - Across hundreds of services and ingestion paths - For each individual customer - In real time - With enough detail to identify where degradation occurred - Customer traffic may take different routes because of partitioning, isolation, and traffic patterns. - Metrics and APM pipelines can each involve hundreds or tens of distinct paths, creating a large number of possible failure points. - The completeness system must remain independent of the services it monitors so it can provide trustworthy diagnostics during incidents. ## Tracking Completeness by Pipeline Segment - Datadog considered watermark-based tracking, but delayed customer data, replayed traffic, and pipeline loops made predictable watermarks impractical. - Pipelines are divided into segments representing steps within or between services. - For example, intake-in to intake-out is one segment. - Intake-out to processing-in is another. - Each segment is measured independently, allowing engineers to locate degradation within a service or between services. - Segment-level tracking also adapts to pipelines whose branches appear or disappear over time. ## Counting Creates and Acknowledgments - When a payload enters a segment, the system records a create event. - When it exits, the system records an acknowledgment using the payload’s unique identifier. - Comparing creates with acknowledgments reveals whether payloads were lost in that segment. - Events are organized into time buckets based on when the payload first entered Datadog, using a Datadog-controlled timestamp rather than the customer’s clock. - Each identifier has a state per segment: - Created - Acknowledged - Acknowledged before the create event arrived - Duplicate create or acknowledgment events are ignored, making the system idempotent despite retries and event reordering. ## Calculating End-to-End Completeness - Segment completeness is the ratio of payloads exiting a segment to those entering it. - For sequential services, overall completeness is calculated by multiplying segment ratios. - Parallel branches require a different approach: - Treating branches as one pipeline would make completeness wait for the slowest branch. - Instead, Datadog uses a weighted average, giving each branch influence proportional to the volume it processes. - In the example, one branch reaches 94% completeness by multiplying 98% and 96% across two sequential services, while another branch reaches 100%. - Combining these branch measurements produces a more accurate view of currently available data without incorrectly marking all data incomplete because one branch is slower. ## Practical Conclusion Segment-level, identifier-based tracking gives Datadog a real-time and customer-specific view of data completeness. It both supports reliable end-to-end calculations and helps humans or automated systems quickly determine where ingestion problems are occurring.
Read original(opens in new tab)