How we measure data completeness at scale (opens in new tab)

Datadog built a real-time data-completeness system to ensure that every customer’s telemetry is available for dashboards, alerts, queries, and AI-driven decisions. Because ingestion spans hundreds of distributed paths and customers may send delayed or retried data, global or watermark-based tracking is unreliable. The system instead tracks payloads segment by segment, using idempotent create and acknowledgment events to identify losses and calculate end-to-end completeness.

Defining Completeness at Datadog’s Scale

• Completeness means every ingested payload—metrics, logs, spans, or other telemetry—is ultimately available to customers.
• The system must measure completeness:
  • Across hundreds of services and ingestion paths
  • For each individual customer
  • In real time
  • With enough detail to identify where degradation occurred
• Customer traffic may take different routes because of partitioning, isolation, and traffic patterns.
• Metrics and APM pipelines can each involve hundreds or tens of distinct paths, creating a large number of possible failure points.
• The completeness system must remain independent of the services it monitors so it can provide trustworthy diagnostics during incidents.

Tracking Completeness by Pipeline Segment

• Datadog considered watermark-based tracking, but delayed customer data, replayed traffic, and pipeline loops made predictable watermarks impractical.
• Pipelines are divided into segments representing steps within or between services.
  • For example, intake-in to intake-out is one segment.
  • Intake-out to processing-in is another.
• Each segment is measured independently, allowing engineers to locate degradation within a service or between services.
• Segment-level tracking also adapts to pipelines whose branches appear or disappear over time.

Counting Creates and Acknowledgments

• When a payload enters a segment, the system records a create event.
• When it exits, the system records an acknowledgment using the payload’s unique identifier.
• Comparing creates with acknowledgments reveals whether payloads were lost in that segment.
• Events are organized into time buckets based on when the payload first entered Datadog, using a Datadog-controlled timestamp rather than the customer’s clock.
• Each identifier has a state per segment:
  • Created
  • Acknowledged
  • Acknowledged before the create event arrived
• Duplicate create or acknowledgment events are ignored, making the system idempotent despite retries and event reordering.

Calculating End-to-End Completeness

• Segment completeness is the ratio of payloads exiting a segment to those entering it.
• For sequential services, overall completeness is calculated by multiplying segment ratios.
• Parallel branches require a different approach:
  • Treating branches as one pipeline would make completeness wait for the slowest branch.
  • Instead, Datadog uses a weighted average, giving each branch influence proportional to the volume it processes.
• In the example, one branch reaches 94% completeness by multiplying 98% and 96% across two sequential services, while another branch reaches 100%.
• Combining these branch measurements produces a more accurate view of currently available data without incorrectly marking all data incomplete because one branch is slower.

Practical Conclusion

Segment-level, identifier-based tracking gives Datadog a real-time and customer-specific view of data completeness. It both supports reliable end-to-end calculations and helps humans or automated systems quickly determine where ingestion problems are occurring.