Detecting faulty deployments: Our journey from unlabeled data to supervised learning

Deployments are a major source of software incidents, making rapid detection of faulty releases essential. Datadog developed Automatic Faulty Deployment Detection to identify releases associated with significant, deployment-related increases in error rates, despite having no reliable labeled dataset. Their solution evolved into an iterative, unsupervised ensemble of statistical checks designed to balance precision, recall, and the diverse behavior of customer applications.

Challenges in Detecting Faulty Deployments

• No universal ground truth exists because teams define “faulty” differently depending on their applications.
• Faulty deployments are rare, creating severe class imbalance:
  • Random manual labeling would produce few useful examples.
  • Even a low false-positive rate could result in poor precision.
• Applications have widely varying traffic and error patterns:
  • Seasonal applications naturally experience periodic changes.
  • Low-traffic services need longer observation periods.
  • Frequent deployments can make it difficult to identify which release caused an incident.

Defining a Faulty Deployment

Datadog focused on deployments that caused a significant and sustained increase in error rate. The definition relied on three attributes:

• Impact
  • The total number of errors must be meaningfully higher than the baseline.
  • The increase must be significantly worse than in previous versions.
• Temporal correlation
  • The error increase should align with the introduction of the new version.
• Persistence
  • The elevated error rate must continue over time rather than reflecting temporary deployment noise.

Building an Iterative Detection Framework

• The initial system applied simple statistical rules to the first 60 minutes after each deployment.
• Manual annotation was used to estimate precision, but this required substantial effort and did not reveal recall.
• Datadog created an iterative framework composed of checks for different deployment requirements.
• Checks included:
  • Comparing error rates before and after deployment.
  • Comparing a release with previous versions.
  • Accounting for periodic traffic and errors.
  • Handling sparse traffic patterns.
• The checks were combined into a unanimous-voting ensemble: a deployment was flagged only when every check classified it as faulty.
• The process began with a high-recall model, then:
  • Manually reviewed predicted faults.
  • Analyzed false positives.
  • Added new checks and adjusted thresholds to improve precision and recall.
• Incident data and version rollbacks provided additional signals for finding faulty deployments the model had missed.

Balancing Detection Speed and Recall

• The model used the first hour after deployment to gather enough data to determine whether increased errors were persistent.
• Increasing the observation period can improve confidence but delays detection.
• The framework became progressively more sophisticated, adapting to:
  • Periodic error and traffic patterns.
  • Sparse traffic.
  • Multiple concurrent application versions.

The practical recommendation is to begin with simple, high-recall statistical rules, then iteratively improve them through targeted manual review, false-positive analysis, and additional operational signals such as incidents and rollbacks. This approach can support other anomaly-detection problems where labels are scarce, failures are rare, and application behavior varies significantly.