Secure publication of Datadog Agent integrations with TUF and in-toto

Datadog built a compromise-resilient CI/CD system to publish Agent integrations independently of full Agent releases. It combines in-toto for end-to-end supply-chain verification with TUF for secure key and metadata distribution. Together, these technologies ensure that users install only integrations derived from developer-approved source code, even if parts of the infrastructure are compromised.

Independent Integration Publishing

• Agent integrations were traditionally bundled into full Agent releases.
• This delayed important integration updates and prevented users from trying new integrations immediately.
• Datadog wanted automation to build and publish integrations on demand without fully trusting the automation itself.

End-to-End Verification with in-toto

• TLS and GPG package signatures help prevent man-in-the-middle attacks but do not protect against compromised build or publishing infrastructure.
• in-toto defines the software supply chain as a fixed sequence of signed steps.
• Each step records the inputs it received and the outputs it produced, allowing the Agent to verify that only authorized parties performed the required work.
• The integration pipeline includes:
  • Developers signing Python and YAML source files.
  • CI/CD packaging the source into Python wheels without modifying existing wheels.
  • A signing step applying TUF signatures to the wheels.
  • The Datadog Agent verifying that the downloaded wheel matches the developer-signed source.

Secure Distribution with TUF

• in-toto does not itself provide a secure way to distribute, revoke, or replace verification keys.
• TUF supplies signed, compromise-resilient metadata for:
  • The root of trust for wheels and supply-chain metadata.
  • The in-toto-defined workflow.
  • Public keys used to verify the workflow.
• TUF protects against tampering, rollback attacks, and indefinite replay of outdated metadata.
• Offline trust bootstrapping and protected developer keys are essential to the overall security model.

Hardware-Protected Developer Signing

• Developers use Yubikeys to generate and store GPG signing keys.
• Private keys cannot be exported from the device, assuming correct firmware.
• Signing requires both a secret PIN and physical interaction with the Yubikey.
• A command-line tool integrates in-toto and GPG, preserving a convenient developer workflow while reducing key-compromise risk.

Transparent Verification for Users

• The Datadog Agent automatically invokes TUF and in-toto when downloading or updating integrations.
• Users need no workflow changes under normal conditions.
• If metadata, signatures, or supply-chain steps fail verification, installation is blocked and the Agent reports the failure.

Datadog’s approach demonstrates that secure automated publishing requires layered controls: in-toto verifies how software was produced, while TUF securely manages the trust and distribution mechanisms needed to validate it.