How we built reliable log delivery to thousands of unpredictable endpoints | Datadog

Datadog’s “Reliable Log Delivery” post explains how log-collection systems can avoid losing data when networks, destinations, or agents fail. Its central recommendation is to combine acknowledgments, buffering, retries, and controlled backpressure to provide at-least-once delivery without allowing outages to overwhelm the collector.

Why Reliable Delivery Matters

• Logs are often needed during incidents, precisely when infrastructure and networks may be unstable.
• Temporary destination failures can cause data loss if collectors only keep logs in memory.
• Retrying without limits can create duplicate logs, unbounded memory usage, or cascading failures.

Buffering and Persistence

• Collectors should buffer logs while downstream services are unavailable.
• In-memory buffers provide speed but cannot survive process crashes or host restarts.
• Disk-backed queues improve durability by preserving unsent logs across transient failures.
• Storage limits are necessary so a prolonged outage does not fill the host’s disk.

Acknowledgments and Retries

• A log should be removed from the queue only after the destination confirms successful receipt.
• Failed or unacknowledged deliveries are retried, allowing temporary network and service failures to recover automatically.
• At-least-once delivery is the practical reliability target, meaning duplicates may occur and downstream systems should handle them safely.
• Retry policies should use delays and backoff rather than continuously retrying at full speed.

Backpressure and Operational Trade-offs

• When downstream systems slow down, collectors must apply backpressure instead of accepting unlimited data.
• Backpressure can limit memory consumption and protect the rest of the host.
• Teams must define what happens when buffers reach capacity, such as dropping the oldest data, rejecting new logs, or prioritizing important streams.
• Reliability also requires monitoring queue size, delivery latency, retry rates, and dropped records.

A dependable logging pipeline is not built from retries alone. It requires durable buffering, explicit delivery acknowledgments, bounded resources, and clear failure behavior; organizations should choose retention and overflow policies according to the operational value of their logs.