How we built reliable log delivery to thousands of unpredictable endpoints
Datadog’s Log Forwarding system resembles a package delivery network: it must move large volumes of data efficiently and reliably to many unpredictable destinations. Kafka provides ordered transport, but its FIFO behavior creates difficult tradeoffs when endpoints are slow or unavailable. The central challenge is preserving delivery guarantees without losing logs, creating duplicates, blocking unrelated destinations, or overwhelming customer infrastructure. ## What Log Forwarding Does - Datadog forwards processed, enriched logs as schemaless JSON records. - Destinations can include: - Elasticsearch - Splunk - Generic HTTP endpoints accepting JSON `POST` requests - The system must support thousands of tenants and external endpoints with widely varying reliability and performance. ## Kafka as the Distribution Network - Logs move through Datadog on Kafka topics, analogous to packages traveling on conveyor belts. - Each Kafka partition provides strict FIFO ordering: - Records are read in the order they were written. - Kafka offsets must be committed in that same order. - Logs for different destinations are spread across multiple partitions, so records for a single destination may need to be regrouped during delivery. - Assigning a dedicated Kafka partition to every destination would be simple conceptually but infeasible at scale. ## Reliability Challenges - External endpoints may be: - Temporarily unavailable - Slow or unstable - Unreachable for hours or days - The system must avoid: - Losing customer logs - Sending duplicate logs - Delaying all destinations because one endpoint is unhealthy - Excessive resource usage - Overwhelming or effectively DDoSing a customer endpoint - Sending one HTTP request per log would be inefficient, so logs should be buffered and delivered in batches, much like packages going to the same address. ## Kafka Ordering and Blocked Progress - Waiting for each forwarding request to succeed before reading more Kafka data protects against data loss but can halt progress. - Continuing to read and acknowledge Kafka records before successful delivery risks losing logs. - Because offsets must be committed in order, one unavailable destination can block later records in the same partition—even if those records belong to healthy destinations. - This makes coordination between Kafka consumption, retries, batching, and concurrent delivery especially complex in a multi-tenant system. ## Lessons from Log Archives - Datadog had prior experience with similar delivery problems in its Log Archives feature. - Archiving was easier because: - Cloud object storage endpoints are generally more reliable. - Archiving has lower latency requirements. - Those lessons helped the team anticipate reliability and ordering pitfalls in Log Forwarding. ## Dedicated Kafka Topics per Destination - A possible solution would be to assign one or more Kafka partitions to each destination. - This would isolate destinations so that one slow endpoint could not block others. - However, the approach would require an impractically large number of Kafka topics or partitions as the number of customers and destinations grows.
Read original(opens in new tab)