Not just another network latency issue: How we unraveled a series of hidden bottlenecks
Repeated high-startup-latency pages in Datadog’s usage estimation service were caused by several independent bottlenecks rather than application changes. The investigation eventually identified four issues: CPU-throttled Envoy sidecars, a Linux kernel bug affecting ENA transmit queues, insufficient EC2 network bandwidth, and requests routed to terminating cache pods. Fixing each layer progressively reduced remote-cache p99 latency from roughly one second to its normal level of about 100 ms. ## Service Architecture and the Original Symptoms - The service consists of router, counter, and aggregator applications. - At startup, `counter` loads data from a remote cache into a local cache. - While the local cache is populating, request processing is slower and backlog grows. - Normal p99 remote-cache latency was approximately 100 ms, but it exceeded one second during deployments. - Scaling the remote cache did not help, indicating that the cache itself was not underprovisioned. ## CPU-Throttled Envoy Sidecars - Requests to the remote cache passed through an Envoy sidecar that batched queries into packets. - When `counter` restarted, Envoy reached its two-core CPU limit and was throttled. - Delayed request and response processing caused retries, TCP retransmits, and increased remote-cache latency. - Increasing Envoy’s CPU allocation eliminated the issue in staging and reduced production latency, but did not fully resolve rollout spikes. ## Linux Kernel and ENA Transmit-Queue Bug - Investigation of system and network metrics revealed a Linux kernel bug affecting AWS Elastic Network Adapter traffic. - The kernel mapped all outbound traffic to the first transmit queue instead of distributing it across eight queues. - This limited throughput and caused retransmits during high-traffic periods such as deployments. - A hotfix distributed traffic across all eight queues. - The change removed non-rollout latency spikes but rollout latency still fluctuated between 200 and 600 ms. ## EC2 Network Bandwidth Limits - ENA metrics showed that instances exceeded AWS inbound and outbound bandwidth allowances. - AWS dropped packets at the hypervisor when those limits were exceeded, causing retransmissions and slower cache requests. - Migrating to network-optimized EC2 instance types with higher bandwidth allowances largely restored p99 latency to around 100 ms. - Occasional one-second spikes continued despite the improvement. ## Requests Sent to Terminating Cache Pods - Remaining spikes correlated with remote-cache pods that were shutting down. - Clients continued sending requests to terminating pods, leading to one-second timeouts and retries. - The cache’s graceful-shutdown behavior did not adequately wait for Envoy clients’ in-flight requests. - The team added a `preStop` hook that sets an `XXX_MAINTENANCE_MODE` key to notify clients before termination and began coordinating shutdown with outstanding requests. The incident demonstrates the importance of tracing latency across the entire request path, from application startup through proxies, kernel networking, hardware interfaces, cloud bandwidth limits, and pod lifecycle behavior. Layered system metrics and component-level investigation were necessary to eliminate alert fatigue and restore reliable deployment behavior.
Read original(opens in new tab)