When "idle" isn't idle: how a Linux kernel optimization became a QUIC bug (opens in new tab)
CUBIC, the default congestion controller in Linux and quiche, can become permanently stuck at its minimum congestion window after an early congestion collapse. Cloudflare found the bug in a QUIC test where packet loss stopped completely, yet CUBIC continued oscillating between recovery and congestion avoidance instead of increasing its sending rate. The problem was traced to a Linux TCP optimization for idle or app-limited connections, and ultimately fixed with an elegant near-one-line change. ## How CUBIC manages traffic - CUBIC controls the sender’s congestion window (`cwnd`), limiting how many bytes can be in flight. - It increases `cwnd` when acknowledgments arrive without loss and reduces it when loss suggests the network is overloaded. - As quiche’s default congestion controller, CUBIC affects a substantial amount of QUIC traffic. - Recovery from the minimum congestion window is an important but relatively under-tested part of congestion control. ## The failing test - The test downloaded a 10 MB file over HTTP/3 between local quiche client and server. - Network conditions included: - 10 ms RTT - 30% random packet loss during the first two seconds - No packet loss after two seconds - A 10-second timeout - The expected result was for CUBIC to reduce its window during loss, then steadily recover once the network became reliable. - Instead, approximately 60% of repeated 100-run test batches failed to finish in time. ## CUBIC becomes stuck at its minimum - After packet loss stopped at two seconds, bytes in flight remained flat rather than increasing. - CUBIC’s congestion window stayed at its minimum of 2,700 bytes—roughly two full-sized packets. - The controller repeatedly switched between recovery and congestion avoidance: - 999 transitions over about 6.7 seconds - One transition approximately every 14 ms - The oscillation closely matched the connection’s RTT, indicating that each ACK round was triggering the behavior. - Because the test was a download, client ACKs caused the server’s bytes in flight to fall to zero; the server then sent another two-packet burst, repeatedly provoking the faulty state transition. - Reno passed the same test 100% of the time, confirming that the issue was specific to CUBIC rather than the test setup. ## The connection to Linux TCP - The investigation focused on behavior when `bytes_in_flight == 0`, effectively an idle or app-limited condition. - A 2017 Linux kernel change addressed a TCP CUBIC issue after application idle periods. - Before the change, CUBIC’s epoch could remain unchanged for a long time while the application was idle. - When sending resumed, the elapsed time used by CUBIC could be extremely large, producing an excessively aggressive growth slope and dangerous congestion-window inflation. - The kernel optimization was intended to align CUBIC with the app-limited exclusion described in RFC 9438 §4.2-12. - Porting this logic to QUIC exposed an unintended interaction: repeated short periods with no bytes in flight could be interpreted incorrectly, causing CUBIC to cycle between states and remain at its minimum window. The practical lesson is that congestion-control implementations must test not only steady-state throughput and ordinary loss recovery, but also recovery from the minimum window and repeated app-limited or idle periods. In this case, a small adjustment to the idle-state handling broke the cycle and allowed CUBIC to recover normally.
