You’ve Got (Too Much) Mail: Behind the Scenes of the 3/25/26 Voice Outage
Discord’s March 25, 2026 voice outage began when a Kubernetes configuration change abruptly terminated 17% of session processes. The resulting reconnection storm propagated through Discord’s realtime systems and overloaded voice-routing infrastructure, preventing many users from starting or joining calls. The incident exposed how failures in one distributed subsystem can create cascading load several services away. ## The Infrastructure Background - Discord is migrating stateful Elixir services to Kubernetes. - Each host runs thousands of in-memory processes for guilds, presence, messaging, and calls. - Deployments normally wait for a server’s entity count to reach zero before shutting it down, allowing processes to hand off their state safely. - The sessions service maintains one process for every connected device and carries websocket traffic, messages, presence updates, and other realtime events. - To reduce weekend CPU utilization, Discord planned to increase pod CPU and memory while proportionally reducing the number of pods. ## The Session Loss - The resource change was deployed to the first availability zone at 12:13 PDT. - Kubernetes terminated half of that zone’s pods because of the reduced replica count. - A safety check delayed process handoffs until other events completed, but the Kubernetes termination grace period expired first. - Because the service operated across three balanced zones, approximately 17% of Discord’s sessions stopped without a graceful handoff. - The outage lasted from 12:13 to 15:30 PDT, with users commonly seeing “Awaiting Endpoint.” ## How Elixir Monitoring Amplified the Failure - Discord relies heavily on Elixir `GenServer` processes, which process one mailbox message at a time. - Process monitors notify dependent processes whenever a monitored process exits. - The sudden loss of sessions therefore generated a large number of `{:DOWN, …}` notifications throughout the realtime infrastructure. - Guild and other processes stopped attempting to deliver updates to disconnected users, while the gateway began driving those users to reconnect. ## Reconnecting Users - The gateway handles websocket ingress and egress, creating sessions and maintaining client connections. - Session disconnections are normally expected and recoverable, whether caused by hardware, network problems, software bugs, or temporary connectivity loss. - When a session disappears, the gateway immediately instructs the client to reconnect. - It optimistically tries to resume the session through a gateway instance in the same zone, but the mass failure created a much larger reconnection surge than the system was designed to absorb. The incident demonstrates that reducing pod count can be dangerous in stateful distributed systems: an apparently routine capacity adjustment can cause abrupt process loss, trigger widespread retries, and overload unrelated downstream services. Changes to stateful workloads should be evaluated not only for steady-state resource usage but also for graceful shutdown behavior and synchronized failure scenarios.
Read original(opens in new tab)