udp

2 posts

discord

How We Moved Discord Voice to the Edge (opens in new tab)

Discord migrated its voice and video infrastructure from roughly 30 hyperscaler regions to Cloudflare’s edge network, which spans more than 300 cities. The move brought measurable gains—over 80% of traffic now runs on Cloudflare, with Frankfurt seeing 34% lower ping and 42% lower packet loss—but the rollout exposed important limitations. Local server proximity alone is not enough; call placement and network peering are equally critical. ## Why Discord Moved Voice to the Edge - Traditional cloud regions left users in places such as Reykjavik, Auckland, Hawaii, and Lagos relatively far from voice servers. - Cloudflare’s 300-plus points of presence offered the possibility of hosting Discord’s software much closer to users. - Unlike typical CDN workloads, Discord needed to route live UDP packets for voice and video calls rather than cache static content. - More than 80% of Discord voice and video traffic now uses Cloudflare, and 70% of regions have improved year over year. ## Iceland: Local Servers Can Hurt Mixed Calls - Discord’s first test took place in Reykjavik, where users previously connected to servers in Rotterdam. - Iceland-only calls improved: - Ping decreased 9%. - Packet loss decreased 11%. - Mixed-region calls became worse: - Ping for non-Iceland participants increased 2.7 times. - Packet loss rose 9%. - Discord assigns one SFU to host an entire call, so all participants send traffic to that server. - A call started by one Icelandic user could therefore force German participants to route packets to Iceland and back. - Discord concluded that new edge locations help only when calls hosted there are primarily local; mixed calls require smarter host-placement logic. ## Rotterdam: Network Peering Was the Bottleneck - Discord next moved Rotterdam traffic to Cloudflare’s Amsterdam PoP. - Most regional ISPs performed normally, but Orange users experienced: - More than one second of latency during peak periods. - A 30% regression in voice freeze ratio. - The issue was not Cloudflare’s Amsterdam infrastructure itself, but the path between Orange and Cloudflare. - Traffic traveled through Telia’s transit backbone, where the Telia–Orange connection was already saturated. - Adding more traffic made the congestion worse. - Discord reverted the migration after about ten days. - Cloudflare pursued direct peering with Orange and added SFUs in Paris and London to provide shorter paths. ## A More Careful Migration Strategy - Discord changed its rollout process from capacity-based scheduling to peering-based analysis. - Before shifting meaningful traffic to a region, the team now checks whether Cloudflare has sufficient peering capacity with major local ISPs. - The migration slowed from a planned large-scale rollout to a region-by-region deployment. - The experience demonstrated that: - A geographically closer PoP does not always produce the best call experience. - SFU host placement matters for calls spanning multiple regions. - Transit routes and ISP peering can be more important than physical distance. Discord’s edge migration is succeeding, but only through incremental deployment, detailed traffic analysis, and improved call-placement decisions. The practical lesson is that low-latency infrastructure depends on the complete network path—not simply on adding servers closer to users.

cloudflare

Introducing Programmable Flow Protection: custom DDoS mitigation logic for Magic Transit customers (opens in new tab)

Programmable Flow Protection lets Magic Transit Enterprise customers define custom DDoS mitigation logic for proprietary UDP protocols. Customers write eBPF programs that identify valid and malicious packets, then deploy them across Cloudflare’s global network to pass, drop, or challenge traffic. Currently in beta for an additional cost, the system addresses the limitations of generic blocking and rate limiting. ## Custom Protection for Proprietary UDP - Existing Cloudflare protections understand established protocols such as TCP, DNS, NTP, RDP, and SIP. - Proprietary UDP protocols are harder to protect because Cloudflare cannot interpret their application-level payloads. - Customers can now define what constitutes “good” and “bad” traffic using their own protocol knowledge. - Programs can drop or challenge invalid packets before they reach the customer’s origin. ## Why Generic UDP Mitigation Falls Short - UDP is connectionless and optimized for speed, making it useful for gaming, VoIP, and streaming. - When traffic does not match a known protocol, mitigation typically falls back to: - Blocking a destination IP and port - Applying a generic rate limit - These approaches cannot distinguish legitimate packets from attack traffic, potentially causing lag or connection loss for real users. - Fixed rate limits may also be inappropriate: - A network expecting 1 Gbps may need stricter limits. - A network expecting 25 Gbps may require more permissive thresholds. ## How Programmable Flow Protection Works - Customers upload custom eBPF programs that run on every packet destined for their network. - Programs execute in userspace rather than kernel space, providing isolation and flexibility across different customers and use cases. - Execution occurs after Cloudflare’s existing DDoS protections, preserving baseline security coverage. - Like kernel-based XDP eBPF programs, these programs: - Compile to BPF bytecode - Pass safety and termination verification - Run inside a lightweight, isolated virtual machine - Cloudflare provides specialized helpers for: - Maintaining client state between packet executions - Performing cryptographic validation - Sending challenge packets to clients ## Example: Protecting a Proprietary Game Protocol - A gaming provider running on UDP port 207 could inspect its proprietary application header. - If the header contains a protocol-specific token, the customer’s eBPF program can: - Parse the packet - Extract part of the token, such as its final byte - Pass packets with the expected value - Drop packets that fail validation - This allows legitimate players’ traffic through even when attacks use randomized source addresses, ports, and payloads. Programmable Flow Protection is best suited to Magic Transit customers whose custom UDP protocols cannot be protected effectively by standard protocol-aware controls. By combining customer-specific packet logic with Cloudflare’s global network and stateful challenge mechanisms, it enables more precise mitigation than blanket blocking or generic rate limiting.