quic

6 posts

cloudflare

An API for MoQ: provision your own isolated relays (opens in new tab)

Cloudflare has moved MoQ from an open testing network toward production use by adding isolated relays and authentication. Its provisioning API and dashboard let applications create globally available relay scopes and issue separate publisher and subscriber credentials, without deploying infrastructure. The beta supports MoQ Transport drafts 14 and 16 and is currently free. ## MoQ and Its Architecture - MoQ is an open IETF publish/subscribe protocol built on QUIC, the transport used by HTTP/3. - Publishers send named data streams, while subscribers request those streams through relays. - Relays copy data to subscribers without inspecting its contents, enabling efficient fan-out. - The same system can support live video, video calls, low-latency messaging, and other real-time workloads. - Using CDN-based relays avoids the cost and complexity of operating specialized media servers. ## From Open Preview to Production - Cloudflare’s initial MoQ preview exposed an unauthenticated relay on servers in more than 330 cities. - Over 1,000 clients continue to use the open endpoints daily for testing and development. - The lack of authentication made the preview unsuitable for applications requiring confidentiality or role-based permissions. - For example, live auction applications need broadcasters to publish while viewers can only subscribe. ## Isolated Cloudflare Relays - Provisioning a relay creates an isolated scope across Cloudflare’s existing global network rather than starting a VM, container, or dedicated process. - Each scope separates an application’s namespaces, tracks, and objects from those of other applications. - Clients connect through an Anycast endpoint, with Cloudflare handling global routing. - Relays become available within seconds, without regional capacity planning, load balancers, or server management. - Cloudflare compares the model to creating a virtual host rather than deploying a new web server. ## Provisioning API and Access Tokens - The control-plane API manages relay configuration and credentials but does not handle the media flowing through relays. - A relay defines the isolated application scope. - Tokens grant `publish`, `subscribe`, or both operations for a specific relay. - Tokens can have expiration times and can be revoked independently. - Creating a relay automatically returns: - A token capable of publishing and subscribing. - A subscribe-only token intended for viewers. - Additional narrowly scoped tokens can be created through the API or dashboard. - Current tokens apply to an entire relay; Cloudflare is working with the MoQ community on more granular authorization. ## Using the API and Dashboard - A relay can be created with one authenticated API request containing its name. - Tokens can be added through an endpoint such as `/moq/relays/$RELAY_ID/tokens`. - The dashboard workflow is available under **Media > Realtime > MoQ Relay**. - Applications should provide broadcasters with publish-capable credentials and viewers with subscribe-only credentials. ## Connecting Clients - Clients send their token when opening a MoQ session. - The relay enforces the token’s permitted operations. - Cloudflare’s open-source `moq-rs` tools can be used with media generated by tools such as `ffmpeg`. Cloudflare’s authenticated, isolated MoQ relays make the protocol more practical for production real-time applications. Developers can use the API or dashboard to provision a globally distributed relay and manage separate, expiring credentials without operating their own media infrastructure.

cloudflare

When "idle" isn't idle: how a Linux kernel optimization became a QUIC bug (opens in new tab)

CUBIC, the default congestion controller in Linux and quiche, can become permanently stuck at its minimum congestion window after an early congestion collapse. Cloudflare found the bug in a QUIC test where packet loss stopped completely, yet CUBIC continued oscillating between recovery and congestion avoidance instead of increasing its sending rate. The problem was traced to a Linux TCP optimization for idle or app-limited connections, and ultimately fixed with an elegant near-one-line change. ## How CUBIC manages traffic - CUBIC controls the sender’s congestion window (`cwnd`), limiting how many bytes can be in flight. - It increases `cwnd` when acknowledgments arrive without loss and reduces it when loss suggests the network is overloaded. - As quiche’s default congestion controller, CUBIC affects a substantial amount of QUIC traffic. - Recovery from the minimum congestion window is an important but relatively under-tested part of congestion control. ## The failing test - The test downloaded a 10 MB file over HTTP/3 between local quiche client and server. - Network conditions included: - 10 ms RTT - 30% random packet loss during the first two seconds - No packet loss after two seconds - A 10-second timeout - The expected result was for CUBIC to reduce its window during loss, then steadily recover once the network became reliable. - Instead, approximately 60% of repeated 100-run test batches failed to finish in time. ## CUBIC becomes stuck at its minimum - After packet loss stopped at two seconds, bytes in flight remained flat rather than increasing. - CUBIC’s congestion window stayed at its minimum of 2,700 bytes—roughly two full-sized packets. - The controller repeatedly switched between recovery and congestion avoidance: - 999 transitions over about 6.7 seconds - One transition approximately every 14 ms - The oscillation closely matched the connection’s RTT, indicating that each ACK round was triggering the behavior. - Because the test was a download, client ACKs caused the server’s bytes in flight to fall to zero; the server then sent another two-packet burst, repeatedly provoking the faulty state transition. - Reno passed the same test 100% of the time, confirming that the issue was specific to CUBIC rather than the test setup. ## The connection to Linux TCP - The investigation focused on behavior when `bytes_in_flight == 0`, effectively an idle or app-limited condition. - A 2017 Linux kernel change addressed a TCP CUBIC issue after application idle periods. - Before the change, CUBIC’s epoch could remain unchanged for a long time while the application was idle. - When sending resumed, the elapsed time used by CUBIC could be extremely large, producing an excessively aggressive growth slope and dangerous congestion-window inflation. - The kernel optimization was intended to align CUBIC with the app-limited exclusion described in RFC 9438 §4.2-12. - Porting this logic to QUIC exposed an unintended interaction: repeated short periods with no bytes in flight could be interpreted incorrectly, causing CUBIC to cycle between states and remain at its minimum window. The practical lesson is that congestion-control implementations must test not only steady-state throughput and ordinary loss recovery, but also recovery from the minimum window and repeated app-limited or idle periods. In this case, a small adjustment to the idle-state handling broke the cycle and allowed CUBIC to recover normally.

slack

From Custom to Open: Scalable Network Probing and HTTP/3 Readiness with Prometheus (opens in new tab)

Slack needed better client-side observability while migrating edge services to HTTP/3, which uses QUIC over UDP rather than TCP. Existing SaaS tools and Prometheus Blackbox Exporter could not probe HTTP/3 endpoints, so an intern added QUIC support using Go’s `quic-go` library and open-sourced it. The result unified HTTP/1.1, HTTP/2, and HTTP/3 monitoring while making the capability available to the broader Prometheus community. ## Limitations of Legacy Monitoring - Slack used a mix of commercial monitoring services and internal tools for network measurements. - HTTP/3 introduced a major observability gap because it runs over QUIC/UDP. - Existing SaaS solutions lacked built-in HTTP/3 probing. - Prometheus Blackbox Exporter had no native QUIC support. - Without probing at scale, Slack could not reliably measure round-trip times, detect regressions to HTTP/2, or monitor hundreds of thousands of HTTP/3 endpoints. ## Adding QUIC Support to Blackbox Exporter - Intern Sebastian Feliciano selected `quic-go` because of its adoption and first-class Go HTTP client support. - The implementation used an `http3.Transport` with TLS and QUIC configuration: ```go http3Transport := &http3.Transport{ TLSClientConfig: tlsConfig, QUICConfig: &quic.Config{}, } ``` - The new transport was attached to a standard Go `http.Client`. - The implementation preserved Blackbox Exporter’s existing configuration and composability patterns. - Sebastian open-sourced the feature and eventually got it accepted upstream. ## In-House Integration and Operational Benefits - Because upstream review could take longer than the internship timeline, Slack built an internal system around the new functionality. - Grafana now provides a unified view of HTTP/1.1, HTTP/2, and HTTP/3 metrics. - Operators can compare protocol performance and correlate it with other telemetry. - Improved visibility supports more accurate alerts and faster debugging of HTTP/3 issues. ## Future Enhancements - **SNI routing tests:** Verify that shared edge infrastructure routes hostnames to the correct backend and presents the correct TLS certificate. - **End-to-end path visualization:** Map network hops between monitoring agents and endpoints to identify latency spikes or packet loss more precisely. ## Broader Lessons - Observability should be established before a major protocol or infrastructure migration. - Filling gaps through open source can benefit both the organization and the wider engineering community. - Supporting emerging protocols such as QUIC early helps future-proof monitoring systems. Slack recommends trying the new QUIC functionality in Prometheus Blackbox Exporter and contributing to its continued development.

cloudflare

Ending the "silent drop": how Dynamic Path MTU Discovery makes the Cloudflare One Client more resilient (opens in new tab)

Cloudflare’s Dynamic Path MTU Discovery (PMTUD) helps prevent connections from silently failing when network paths cannot carry large encrypted packets. Using active probing through MASQUE and QUIC, the Cloudflare One Client determines the largest reliable packet size and adjusts its virtual interface accordingly. This makes applications more resilient across restrictive, changing networks without relying on ICMP error messages. ## The PMTUD Black Hole Problem - Networks have a maximum transmission unit (MTU), typically 1500 bytes on Ethernet. - Encryption and security metadata reduce the space available for application data. - LTE/5G, satellite, public safety, and other specialized networks may support smaller MTUs, such as 1300 bytes. - Routers should send ICMP messages when packets are too large, but firewalls and middleboxes often drop those messages. - The sender continues transmitting oversized packets, leaving uploads, video calls, SSH sessions, or other applications stuck until they time out. ## Active Path Discovery with MASQUE - Cloudflare implements RFC 8899 Datagram Packetization Layer PMTUD. - The Cloudflare One Client sends encrypted probes of different sizes to the Cloudflare edge. - By observing which probes arrive, it identifies the usable MTU without depending on blocked ICMP feedback. - Probing narrows the range from the supported maximum toward the precise path capacity. - The process runs in the background and is designed not to disrupt active connections. ## Adapting to Changing Networks - The client dynamically changes its virtual interface MTU based on the discovered path. - It periodically revalidates the path, allowing it to respond when users move between networks. - For example, a connection can transition from 1500-byte Wi-Fi to 1300-byte cellular connectivity without interrupting application sessions. ## Benefits for Critical and Everyday Connectivity - First responders can maintain stable CAD and other mission-critical connections across NAT layers, tower handoffs, and fluctuating signal conditions. - Hybrid workers benefit from fewer stalled transfers and more reliable video calls on hotel, cellular, and double-NAT networks. - The client hides much of the underlying network instability from applications. Cloudflare One Client users running the MASQUE protocol can use PMTUD on Windows, macOS, and Linux at no additional cost.

cloudflare

A QUICker SASE client: re-building Proxy Mode (opens in new tab)

Cloudflare rebuilt the Cloudflare One Client’s proxy mode to address performance problems caused by translating TCP traffic into IP packets through WireGuard. The new design uses HTTP/3 and QUIC streams for direct Layer 4 proxying, eliminating the smoltcp translation layer. Internal tests showed download and upload speeds doubling while latency decreased significantly. ## Limitations of the Original Proxy Architecture - Proxy mode exposed a local SOCKS5 or HTTP proxy for broad application compatibility. - WireGuard operates at Layer 3, while proxy traffic arrives as Layer 4 TCP streams. - The Client used the Rust-based `smoltcp` stack to convert TCP streams into IP packets. - Cloudflare’s edge then converted those packets back into TCP streams. - This added overhead, limited access to modern TCP features, and caused sluggish performance for media-heavy websites, large transfers, and video calls. ## Direct Layer 4 Proxying with QUIC - Cloudflare deprecated WireGuard for proxy mode and adopted QUIC-based transport. - HTTP/3’s `CONNECT` method encapsulates proxy traffic directly in QUIC streams rather than breaking it into Layer 3 packets. - The new architecture: - Removes the smoltcp translation layer. - Uses QUIC’s built-in congestion and flow control. - Allows the Client and Cloudflare edge to tune transport parameters for performance. - Testing showed approximately doubled upload and download speeds and substantially reduced latency. ## Use Cases That Benefit - **Third-party VPN coexistence:** Users can combine legacy VPNs for on-premises resources with zero trust web security without imposing as much performance loss. - **Application partitioning:** Specific browser traffic can be routed through Cloudflare Gateway while other operating-system traffic remains on the local network. - **High-bandwidth workloads:** Streaming, large dataset transfers, and other data-intensive applications receive faster proxy connections. - **Developer and CLI workflows:** Tools using the SOCKS5 listener benefit from lower-latency API calls and data transfers. ## Availability and Configuration - The improvement requires Cloudflare One Client version `2025.8.779.0` or later on Windows, macOS, or Linux. - In the Cloudflare One dashboard: - Go to **Teams & Resources > Devices > Device profiles > General profiles**. - Set **Service mode** to **Local proxy mode**. - Set **Device tunnel protocol** to **MASQUE**. - Verify the active protocol with: ```bash warp-cli settings | grep protocol ``` Organizations using proxy mode should upgrade the client and switch to MASQUE to gain the new QUIC-based performance improvements.

cloudflare

What we know about Iran’s Internet shutdown (opens in new tab)

Iran’s government effectively disconnected the country from the global Internet on January 8, 2026, amid escalating nationwide protests. Cloudflare observed a near-total loss of traffic after major Iranian networks withdrew most announced IPv6 address space and then lost connectivity almost entirely. Brief access windows on January 9 quickly ended, and the shutdown remained in place through January 10. ## Background and Earlier Shutdowns - Iran has previously restricted Internet access during protests: - More than five days of disruption followed fuel-price protests in November 2019. - Connectivity was disrupted across multiple providers during protests after Mahsa/Zhina Amini’s death in September 2022. - Internet traffic had already been below normal at the beginning of 2026, suggesting connectivity problems preceded the complete shutdown. ## Connectivity Collapsed on January 8 - At 11:50 UTC, Iranian networks reduced announced IPv6 address space by 98.5%, from over 48 million `/48` blocks to roughly 737,000. - This caused IPv6’s share of human-generated traffic to fall from about 12% to 2%, before IPv6 traffic nearly disappeared later that afternoon. - Between 16:30 and 17:00 UTC, overall traffic dropped by nearly 90%. - Major providers affected included: - MCCI (AS197207) - IranCell (AS44244) - TCI (AS58224) - By approximately 18:45 UTC, traffic from Iran had fallen effectively to zero, indicating a nationwide disconnection from the global Internet. ## Brief Connectivity on January 9 - Internal traffic remained below 0.01% of pre-shutdown peaks. - Access to Cloudflare’s `1.1.1.1` DNS resolver briefly returned around 10:00 UTC, producing a short-lived spike in requests. - Several universities also regained connectivity temporarily, including the University of Tehran, Sharif University of Technology, Tehran University of Medical Science, and Tarbiat Modares University. - Traffic from these networks disappeared again by roughly 15:00 UTC. ## Filtering Changes Before the Shutdown - HTTP/3 and QUIC usage declined sharply before the full outage. - On IranCell, HTTP/3 usage fell from as high as 40% to 5% by December 31 and continued declining. - On TCI, HTTP/3 dropped below 5% around January 3. - These changes may indicate increasingly severe filtering and upgraded whitelisting, according to MahsaNet. ## Ongoing Disconnection - Since January 10, Iran’s Internet traffic has shown no significant recovery. - Traffic remains at only a fraction of one percent of previous levels. - Cloudflare continues monitoring the situation through Radar’s traffic and routing data. The available measurements strongly indicate a deliberate, nationwide Internet shutdown rather than an ordinary network failure. Cloudflare Radar’s traffic and routing pages provide the most practical way to follow any restoration or further changes.