discordMay 4, 2026

Discord Patch Notes: May 4, 2026 (opens in new tab)

Discord’s May 4, 2026 patch focuses on reliability, performance, usability, and server administration. Major improvements include nearly 5% faster Android video startup, reorganized desktop settings, faster Soundboard access, and self-updating Linux support. The release also fixes numerous bugs across account profiles, search, payments, moderation, notifications, and mobile platforms. ## Performance and Platform Improvements - Android video startup improved by 4.89%, bringing average feed startup time below 600 milliseconds. - Soundboard data now loads when users join a voice channel rather than when they first open the Soundboard, reducing initial access time. - Linux now uses Discord’s Rust-based automatic updater, eliminating the need for manual update installation. - Linux installation now supports `.rpm` and `.pkg.tar.zst` packages. ## Server Administration and Moderation - Discord fixed several issues affecting server administrators and moderators. - Fixes covered permissions, user states, and interactions with account-safety systems. - Discord encouraged administrators to report remaining problems through its community bug megathread. ## Desktop Settings and Organization - Desktop settings were consolidated into three pages: - Appearance - Accessibility - Developer - The former Appearance, Accessibility, Chat, Streamer Mode, and Advanced sections were reorganized. - Several settings received clearer wording and improved layouts. - Fixed problems involving theme synchronization, profile editing, unsaved-change warnings, currency labels, and server administration prompts. ## General User Interface Fixes - “GODLIKE!!” and “BEYOND GODLIKE!!” copy-username messages now have opaque backgrounds. - The Quick Switcher can now accept invite links, join the associated server, and navigate to it. - Fixed display problems involving long nicknames, search filters, avatar controls, profile links, status indicators, and Nitro badges. - French users can now search for “sondage” without the search term being incorrectly split. - Android search filters now correctly display their active blue state. - Fixed several modal, button-spacing, and navigation issues across desktop and mobile. ## Profiles, Avatars, and Customization - Per-server avatar links now copy correctly instead of copying the main profile avatar link. - Clearing per-server pronouns no longer repopulates them from the main profile; the main pronouns appear only as a placeholder. - iOS no longer switches unexpectedly to a per-server profile after copying a username. - Recent-avatar delete controls now appear correctly. - Custom Status editing opens above the full profile instead of replacing it. - Fixed an issue where clicking outside profile editing discarded changes without warning. ## Notifications, Search, and Media - Desktop Inbox no longer crashes when many notifications are cleared rapidly. - Mobile search tabs for Media, Pins, Files, and Links no longer spam errors or retry repeatedly after a connection loss. - Expired public image links in the Inbox preview now behave more reliably. - Fixed an issue where long mobile search-result nicknames obscured timestamps. ## Payments and Server Boosting - Server Boost marketing audio now stops when users enter the purchase flow. - Pressing Escape during payment no longer closes the underlying Server Boost page instead of the payment window. - Nitro trial recipient checkboxes now select friends correctly. - Fixed overlapping controls in Profile Settings and spacing issues on the domain-connection page. - GBP subscription settings now show the currency’s full name. Discord’s changes combine small interface corrections with measurable performance and platform improvements. Users should receive the fixes progressively, since deployment may still be rolling out across platforms.

networkingdiscordandroidios+3

cloudflareMar 5, 2026

Ending the "silent drop": how Dynamic Path MTU Discovery makes the Cloudflare One Client more resilient (opens in new tab)

Cloudflare’s Dynamic Path MTU Discovery (PMTUD) helps prevent connections from silently failing when network paths cannot carry large encrypted packets. Using active probing through MASQUE and QUIC, the Cloudflare One Client determines the largest reliable packet size and adjusts its virtual interface accordingly. This makes applications more resilient across restrictive, changing networks without relying on ICMP error messages. ## The PMTUD Black Hole Problem - Networks have a maximum transmission unit (MTU), typically 1500 bytes on Ethernet. - Encryption and security metadata reduce the space available for application data. - LTE/5G, satellite, public safety, and other specialized networks may support smaller MTUs, such as 1300 bytes. - Routers should send ICMP messages when packets are too large, but firewalls and middleboxes often drop those messages. - The sender continues transmitting oversized packets, leaving uploads, video calls, SSH sessions, or other applications stuck until they time out. ## Active Path Discovery with MASQUE - Cloudflare implements RFC 8899 Datagram Packetization Layer PMTUD. - The Cloudflare One Client sends encrypted probes of different sizes to the Cloudflare edge. - By observing which probes arrive, it identifies the usable MTU without depending on blocked ICMP feedback. - Probing narrows the range from the supported maximum toward the precise path capacity. - The process runs in the background and is designed not to disrupt active connections. ## Adapting to Changing Networks - The client dynamically changes its virtual interface MTU based on the discovered path. - It periodically revalidates the path, allowing it to respond when users move between networks. - For example, a connection can transition from 1500-byte Wi-Fi to 1300-byte cellular connectivity without interrupting application sessions. ## Benefits for Critical and Everyday Connectivity - First responders can maintain stable CAD and other mission-critical connections across NAT layers, tower handoffs, and fluctuating signal conditions. - Hybrid workers benefit from fewer stalled transfers and more reliable video calls on hotel, cellular, and double-NAT networks. - The client hides much of the underlying network instability from applications. Cloudflare One Client users running the MASQUE protocol can use PMTUD on Windows, macOS, and Linux at no additional cost.

networkingcloudflarequicencryption+4

cloudflareFeb 21, 2026

Cloudflare outage on February 20, 2026 (opens in new tab)

Cloudflare suffered a 6-hour, 7-minute outage on February 20, 2026, after a software change unintentionally withdrew Internet routes for some Bring Your Own IP (BYOIP) customers. The incident was not related to a cyberattack; a buggy automated cleanup task altered customer prefix and service configurations. Cloudflare reverted the change, restored affected prefixes, and is revising its Addressing API workflows to reduce production risk. ## Customer Impact - Approximately 1,100 of Cloudflare’s 6,500 advertised prefixes were withdrawn between 17:56 and 18:46 UTC. - This affected about 25% of the 4,306 BYOIP prefixes advertised globally. - Impacted applications became unreachable from the Internet and experienced connection failures and timeouts. - Customers initially encountered BGP Path Hunting, where networks repeatedly searched for a route until connections timed out. - The `one.one.one.one` website returned HTTP 403 errors and an “Edge IP Restricted” message. - DNS resolution through the 1.1.1.1 resolver, including DNS over HTTPS, was not affected. - The incident did not affect every BYOIP customer because the configuration change was applied incrementally and was reverted before reaching everyone. ## Recovery Efforts - Engineers detected the issue through failures involving `one.one.one.one` and reverted the change. - Cloudflare published dashboard guidance at 19:19 UTC, allowing many customers to re-advertise their prefixes themselves. - Around 800 prefixes were restored by approximately 20:20 UTC. - About 300 prefixes could not be restored through the dashboard because their service configurations had been removed from edge servers. - Engineers manually restored those remaining prefixes at 23:03 UTC. - Some customers continued to experience latency and failures while addressing configuration state propagated back to the edge. ## The Addressing API - Cloudflare’s Addressing API is the authoritative dataset for IP addresses present on its network. - Changes to the API drive workflows that propagate address and routing updates across Cloudflare’s edge. - The normal process is: - Customers request advertisement or withdrawal through the Addressing API or BGP Control. - The API instructs machines to change prefix advertisements. - Routers update BGP after enough machines receive the change. - Customers bind Cloudflare products to their BYOIP ranges. - Because the API is closely connected to production systems, manual changes are risky. - Cloudflare’s “Code Orange: Fail Small” initiative aims to replace manual Addressing API operations with safer, automated, health-checked workflows. ## Root Cause: Faulty BYOIP Cleanup Automation - The failed change automated the removal of prefixes from BYOIP, a task that had previously been performed manually. - A recurring cleanup sub-task searched for BYOIP prefixes marked for deletion and removed them. - The cleanup task issued the API request: ```go /v1/prefixes?pending_delete ``` - The request contained a bug in how the API query was interpreted. - As a result, the cleanup process unintentionally withdrew customer prefixes and removed related service configurations from some edge servers. - The incident lasted much longer than the initial withdrawal because restoring both advertisements and edge configuration state required extensive automated and manual recovery. Cloudflare’s main corrective direction is to make Addressing API changes safer through incremental, health-mediated deployment, stronger safeguards around automated deletion, and elimination of risky manual production workflows.

networkinggoapi-designdns+3

cloudflareJan 23, 2026

Route leak incident on January 22, 2026 (opens in new tab)

On January 22, 2026, a routing-policy automation error caused Cloudflare to leak IPv6 BGP prefixes from its Miami data center. For 25 minutes, external traffic was redirected through Miami, congesting backbone links, increasing latency and packet loss, and causing some traffic to be dropped by firewall filters. The incident resulted from removing prefix-list constraints, leaving a Juniper policy that accepted and externally advertised unintended internal routes. ## What a BGP Route Leak Is - A route leak occurs when an autonomous system advertises routes it is not supposed to forward. - This violates valley-free routing principles, such as sending routes learned from a peer onward to another peer or provider. - The leaking network may lack the capacity or firewall rules to handle the redirected traffic. - Cloudflare’s incident involved a mixture of Type 3 and Type 4 route leaks under RFC 7908. ## Incident Impact and Timeline - The triggering automation change was merged at **19:52 UTC**. - At **20:25 UTC**, it ran on a single Miami edge router and caused unexpected advertisements to peers and transit providers. - The network team began investigating at **20:40 UTC** and formally coordinated the incident at **20:44 UTC**. - At **20:50 UTC**, an operator reverted the bad configuration and paused automation on the router. - The leak lasted approximately **25 minutes**, causing: - Congestion on Miami backbone infrastructure - Increased packet loss and latency for some Cloudflare customers - Traffic from unrelated external networks to be funneled through Miami - Packet drops from firewalls configured to accept only Cloudflare-related traffic - The triggering code change was reverted at **21:47 UTC**, and automation was later verified and resumed. ## The Configuration Error - Cloudflare intentionally removed Miami advertisements for a Bogotá data center after infrastructure changes made the Miami-to-Bogotá forwarding path unnecessary. - The change removed `6-BOG04-SITE-LOCAL` prefix-list conditions from multiple export policies. - Those prefix lists had constrained policies to specific Bogotá prefixes. - Once removed, policies such as `6-TELIA-ACCEPT-EXPORT` still matched routes with `route-type internal` and accepted them for export. - On Juniper JunOS and JunOS EVO, `route-type internal` matches any non-external route, including Internal BGP (IBGP) routes. - Consequently, routes that should have remained internal were treated as exportable and advertised to external peers and providers. - The problem affected IPv6 traffic only. ## Response and Remediation - Operators manually reverted the generated router configuration. - Automation was paused on the affected Miami router to prevent the faulty policy from being reapplied. - The source-code change was reverted, and the router was checked before automation resumed. Cloudflare’s incident demonstrates that removing seemingly obsolete route filters can unintentionally broaden a policy. Export policies should explicitly constrain acceptable prefixes and include validation or safeguards that prevent internal routes from being advertised externally.

networkingbgpipv6route-leak+2

cloudflareJan 14, 2026

What came first- the CNAME or the A record (opens in new tab)

A memory-optimization change in Cloudflare’s 1.1.1.1 resolver accidentally reordered DNS records, placing CNAMEs after A/AAAA records. Although DNS record order is generally considered irrelevant, some clients—including glibc’s `getaddrinfo`—process answers sequentially and require CNAMEs to appear first. The resulting failures affected users globally until Cloudflare reverted the release. ## Incident Timeline - **December 2, 2025:** The record-reordering change was added. - **December 10:** It reached the testing environment. - **January 7, 2026:** Global deployment began. - **January 8, 17:40 UTC:** The change reached 90% of servers. - **18:19:** The incident was declared. - **18:27:** The release was reverted. - **19:55:** The revert completed and the impact ended. ## How CNAME Chains Are Resolved - A hostname may point through multiple aliases before reaching an A or AAAA record: - `www.example.com → cdn.example.com → server.cdn-provider.com → 198.51.100.1` - Each record has its own TTL and may expire independently. - If only part of a chain expires, 1.1.1.1 can reuse the cached portion and resolve only the missing records. - The resolver then combines the cached CNAME records with newly resolved address records. ## The Memory Optimization That Changed Ordering - Previously, the resolver created a new list: - Insert the existing CNAME chain first. - Append the newly resolved A/AAAA records afterward. - The optimization avoided allocations and copies by appending new CNAME records directly to the existing answer list. - This caused some responses to place address records before CNAME records. ## Why Some DNS Clients Failed - Many clients treat answer-section ordering as irrelevant, but some parse records sequentially. - These clients: - Start by looking for records matching the original queried name. - Update the expected name when they encounter a CNAME. - Accept the corresponding A or AAAA record only after that update. - With the expected order, the client sees the CNAME first and then accepts the address record. - With the address record first, it ignores the address because it does not yet match the expected name. After encountering the CNAME, there are no records left to process, so it reports an empty response. - The affected implementation included glibc’s `getaddrinfo`, widely used for DNS resolution on Linux. The incident demonstrates that even seemingly insignificant DNS response ordering can matter in practice. Resolver implementations should preserve CNAME-before-address ordering, and DNS clients should avoid assuming that record order is meaningful unless the protocol explicitly requires it.

networkingrustdnsglibc+3

datadogJan 7, 2026

Hardening eBPF for runtime security: Lessons from Datadog Workload Protection (opens in new tab)

eBPF gives security tools broad, efficient, and relatively safe access to Linux kernel activity, making it well suited for runtime threat detection. Datadog chose it for Workload Protection after comparing kernel modules, tracing interfaces, ptrace, seccomp, Linux Audit, and other approaches. However, five years of production use across diverse kernels showed that eBPF’s safety and performance benefits are not automatic; reliability, compatibility, observability, and operational discipline are essential at scale. ## Why Runtime Workload Protection Is Needed - Static analysis and vulnerability scanning cannot catch every threat. - Zero-days and vulnerable third-party dependencies can remain active while patches are being prepared or deployed. - Workload Protection is intended to: - Monitor known-vulnerable workloads until they can be patched. - Continuously observe all workloads. - Detect and help mitigate previously unknown vulnerabilities during incident response. ## Alternatives Evaluated Datadog evaluated a broad range of Linux monitoring and instrumentation mechanisms: - **Linux kernel modules** - Offer deep access and can hook or replace almost any kernel function. - Are invasive and often considered too risky for production infrastructure. - **Traditional tracing interfaces** - Include inotify, fanotify, kprobes, tracepoints, and perf events. - Provide useful visibility but generally need to be combined for comprehensive coverage. - **ptrace and seccomp-bpf** - Can provide detailed user-space process visibility. - Are less suitable as a unified solution for monitoring the whole system. - **Linux Audit** - Produces configurable streams for process execution, file access, and network activity. - Is widely used by security tooling but has its own performance and operational tradeoffs. - **Other mechanisms** - Netlink, LD_PRELOAD, and binfmt_misc were also considered. - Each involves compromises in reliability, visibility, or system impact. ## Why eBPF Stood Out - **Safety checks** - The kernel statically verifies eBPF bytecode before loading it. - Verification detects issues such as infinite loops and unsafe memory access. - This is safer than deploying custom kernel modules, though eBPF can still cause harm or performance problems. - **Performance** - eBPF generally has lower overhead than approaches such as Linux Audit or ptrace. - Actual impact depends heavily on implementation and workload. - **Unified visibility** - A single mechanism can observe process, filesystem, and network activity. - This avoids assembling multiple specialized tracing systems. - **Container and namespace coverage** - eBPF provides consistent visibility across namespaces, cgroups, and containers. - CO-RE (Compile Once–Run Everywhere) improves portability across Linux distributions and kernel versions. - **Enforcement capabilities** - BPF LSM programs support mandatory access controls. - This gives eBPF enforcement power beyond ordinary tracing mechanisms, which is important for runtime security. ## Lessons from Operating eBPF at Scale After five years of operating an agent that hooks process scheduling, filesystem, and networking internals, Datadog emphasizes that production eBPF is more complicated than its reputation suggests. The six areas of operational experience are: - Ensuring programs load, attach, and continue firing across kernel versions. - Capturing and enriching event data accurately. - Monitoring and auditing eBPF usage to reduce the attack surface. - Coexisting with other eBPF-based tools on the same host. - Measuring and controlling performance overhead. - Shipping changes safely through disciplined rollout practices. The practical recommendation is to treat eBPF as powerful infrastructure rather than a maintenance-free kernel feature: validate behavior across kernels and workloads, monitor its own operation, measure overhead continuously, and use cautious deployment practices.

networkingdatabase-designobservabilityebpf+4

datadogJun 6, 2022

Engineering Spotlight: Tay Nishimura (opens in new tab)

Tay Nishimura’s career shows that succeeding in tech is often less about fitting a standard engineering mold and more about finding work that matches one’s strengths. Although she initially struggled with the speed and coding demands of software development, her rigor, visual thinking, and careful approach became valuable in site reliability engineering. Her transition was enabled by self-directed learning, community education, and ToyNet, an open source networking simulator that demonstrated her practical abilities. ## Entering Tech from Mathematics - Tay began as a mathematics major focused on real analysis, then added computer science after advice from a professor. - Internships at Amazon and Google introduced her to the technology industry. - She found a sharp contrast between academia and industry: - School rewarded theoretical rigor. - Industry emphasized practical, fast, and agile solutions. - Tay also felt like an outsider because she had little exposure to computers growing up. ## Struggling with Traditional Software Engineering - Coding did not come naturally to Tay’s visual way of thinking. - She translated code into drawings to understand and modify it, then converted those ideas back into code. - This process produced high-quality, careful work but made her slower than colleagues expected. - A manager suggested product management and site reliability engineering as possible alternatives. - Tay discovered that her deliberate pace was useful for SRE work, particularly when evaluating failure modes and making critical changes. - Because her company offered no path into those roles, she eventually left rather than continue facing increasing stress. ## Discovering Networking and Technical Program Work - Tay’s next role had a software engineer title but involved work closer to product or technical program management. - She learned that job titles and responsibilities vary significantly between companies. - With better work-life balance, she began studying computer networking in her free time. - She created visual diagrams and learning modules to explain switches, routers, and packet flows. - These efforts became Project Reclass, a nonprofit teaching technical skills to incarcerated people and military veterans. - The program used improvised equipment, such as fake routers and switches, to teach concepts in environments where real networking hardware was unavailable. ## Building ToyNet During the Pandemic - After her company laid off its entire office during COVID-19, Tay decided to pursue SRE directly. - When prisons suspended in-person education, Project Reclass adapted by creating a digital networking simulator. - Tay architected ToyNet, an open source platform built with: - React - A Flask backend - Containerized Mininet instances for network emulation - Users can connect simulated routers, switches, and hosts, configure IP addresses, and run commands such as `ping` and `arp`. - ToyNet was designed to work for incarcerated learners with restricted internet access. - Deploying it in the cloud also gave Tay practical experience that helped compensate for limited professional cloud experience. - Companies interested in the project were more likely to advance her through the interview process, eventually leading to Datadog. ## Finding the Right Environment at Datadog - At Datadog, Tay learned Kubernetes, chaos engineering, network traffic control, and Go. - She found that her rigor and visual thinking were assets rather than liabilities. - While learning Datadog’s Chaos Controller codebase, she mapped files and dependencies by drawing boxes and arrows. - Her experience suggests that engineers do not need to learn or reason in a single conventional way; the right environment can turn an apparent weakness into a strength. Tay’s path recommends experimenting broadly, studying independently, and building concrete projects that reveal how you think and solve problems. The most suitable tech role may emerge only after moving between companies and disciplines rather than forcing yourself to succeed in an ill-fitting position.

networkingk8sreactopen-source+4

datadogApr 13, 2022

It's always DNS . . . except when it's not: A deep dive through gRPC, Kubernetes, and AWS networking (opens in new tab)

A routine update to a critical metrics query service caused intermittent errors and increased latency. Although logs initially pointed to DNS failures, the investigation revealed a deeper networking problem involving dropped packets and saturated AWS VPC connection tracking. The incident highlighted how Kubernetes, Cilium, AWS networking, and DNS behavior can interact in ways that obscure the true cause. ## Initial Symptoms and Apparent DNS Failures - Errors increased whenever the metrics query service was rolled out. - The service retrieves data from metric stores for dashboards and monitor evaluations. - Automatic retries reduced user-facing failures but increased latency. - Service logs showed DNS errors when connecting to dependencies inside Kubernetes. - The investigation therefore began with the cluster’s DNS infrastructure. ## NodeLocal DNSCache Reaches Its Limits - NodeLocal DNSCache runs as a `node-local-dns` DaemonSet on every Kubernetes node. - DNS pods had: - A 64 MB memory limit - A `max_concurrent` limit of 1,000 requests - The pods experienced out-of-memory errors and rejected requests during rollouts. - Increasing memory to 256 MB stopped the OOM errors, but DNS failures continued. - Request volume was far below the expected capacity: - Normally about 400 queries per second - Nearly 2,000 queries per second during rollouts - Expected capacity of at least 200,000 queries per second - Upstream resolvers were marked unhealthy, suggesting that NodeLocal DNSCache could not establish or maintain connections. - Because upstream requests could wait up to five seconds, connection failures consumed concurrency slots and made the cache appear overloaded. ## Evidence of a Network Problem - The instances were below their 5-Gbps sustained throughput limits. - TCP retransmits increased in correlation with service rollouts. - Engineers suspected brief traffic spikes, or microbursts, that were not visible in aggregate throughput metrics. - This shifted the investigation from DNS configuration toward lower-level AWS networking behavior. ## AWS VPC Connection Tracking - ENA metrics revealed a significant increase in `conntrack_allowance_exceeded`. - This metric counts packets dropped when VPC connection tracking becomes saturated. - Connection tracking maintains state for network flows and supports features such as stateful EC2 security groups. - The infrastructure used two tracking layers: - VPC conntrack maintained at the hypervisor level - Linux conntrack inside each instance - VPC conntrack appeared saturated even though Linux conntrack contained fewer than 60,000 entries—well within the observed capacity of similar instances. - AWS Support confirmed that conntrack capacity varies by instance type and that VPC conntrack limits could differ substantially from Linux conntrack limits. - Scaling to larger instances resolved the symptoms, but the engineers wanted to understand the traffic pattern and find a more efficient long-term solution. ## VPC Flow Logs as the Next Investigation Tool - The team turned to Amazon VPC Flow Logs to examine the service’s low-level network behavior. - These logs were expected to clarify why connection tracking filled up and how rollout traffic contributed to the saturation. - The investigation was still ongoing at the point where the provided article excerpt ends.

networkingdatabase-designk8spostgresql+4