cloudflare·Aug 10, 2026·3 min readCurated summary

Everything we launched during Agents Week

Cloudflare’s Agents Week presented agents as a new class of software requiring more than advanced models: they need runtimes, identity, orchestration, memory, observability, and security. Across five days, Cloudflare introduced infrastructure and tools for building, deploying, governing, and connecting agents to the web. The broader conclusion is that an “Agentic Internet” must support cooperation between humans and autonomous software while preserving control, trust, and openness. ## Agent Runtime and Infrastructure - Cloudflare introduced `@cloudflare/computer`, a runtime that lets agents select suitable computing environments rather than relying solely on containers. - Workers RPC now supports communication between Python and JavaScript Workers. - Cloudflare detailed efforts to run models such as Kimi and GLM more efficiently while maintaining quality, reliability, and safety. - The Billable Usage API provides programmatic access to Cloudflare product costs and usage. - Workers and Containers gained inbound TCP and gRPC support, enabling real-time applications such as voice AI backends. ## From Prototypes to Production with the ADLC - Cloudflare proposed the Agent Development Lifecycle (ADLC) as an evolution of the traditional Software Development Lifecycle for agent-based software. - Cloudflare Agents provide live run monitoring, tracing, replay, and human approval workflows. - Local tracing helps agents and developers debug Workers before deployment. - Cloudflare Wallets give agents programmable, secure capabilities for participating in transactions. - Programmable CI/CD pipelines can run across millions of repositories and use agents to diagnose failures and prepare fixes for review. - Cloudflare described internal AI-powered engineering workflows, including automated standards enforcement and an Astro software factory that reduced GitHub issue-management toil. ## Identity, Security, and Governance - The Agent Access Model defines how agents can access services and resources on behalf of users. - Cloudflare OS embeds AI into internal work while retaining security and human oversight, and its platform was open-sourced for building applications and automations. - Identity-aware analytics connect AI activity to users and systems, helping detect anomalous behavior and unexpected spending. - WriteGuard adds fine-grained controls to MCP servers to restrict dangerous or unwanted tool calls. ## Building an Agentic Internet - Cloudflare outlined an Internet that is readable, discoverable, callable, and payable, allowing publishers to control access while enabling agents to interact and transact. - WebMCP gives websites and web applications a simple interface that agents can discover and use. - Answer Engine Optimization (AEO) adapts SEO practices for content surfaced by AI agents. - Kitesurf is an agent-focused browser running in V8 isolates, prioritizing efficiency over pixel-perfect rendering. - MCPv2 simplifies the deployment and scaling of agentic applications. - Cloudflare AI Search turns websites and files into searchable, agent-ready data sources. ## Observing the Agent Ecosystem - Cloudflare argued that bot behavior should be evaluated through continuous trust rather than assuming bots are inherently harmful. - Workers AI and AI Gateway are being unified into a single AI control plane with one binding, wallet, and dashboard for model access. - New Cloudflare Ambassadors and Community Engineers programs support community leaders and open-source maintainers, alongside an additional $1 million in open-source funding. - Radar Researcher lets users explore Internet data through natural-language questions and interactive charts. Cloudflare’s vision is an Agent Cloud combining execution infrastructure, an increasingly automated development lifecycle, secure identity and access, agent-ready web protocols, and strong human communities. Building agents successfully will require treating them as participants in a broader computing ecosystem—not merely as model-powered features.

Read original(opens in new tab)

Distributed TracingAI AgentsCI/CDModel Context Protocol+4

netflix·Jul 14, 2026·3 min readCurated summary

Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned

The post explains how Netflix built a real-time service topology system capable of processing millions of network-flow records per second at production scale. Its core design combines streaming ingestion, reactive backpressure, physically separate data layers, and a distributed aggregation pipeline that resolves network intermediaries into meaningful service dependencies. The system favors slightly delayed but complete updates over stale batch data or incomplete results caused by dropping records. ## The Need for Real-Time Topology - Traditional topology tools rely on hourly or daily batch processing, making their data outdated during incidents. - Netflix combines: - eBPF network flows - IPC metrics delivered through Server-Sent Events - Distributed tracing data - These sources are stored in separate graph or columnar storage layers and can be queried independently or merged. - The goal is near-real-time freshness, faster incident response, blast-radius analysis, and immediate change validation. ## Backpressure for Reliable Streaming - Processing millions of flow records per second creates a risk that downstream systems will become overwhelmed. - Common alternatives are inadequate: - Unbounded queues eventually exhaust memory. - Dropping records produces incomplete topology. - Batch processing introduces unacceptable delays. - Reactive streams propagate slowdown upstream: - A graph database signals Stage 2. - Stage 2 slows Stage 1. - Stage 1 pauses Kafka consumption. - Kafka retains the data until capacity returns. - This allows the system to degrade gracefully during traffic spikes, garbage-collection pauses, or temporary storage slowdowns. - Updates may be delayed by seconds or minutes, but the data remains substantially more complete than a dropped or hourly-processed stream. ## Physically Separate Topology Layers Netflix keeps each data source in storage optimized for its characteristics: - **Network layer:** eBPF flow logs provide broad coverage but limited application context. - **IPC layer:** Application metrics offer detailed endpoint information but cover only instrumented services. - **Tracing layer:** Parquet-based distributed traces show actual request paths but are sampled. - Separate storage enables each layer to evolve and scale independently. - Queries can run in parallel and merge results while preserving sub-second response times. ## Three-Stage Distributed Aggregation The network layer uses a distributed pipeline to transform individual network hops into logical service dependencies. - Cloud traffic commonly passes through load balancers, NAT gateways, API gateways, and proxies. - Flow logs therefore show relationships such as: - `App A → Load Balancer` - `Load Balancer → App B` - The useful topology must infer the logical dependency: `App A → App B`. ### Stage 1: Initial Flow Aggregation - Consumes flow logs from Kafka across four regions. - Filters invalid records. - Groups data into five-minute windows. - Creates initial aggregators for each window. - Uses consistent hashing to distribute aggregators. - Streams the results to Stage 2 through SSE. ### Stage 2: Intermediary Resolution - Receives the initial aggregators from Stage 1. - Groups flows by intermediary components. - Resolves multi-hop network paths into application-level relationships. - This prevents infrastructure components from dominating the resulting service graph. ## Engineering Trade-offs - Streaming provides much fresher data than batch processing but introduces greater operational and conceptual complexity. - Backpressure is essential for stability at Netflix’s scale, even though reactive pipelines are harder to reason about than synchronous systems. - The architecture prioritizes reliable, complete topology updates over perfectly immediate processing. - Production behavior differed substantially from local testing: consumers lagged, memory was exhausted, traffic became unevenly distributed, and garbage collection consumed significant resources. Netflix’s approach demonstrates that large-scale real-time topology requires streaming ingestion, end-to-end backpressure, specialized storage, and staged aggregation. For similar distributed systems, the practical recommendation is to design explicitly for overload and partial slowdown rather than relying on unbounded buffering, dropped data, or stale batch snapshots.

Read original(opens in new tab)

Distributed TracingApache KafkaEbpfGraph Database+4

discord·Mar 4, 2026·2 min readCurated summary

Tracing Discord's Elixir Systems (Without Melting Everything)

Discord runs each guild independently using Elixir’s concurrency model, helping chats and reactions feel instantaneous at scale. When a guild becomes overloaded, metrics and logs can reveal activity spikes but often fail to show the actual user experience or downstream effects. To fill this gap, Discord built distributed tracing for its Elixir services and integrated it without downtime. ## Guild-Level Isolation and Outages - Each Discord server, or “guild,” runs independently from others. - This isolation supports high concurrency and limits failures to individual guilds. - A guild may become laggy or go offline when user activity exceeds its processing capacity. - If it cannot recover automatically, on-call engineers investigate the incident. ## Limits of Metrics and Logs - Engineers inspect metrics showing: - How often each user action type is processed. - How long processing takes. - These metrics can identify bursts of activity, such as sudden waves of reactions or messages. - However, they do not clearly show how those conditions affected users. - Metrics are comparable to a car dashboard: they expose internal conditions but not necessarily the consequences. ## Guild Timings - Discord’s custom “guild timings” tool records the amount of each minute spent processing different action types. - The data is stored in memory and provides more detail than standard metrics. - Its high volume makes long-term storage impractical, so data is frequently rotated. - The tool also focuses on guild-local processing and does not capture downstream effects or complete end-to-end request experience. ## Building Distributed Tracing for Elixir - Distributed tracing shows how long each part of an operation takes across services. - Other Discord teams had already benefited from tracing and application performance monitoring. - Typical tracing systems propagate operation context through metadata such as HTTP headers. - Elixir’s built-in communication mechanisms do not provide an equivalent metadata layer. - Discord therefore built its own mechanism for propagating tracing information between services. ## Deployment Without Downtime - Although the tracing system changed how Discord services communicate, it was integrated without taking the platform offline. - The result gives engineers a more complete view of request paths, helping them understand both the source of guild problems and their impact on users. Discord’s experience suggests that detailed distributed tracing is essential when local metrics and logs cannot explain end-to-end behavior. For highly concurrent systems, investing in tracing infrastructure tailored to the platform can significantly improve incident diagnosis without requiring disruptive deployment changes.

Read original(opens in new tab)

Distributed TracingDatabase DesignApplication Performance MonitoringElixir+3

toss·Nov 13, 2025Original article

Frontend Code That Lasts 1 (opens in new tab)

Toss Payments evolved its Payment SDK to solve the inherent complexities of integrating payment systems, where developers must navigate UI implementation, security flows, and exception handling. By transitioning from V1 to V2, the team moved beyond simply providing a library to building a robust, architecture-driven system that ensures stability and scalability across diverse merchant environments. The core conclusion is that a successful SDK must be treated as a critical infrastructure layer, relying on modular design and deep observability to handle the unpredictable nature of third-party runtimes. ## The Unique Challenges of SDK Development * SDK code lives within the merchant's runtime environment, meaning it shares the same lifecycle and performance constraints as the merchant’s own code. * Internal logging can inadvertently create bottlenecks; for instance, adding network logs to a frequently called method can lead to "self-DDoS" scenarios that crash the merchant's payment page. * Type safety is a major hurdle, as merchants may pass unexpected data types (e.g., a number instead of a string), causing fatal runtime errors like `startsWith is not a function`. * The SDK acts as a bridge for technical communication, requiring it to function as both an API consumer for internal systems and an API provider for external developers. ## Ensuring Stability through Observability * To manage the unpredictable ways merchants use the SDK, Toss implemented over 300 unit tests and 500 E2E integration tests based on real-world use cases. * The team utilizes a "Global Trace ID" to track a single payment journey across both the frontend and backend, allowing for seamless debugging across the entire system. * A custom Monitoring CLI was developed to compare payment success rates before and after deployments, categorized by merchant and runtime environment (e.g., PC Chrome vs. Android WebView). * This observability infrastructure enables the team to quickly identify edge-case failures—such as a specific merchant's checkout failing only on mobile WebViews—which are often missed by standard QA processes. ## Scaling with Modular Architecture * To avoid "if-statement hell" caused by merchant-specific requirements (e.g., fixing installment months or custom validation for a specific store), Toss moved to a "Lego-block" architecture. * The SDK is organized into three distinct layers based on the "reason for change" principle: * **Public Interface Layer:** Manages the contract with the merchant, validating inputs and translating them into internal domain models. * **Domain Layer:** Encapsulates core business logic and payment policies, keeping them isolated from external changes. * **External Service Layer:** Handles dependencies like Server APIs and Web APIs, ensuring technical shifts don't leak into the business logic. * This separation allows the team to implement custom merchant logic by swapping specific blocks without modifying the core codebase, reducing the risk of regressions and lowering maintenance costs. For developers building SDKs or integration tools, the shift from monolithic logic to a layered, observable architecture is essential. Prioritizing the separation of domain logic from public interfaces and investing in environment-specific monitoring allows for a highly flexible product that remains stable even as the client-side environment grows increasingly complex.

Distributed TracingObservabilityTypeScriptSoftware Architecture+3