cloudflare·Aug 10, 2026·3 min readCurated summary

Everything we launched during Agents Week

Cloudflare’s Agents Week presented agents as a new class of software requiring more than advanced models: they need runtimes, identity, orchestration, memory, observability, and security. Across five days, Cloudflare introduced infrastructure and tools for building, deploying, governing, and connecting agents to the web. The broader conclusion is that an “Agentic Internet” must support cooperation between humans and autonomous software while preserving control, trust, and openness. ## Agent Runtime and Infrastructure - Cloudflare introduced `@cloudflare/computer`, a runtime that lets agents select suitable computing environments rather than relying solely on containers. - Workers RPC now supports communication between Python and JavaScript Workers. - Cloudflare detailed efforts to run models such as Kimi and GLM more efficiently while maintaining quality, reliability, and safety. - The Billable Usage API provides programmatic access to Cloudflare product costs and usage. - Workers and Containers gained inbound TCP and gRPC support, enabling real-time applications such as voice AI backends. ## From Prototypes to Production with the ADLC - Cloudflare proposed the Agent Development Lifecycle (ADLC) as an evolution of the traditional Software Development Lifecycle for agent-based software. - Cloudflare Agents provide live run monitoring, tracing, replay, and human approval workflows. - Local tracing helps agents and developers debug Workers before deployment. - Cloudflare Wallets give agents programmable, secure capabilities for participating in transactions. - Programmable CI/CD pipelines can run across millions of repositories and use agents to diagnose failures and prepare fixes for review. - Cloudflare described internal AI-powered engineering workflows, including automated standards enforcement and an Astro software factory that reduced GitHub issue-management toil. ## Identity, Security, and Governance - The Agent Access Model defines how agents can access services and resources on behalf of users. - Cloudflare OS embeds AI into internal work while retaining security and human oversight, and its platform was open-sourced for building applications and automations. - Identity-aware analytics connect AI activity to users and systems, helping detect anomalous behavior and unexpected spending. - WriteGuard adds fine-grained controls to MCP servers to restrict dangerous or unwanted tool calls. ## Building an Agentic Internet - Cloudflare outlined an Internet that is readable, discoverable, callable, and payable, allowing publishers to control access while enabling agents to interact and transact. - WebMCP gives websites and web applications a simple interface that agents can discover and use. - Answer Engine Optimization (AEO) adapts SEO practices for content surfaced by AI agents. - Kitesurf is an agent-focused browser running in V8 isolates, prioritizing efficiency over pixel-perfect rendering. - MCPv2 simplifies the deployment and scaling of agentic applications. - Cloudflare AI Search turns websites and files into searchable, agent-ready data sources. ## Observing the Agent Ecosystem - Cloudflare argued that bot behavior should be evaluated through continuous trust rather than assuming bots are inherently harmful. - Workers AI and AI Gateway are being unified into a single AI control plane with one binding, wallet, and dashboard for model access. - New Cloudflare Ambassadors and Community Engineers programs support community leaders and open-source maintainers, alongside an additional $1 million in open-source funding. - Radar Researcher lets users explore Internet data through natural-language questions and interactive charts. Cloudflare’s vision is an Agent Cloud combining execution infrastructure, an increasingly automated development lifecycle, secure identity and access, agent-ready web protocols, and strong human communities. Building agents successfully will require treating them as participants in a broader computing ecosystem—not merely as model-powered features.

Read original(opens in new tab)

GrpcAI AgentsCI/CDModel Context Protocol+4

cloudflare·Aug 3, 2026·2 min readCurated summary

Cloudflare Workers and Containers now support inbound TCP connections and gRPC

Cloudflare is expanding Workers to support low-latency, TCP-based applications such as real-time voice AI and gRPC services. New inbound socket handling lets Workers route connections through Durable Objects and Containers, while Cloudflare also enables full-duplex gRPC servers running in containers. Together, these capabilities allow developers to deploy language-agnostic TCP and gRPC services closer to users across Cloudflare’s global network. ## Inbound TCP with `connect(socket)` - Workers can now accept inbound TCP sockets through a new `connect()` handler. - The socket exposes readable and writable streams, allowing Workers to send, receive, and proxy raw bytes. - Connections can be routed: - Between Workers - From Workers to Durable Objects - From Durable Objects to Cloudflare Containers - Developers can pipe data in both directions to preserve full-duplex communication. - Containers can run arbitrary TCP servers written in any language, such as Python services listening on port `8080`. - Cloudflare Spectrum provides the TCP ingress layer and routes incoming connections to the selected Worker. ## Full-Duplex gRPC in Containers - Developers can deploy gRPC servers written in languages such as Go inside Cloudflare Containers. - Bidirectional streaming allows clients and servers to exchange messages over one persistent connection. - This is particularly useful for: - Real-time voice AI - Low-latency inference - Mobile and distributed applications - Streaming RPC workflows - A sample Go server sends an initial connection message, echoes incoming messages, and sends a closing message when the client disconnects. - Cloudflare’s network of more than 330 locations can bring gRPC workloads closer to users, reducing latency. ## gRPC APIs from Workers - Workers can serve unary and server-streaming gRPC APIs. - Workers can also call external gRPC servers. - Developers write the application using gRPC-Web, while Cloudflare automatically converts incoming and outgoing requests to standard gRPC. - This provides a simpler integration path for applications that need gRPC without managing raw protocol translation themselves. ## Availability - The features are being introduced through a private beta. - Interested developers must sign up to gain access. Cloudflare recommends these capabilities for applications requiring persistent, low-latency, bidirectional communication. The combination of Spectrum, Workers, Durable Objects, and Containers provides a flexible path for running raw TCP protocols and gRPC services close to end users.

Read original(opens in new tab)

GrpcCloudflare WorkersDurable ObjectsTcp+3

netflix·Jul 17, 2026·4 min readCurated summary

In-House LLM Serving at Netflix

Netflix built an in-house LLM serving platform within its existing production ML infrastructure rather than creating a separate ML stack. The platform combines a JVM-based serving layer, NVIDIA Triton, GPU-backed Model Scoring Service, and an OpenAI-compatible HTTP frontend. Its main design choices—vLLM, model packaging, API compatibility, and deployment strategy—prioritize operational flexibility and seamless movement from hosted models to self-hosted ones, while production exposed versioning and compatibility risks. ## Architecture and Serving Model - Netflix’s unified JVM serving system handles routing, A/B testing, feature retrieval, inference, post-processing, and logging. - Callers access models through: - A gRPC path integrated with the existing serving system. - A direct HTTP path for newer LLM applications. - Small CPU models run in-process to avoid remote-call overhead. - Larger GPU models run through Model Scoring Service (MSS), which supports XGBoost, TensorFlow, PyTorch, and LLMs. - NVIDIA Triton manages model loading, batching, and GPU scheduling. - A Java control plane provides deployment, versioning, health checks, autoscaling, and multi-region rollout. ## Choosing vLLM as the Standard Engine - Netflix originally used TensorRT-LLM, but re-evaluated its choice as open-source engines improved and workloads diversified. - vLLM was selected based on operational fit rather than benchmark performance alone: - Supports custom model architectures without lengthy compilation. - Provides hooks for custom decoding and constraint logic. - Is easier to debug than earlier compiled-engine workflows. - Is familiar to many researchers, reducing the research-to-production transition cost. - The workload includes embeddings, prefill-only inference, autoregressive decoding, and custom per-step decoding constraints. ## Triton Integration and Model Packaging - Triton offers both a Python backend and a dedicated vLLM backend. - The Python backend requires explicit input and output tensor definitions, coupling packaged artifacts to frontend changes. - The vLLM backend uses a JSON configuration pointing to model weights and tokenizers, generating tensor specifications dynamically. - Netflix considers the vLLM backend the preferred default because models and frontends can evolve independently. - Production revealed two limitations: - Triton and vLLM must be version-pinned because incompatible APIs can prevent the backend from loading entirely. - Models requiring custom preprocessing, postprocessing, tokenization, or ensemble execution still need Triton’s Python backend. ## OpenAI-Compatible HTTP Frontend - Netflix keeps LLMs compatible with the same internal gRPC model-serving interface used by other model types. - It also exposes an OpenAI-compatible API because that interface is widely supported by inference engines, orchestration tools, evaluation systems, and client libraries. - This makes replacing a hosted model with a fine-tuned self-hosted model largely transparent to callers. - The implementation uses Triton’s OpenAI-compatible frontend, FastAPI, and a `TritonLLMEngine` that translates requests into Triton inference calls. - KServe HTTP and gRPC frontends remain available for the Java control plane. - Netflix found that Triton’s frontend silently discarded the `response_format` parameter, meaning JSON requests could reach vLLM without guided decoding and produce malformed output. - The team patched the frontend to translate `response_format` into vLLM guided-decoding parameters. ## Deployment and Rollout Strategies - GPU services require longer startup times than CPU services, and model versions may change input/output schemas. - Netflix supports Red-Black deployment: - Runs the new version alongside the old one. - Performs health checks before shifting traffic. - Gradually scales up the new version while scaling down the old one. - Supports atomic rollback if deployment fails. - Red-Black deployment works well when the model interface remains stable. - Production exposed a schema-coordination problem: if a new model changes tensor dimensions or other I/O requirements, upstream callers may send old requests to the new model during the migration window, causing failures. - The post introduces a Versioned strategy as a solution, but the provided text ends before explaining its implementation. Netflix’s experience suggests that successful in-house LLM serving depends as much on compatibility and deployment mechanics as on raw inference speed. A practical platform should standardize on an extensible engine such as vLLM, preserve ecosystem-compatible APIs, tightly control engine versions, retain escape hatches for custom models, and explicitly coordinate model-schema changes during rollout.

Read original(opens in new tab)

GrpcVllmGpu InferenceFastapi+4

datadog·Jun 23, 2026·3 min readCurated summary

How we migrated a live routing system using AI-assisted refactoring

Stream Router evolved from a small configuration file into a critical control-plane service routing Datadog’s massive metrics workload. Its original FoundationDB key-value model eventually hit transaction-size and performance limits because relational relationships were reconstructed in application code. Datadog redesigned the system around PostgreSQL and DuckDB, using AI-assisted, test-driven refactoring to accelerate the migration without disrupting production traffic. ## Stream Router’s Role in Datadog’s Metrics Pipeline - Datadog processes more than a hundred trillion events per day. - Stream Router determines which Kafka cluster, topic, partitions, and sharding strategy should handle each datapoint. - It serves both producers and queriers but does not process Kafka messages itself. - Routing decisions change frequently as infrastructure evolves, making correctness and historical tracking essential. ## From Configuration File to Control Plane - In 2016, routing was managed through a small configuration file distributed to services. - As the platform grew, the file expanded to thousands of lines and required manual edits and rollouts. - Stream Router replaced this workflow with: - A centralized gRPC service - API-managed routes - Automated, gradual rollouts - The write path used FoundationDB, while the read path served static RocksDB snapshots restored into memory. - This eventually became a bottleneck as routing tables and operational changes grew larger. ## Why the Key-Value Model Stopped Scaling - Routes reference streams and sharding strategies, while rules reference routes. - These relationships are inherently relational and require cross-entity validation. - The KV implementation loaded tens of thousands of records into application processes and reconstructed database-like relationships in code. - Some operations exceeded FoundationDB transaction-size limits. - Moving to PostgreSQL without changing the access patterns would not solve the issue; certain operations were estimated to require 45 minutes because of thousands of sequential database round trips. - The fundamental problem was the data model and application logic, not simply the choice of database. ## Designing the New Storage Architecture - The team redesigned the schema manually before using AI tools. - The relational model introduced explicit foreign keys between: - Streams - Sharding strategies - Routes - Rules - PostgreSQL was selected for the write path because it provided the required relational semantics and transaction model. - DuckDB was selected for the read path because: - It is embeddable and suitable for snapshot-based serving - It supports array columns - Its SQL dialect is closely compatible with PostgreSQL - Shared query logic could therefore work across both storage engines. ## AI-Assisted Refactoring - Claude and Cursor were used to accelerate a systematic, test-driven migration. - For each method, developers supplied: - The old implementation - The new schema - A failing test - AI generated an initial implementation, while tests determined whether it was correct. - The models assisted with method-level refactoring rather than autonomously designing the architecture. - Human expertise remained central to schema design, migration strategy, and evaluating system-level risks. ## Foundations for a Safe Migration - The migration benefited from infrastructure already present at Datadog. - Stream Router’s storage layer was isolated behind an internal `Controller` interface. - This modularity helped contain storage changes and enabled incremental refactoring. - Existing tests and clear boundaries provided confidence in generated implementations while production traffic continued. The central lesson is that AI was most effective as an accelerator inside a disciplined engineering process. A well-designed relational schema, modular storage abstraction, and failing tests provided the safety mechanisms; AI helped implement the resulting changes faster, but did not replace human architectural judgment.

Read original(opens in new tab)

GrpcDatabase DesignPostgreSQLApache Kafka+4

toss·Jun 19, 2026·5 min readCurated summary

Spark Connect on Kubernetes #1: Building a Robust Spark Connect

Toss Securities operates Spark Connect as a production service on Kubernetes so analysts and engineers can use Spark without complex setup. Spark Connect replaces per-application Drivers with long-running servers, making clients lighter and sessions faster, but it also introduces shared-failure and resource-contention problems. The post argues that production reliability requires both reducing server-wide failure triggers and distributing sessions across multiple replicas. ## How Classic Spark Works - Spark consists of: - A **Driver**, which plans jobs, schedules tasks, and collects results. - **Executors**, which perform the distributed computations. - In Classic Spark: - **Client mode** runs the Driver inside the client process. - **Cluster mode** launches the Driver in the cluster for each submitted application. - Both modes assume that one application has one Driver and one workload. - Clients also need Spark libraries, JVM support, and configuration. ## What Spark Connect Changes - Spark Connect turns the Driver into a pre-started, long-running server. - Clients send unresolved logical plans encoded with Protocol Buffers over gRPC. - The server handles analysis, optimization, scheduling, and execution. - Results are streamed back using Arrow. - This resembles a database accessed through JDBC. ### Benefits - **Thin clients:** Clients do not need the full Spark runtime or JVM. - **Language and platform independence:** Notebooks, BI tools, SQL clients, and different programming languages can use the same server. - **Fast session creation:** Sessions connect to an already-running server. - **Better client-failure tolerance:** A disconnected notebook does not necessarily terminate server-side work. ## Problems Created by Shared Long-Running Servers Spark’s internal design often assumes “one application equals one workload.” Sharing one application across many users breaks that assumption. ### A Shared Driver Becomes a Single Point of Failure - Multiple sessions share one `SparkContext` and Driver JVM. - A Driver failure terminates all sessions, jobs, and caches attached to it. - Spark’s global `spark.executor.maxNumFailures` counter can shut down the entire application after enough executor failures. - Because all sessions contribute to the same counter, one user’s unstable or memory-intensive query can terminate unrelated users’ workloads. - The counter is global, persists over time, and is separate from per-query task-level fault tolerance such as `spark.task.maxFailures`. ### Resource Contention and Scheduling Limits - `newSession()` isolates SQL state and namespaces, but not CPU, memory, or executors. - Heavy workloads can occupy all task slots and delay smaller queries. - FIFO scheduling favors earlier jobs, and Spark does not preempt tasks already using slots. - Fair Scheduler pools can influence task-slot ordering, but cannot provide true CPU or memory isolation. - Spark Connect does not automatically propagate `spark.scheduler.pool` to the server-side execution thread, causing queries to fall into the default pool unless the server explicitly assigns pools. - Actual resource isolation must therefore be implemented outside Spark’s task scheduler. ### Fixed Server Capacity - A server’s image, Driver and Executor resources, and Spark configuration are fixed when it starts. - Dynamic Resource Allocation can adjust executor counts, but cannot change the server’s basic specification. - Flexible scaling and team-level isolation require creating or replacing servers, which is addressed in a later part of the series. ## Reducing Server-Wide Failures Before adding replicas, Toss Securities reduces the chance that one bad query can kill the shared server. - Set `spark.executor.maxNumFailures` effectively high enough to disable the global shutdown mechanism. - Use `spark.executor.failuresValidityInterval` to periodically clear accumulated failure records. - Rely on query-scoped controls: - `spark.task.maxFailures` stops tasks that repeatedly fail due to OOMs or exceptions. - `spark.stage.maxConsecutiveAttempts` stops jobs whose stages repeatedly fail, such as from shuffle-fetch errors. - These limits must be tuned carefully: overly aggressive values can cause healthy queries to fail during temporary infrastructure problems. - With this approach, executor failures terminate the problematic query rather than the entire Spark Connect server. ## Protecting Driver Memory from Large Results - Spark Connect streams query results through the Driver, so a large `collect()` can threaten Driver memory. - `spark.driver.maxResultSize` aborts an action when accumulated task results exceed the configured limit. - The limit is checked before large executor-side results are fetched into Driver memory. - The default 1 GB value assumes a single workload; in a multi-session server, it should be reduced or tuned based on the number of concurrent queries. ## Replicating Spark Connect Servers - Configuration alone cannot prevent Driver OOMs, node failures, or other catastrophic events. - The stronger isolation boundary is a separate SparkContext. - Multiple identical Spark Connect replicas are deployed: - Each replica has its own Driver, SparkContext, and Executors. - A failure affects only the sessions assigned to that replica. - Other replicas can continue accepting sessions. - Replica-based deployment reduces the blast radius from the entire Spark Connect service to an individual server instance. ## Practical Recommendation For a multi-user Spark Connect service, disable global executor-failure shutdown, enforce query-level failure limits, protect Driver memory with `spark.driver.maxResultSize`, and use multiple replicas to contain unavoidable Driver or node failures. Scheduler pools can improve ordering, but they should not be treated as true resource isolation.

Read original(opens in new tab)

GrpcKubernetesApache SparkFault Tolerance+4

netflix·May 1, 2026·3 min readCurated summary

State of Routing in Model Serving

Netflix’s centralized ML serving platform provides a single, domain-independent API for model inference across personalized experiences and other use cases. Rather than exposing individual scoring functions, Netflix packages feature computation, preprocessing, inference, and postprocessing into self-contained model workflows. The core routing challenge is directing each request to the correct model version and serving cluster while keeping client services independent from model changes and infrastructure topology. ## Models as End-to-End Workflows - Netflix distinguishes **model serving** from traditional model inference: - Inference typically means `infer(features) -> score`. - Serving includes preprocessing, feature computation, optional trained components, and postprocessing. - Example workflows include: - Ranking titles for a personalized Continue Watching row using user, country, and device context. - Predicting payment fraud using user, country, and transaction details. - Models declare the facts they need, while the serving platform retrieves those facts from other microservices. - During offline training, Netflix’s ML fact store provides snapshots for bulk feature computation. - Calling services provide standard request context and domain-specific inputs, while the platform handles feature generation, model selection, and execution. ## Platform Design Principles - **Model innovation without client changes** - Client applications integrate with the platform once. - Model versions, A/B tests, additional experimental data, logging, and model selection remain hidden behind the platform API. - **Clients decoupled from model sharding** - Models run across multiple serving cluster shards, each with its own Virtual IP address. - Shard assignments can change based on traffic, SLAs, model architecture, and resource availability. - Clients should not need to track these VIP changes. - **Flexible traffic routing** - Routing must support A/B allocations, gradual traffic shifts, new model versions, new VIPs, and client-specific overrides. - Safe lifecycle management requires support for shadow deployments, canaries, rollbacks, and migrations. ## Switchboard: Context-Aware Routing - Generic API gateways and service-mesh proxies did not satisfy Netflix’s requirements. - Netflix needed: - Native integration with its experimentation platform. - gRPC support. - Routing based on rich, domain-specific request context. - Model-specific rollout and migration controls. - Netflix built **Switchboard**, a custom proxy layer handling more than one million requests per second. - Switchboard is the mandatory entry point for clients and: - Routes requests to the appropriate model based on request context. - Applies configured context enrichment before invoking the model. - Hides model locations and infrastructure changes from client services. ## Objective Abstraction - Every request must provide an **Objective**, an enumeration defined by the serving platform. - The excerpt introduces Objectives as a central abstraction for identifying the business purpose of a serving request, but the supplied text ends before describing its full roles. Netflix’s approach is to centralize routing, experimentation, and model execution behind one stable API. This allows client applications to evolve independently while researchers can iterate on models and safely manage large-scale production rollouts.

Read original(opens in new tab)

GrpcDatabase DesignMachine LearningPostgreSQL+4

discord·Feb 18, 2026·3 min readCurated summary

Osprey: Open Sourcing our Rule Engine

Discord is open-sourcing Osprey, a rule engine designed to help platforms detect and respond to emerging safety threats in real time. Built with ROOST and internet.dev, it processes platform events, evaluates configurable rules, and produces actionable verdicts with minimal engineering effort. Osprey emphasizes scale, rapid rule deployment, transparency, extensibility, and continuous improvement. ## Goals for a Modern Rule Engine Osprey was designed around several requirements: - Process thousands of events per second in real time. - Let teams create and deploy expressive rules within minutes. - Return clear verdicts indicating whether activity is safe, suspicious, or malicious. - Explain how rules were executed and expose errors for investigation and debugging. - Support feedback loops that improve future detection rules. - Remain extensible enough to address new attack patterns. ## Osprey’s Processing Model Osprey accepts platform events called **Actions** through either: - Synchronous gRPC requests. - Asynchronous message queues. The engine evaluates these actions using rules written in SML, a Python-based rule language. Rules can use Python UDFs, Features, and Effects, while synchronous requests can return Verdict effects directly to callers. Outputs are sent to Apache Druid, which powers investigation and analysis tools. ## Actions Actions are JSON-like events submitted to Osprey. - Each action type has a unique name and schema. - Callers can customize the payload with relevant platform data. - Example data includes login attempts, user IDs, usernames, email addresses, and IP addresses. - Rules extract and evaluate values from these action payloads. ## Rules and SML Rules are the central mechanism for detecting suspicious behavior. - SML uses a Python-inspired syntax intended to be accessible to less-technical rule authors. - Rules can reference other rules and extracted data. - Static validation enforces consistent rule-writing practices. - Validation can be extended with Python, from naming conventions to more complex domain-specific checks. - Example rules identify a known spammer by email and apply a `spammer` label to the associated user entity. ## User-Defined Functions UDFs are regular Python functions that extend Osprey’s rule language and standard library. - Built-in capabilities such as `Rule`, `WhenRules`, and `JsonData` are implemented as UDFs. - Teams can add their own UDFs when integrating Osprey into other products. - UDFs can retrieve information from external services, including machine-learning models. - They can be configured for asynchronous execution and access external-service providers through the execution context. - A sample UDF obtains a link-spam score from an external prediction service. ## Features and Entities Features are globally named variables produced during Osprey executions. - Features are exported to Apache Druid for later querying and investigation. - Prefixing a variable name with `_` keeps it local instead of exporting it. - Examples include `UserId` and `UserEmail`, extracted from JSON action data. - Entities are a specialized type of Feature representing persistent objects such as users, servers, or email addresses. - Entities can receive effects such as labels, classifications, and signals. - Entity types determine which effects are valid through static validation. - The Osprey interface provides dedicated Entity Views for examining an entity’s history. ## Effects Effects are outcomes triggered when rules evaluate as true. - They are validated and processed in aggregate after execution. - Effects can modify or annotate entities with labels, classifications, or signals. - Verdict effects can be returned synchronously to inform the requesting service of a safety determination. Osprey’s open-source release gives platforms a reusable foundation for real-time trust and safety enforcement. Teams interested in adopting it can explore the repository at [github.com/roostorg/osprey](https://github.com/roostorg/osprey).

Read original(opens in new tab)

GrpcPythonApache DruidTrust And Safety+3

datadog·Aug 4, 2025·4 min readCurated summary

Evolving our real-time timeseries storage again: Built in Rust for performance at scale

Datadog built a sixth-generation real-time timeseries database in Rust to keep pace with rapidly growing metric volume, cardinality, and query complexity. The new engine is designed for high throughput and low latency, reportedly achieving 60× higher ingestion performance and 5× faster peak-scale queries. Its development reflects a long evolution from general-purpose databases toward a purpose-built system with tighter control over storage, I/O, and execution. ## Datadog’s Metrics Storage Architecture - The metrics platform includes ingestion, enrichment, real-time and long-term storage, querying, and alerting. - This post focuses on real-time storage, which is split into two independently deployed services: - **RTDB:** Stores raw metric tuples of `<timeseries_id, timestamp, value>`, performs aggregations, and serves recent data. - **Index database:** Stores metric identifiers and their tags as `<timeseries_id, tags>`. - A storage router distributes incoming metrics across RTDB nodes based on load. - The query service contacts the relevant RTDB and index nodes, retrieves results, and combines them. - Each RTDB node includes: - An ingestion subsystem - A storage engine - A durability snapshot module - A gRPC query layer - Throttlers for resource management - A shared control plane coordinating these components ## Generation 1: Cassandra - Cassandra provided strong write scalability and a familiar operational model. - It was influenced by systems such as OpenTSDB and HBase. - Its main weaknesses were: - Limited flexibility for real-time queries - Difficulty supporting complex alerting and analytical workloads - Inefficient retrieval of large datasets - These limitations prompted Datadog to move to Redis. ## Generation 2: Redis - Redis improved read performance and offered a flexible, easy-to-understand storage model. - Datadog avoided Redis’s built-in clustering for reliability reasons, requiring the team to operate many independent instances. - Important drawbacks included: - Single-threaded execution limiting snapshotting during live traffic - Severe but uncommon memory-management and threading failures - Serialization and cross-process communication overhead - Inefficient memory layout, disk I/O, and CPU usage at scale - Redis nevertheless provided valuable operational insight and clarified the need for a purpose-built engine with direct control over I/O and system resources. ## Generation 3: MDBM and Memory-Mapped I/O - MDBM provided a memory-mapped key-value store based on `mmap`. - The operating system’s page cache loaded database pages on demand, making disk-backed data behave similarly to in-memory structures. - This simplified storage interactions initially, but performance degraded as workloads intensified. - Memory-mapped I/O introduced subtle performance and correctness concerns, leading Datadog to conclude that explicit I/O management would scale better. ## Generation 4: A Go-Based B+ Tree - Datadog replaced MDBM with a custom B+ tree written in Go. - The engine supported a thread-per-core-oriented design, with Go’s scheduler providing a useful foundation. - This change significantly improved throughput and latency. - It also created a platform that could be optimized more aggressively for Datadog’s workload. ## Generation 5: DDSketch and RocksDB - Datadog introduced DDSketch to support distribution metrics and accurate percentile estimation. - The existing Go engine was optimized for scalar floating-point values and was difficult to extend for sketches. - RocksDB was therefore integrated to store DDSketch data, offering flexibility and strong performance. - Over time, maintaining separate storage technologies created pressure to build a unified engine capable of handling multiple metric types efficiently. ## The Move Toward a New Engine - The progression from Cassandra to Redis, MDBM, a custom Go B+ tree, and RocksDB shows a pattern of replacing general-purpose components as scale and workload diversity increased. - Each generation solved important problems but introduced new operational or architectural trade-offs. - Datadog ultimately needed a unified, purpose-built storage system with: - High-throughput ingestion - Low-latency queries - Better support for high-cardinality data - Efficient handling of different metric types - More direct control over concurrency, memory, and I/O - The sixth generation addresses these requirements through a Rust-based real-time timeseries database. Datadog’s experience suggests that general-purpose storage systems can be effective early on, but sustained growth eventually favors a specialized engine. The practical lesson is to optimize existing infrastructure first while developing a purpose-built replacement before scale and workload complexity make incremental fixes insufficient.

Read original(opens in new tab)

GrpcDatabase DesignRustRedis+4

line·Aug 1, 2025Original article

Replacing the Payment System DB Handling (opens in new tab)

The LINE Billing Platform successfully migrated its large-scale payment database from Nbase-T to Vitess to handle high-traffic global transactions. While initially exploring gRPC for its performance reputation, the team transitioned to the MySQL protocol to ensure stability and reduce CPU overhead within their Java-based environment. This implementation demonstrates how Vitess can manage complex sharding requirements while maintaining high availability through automated recovery tools. ### Protocol Selection and Implementation - The team initially attempted to use the gRPC protocol but encountered `http2: frame too large` errors and significant CPU overhead during performance testing. - Manual mapping of query results to Java objects proved cumbersome with the Vitess gRPC client, leading to a shift toward the more mature and recommended MySQL protocol. - Using the MySQL protocol allowed the team to leverage standard database drivers while benefiting from Vitess's routing capabilities via VTGate. ### Keyspace Architecture and Data Routing - The system utilizes a dual-keyspace strategy: a "Global Keyspace" for unsharded metadata and a "Service Keyspace" for sharded transaction data. - The Global Keyspace manages sharding keys using a "sequence" table type to ensure unique, auto-incrementing identifiers across the platform. - The Service Keyspace is partitioned into $N$ shards using a hash-based Vindex, which distributes coin balances and transaction history. - VTGate automatically routes queries to the correct shard by analyzing the sharding key in the `WHERE` clause or `INSERT` statement, minimizing cross-shard overhead. ### MySQL Compatibility and Transaction Logic - Vitess maintains `REPEATABLE READ` isolation for single-shard transactions, while multi-shard transactions default to `READ COMMITTED`. - Advanced features like Two-Phase Commit (2PC) are available for handling distributed transactions across multiple shards. - Query execution plans are analyzed using `VEXPLAIN` and `VTEXPLAIN`, often managed through the VTAdmin web interface for better visibility. - Certain limitations apply, such as temporary tables only being supported in unsharded keyspaces and specific unsupported SQL cases documented in the Vitess core. ### Automated Operations and Monitoring - The team employs VTOrc (based on Orchestrator) to automatically detect and repair database failures, such as unreachable primaries or replication stops. - Monitoring is centralized via Prometheus, which scrapes metrics from VTOrc, VTGate, and VTTablet components at dedicated ports (e.g., 16000). - Real-time alerts are routed through Slack and email, using `tablet_alias` to specifically identify which MySQL node or VTTablet is experiencing issues. - A web-based recovery dashboard provides a history of automated fixes, allowing operators to track the health of the cluster over time. For organizations migrating high-traffic legacy systems to a cloud-native sharding solution, prioritizing the MySQL protocol over gRPC is recommended for better compatibility with existing application frameworks and reduced operational complexity.

GrpcDatabase DesignMysqlDatabase Migration+4

line·Jul 28, 2025Original article

Introducing a case of utilizing DDD in (opens in new tab)

LY Corporation’s ABC Studio developed a specialized retail Merchant system by leveraging Domain-Driven Design (DDD) to overcome the functional limitations of a legacy food-delivery infrastructure. The project demonstrates that the primary value of DDD lies not just in technical implementation, but in aligning organizational structures and team responsibilities with domain boundaries. By focusing on the roles and responsibilities of the system rather than just the code, the team created a scalable platform capable of supporting diverse consumer interfaces. ### Redefining the Retail Domain * The legacy system treated retail items like restaurant entries, creating friction for specialized retail services; the new system was built to be a standalone platform. * The team narrowed the domain focus to five core areas: Shop, Item, Category, Inventory, and Order. * Sales-specific logic, such as coupons and promotions, was delegated to external "Consumer Platforms," allowing the Merchant system to serve as a high-performance information provider. ### Clean Architecture and Modular Composition * The system utilizes Clean Architecture to ensure domain entities remain independent of external frameworks, which also provided a manageable learning curve for new team members. * Services are split into two distinct modules: "API" modules for receiving external requests and "Engine" modules for processing business logic. * Communication between these modules is handled asynchronously via gRPC and Apache Kafka, using the Decaton library to increase throughput while maintaining a low partition count. * The architecture prioritizes eventual consistency, allowing for high responsiveness and scalability across the platform. ### Global Collaboration and Conway’s Law * Development was split between teams in Korea (Core Domain) and Japan (System Integration and BFF), requiring a shared understanding of domain boundaries. * Architectural Decision Records (ADR) were implemented to document critical decisions and prevent "knowledge drift" during long-term collaboration. * The organizational structure was intentionally designed to mirror the system architecture, with specific teams (Core, Link, BFF, and Merchant Link) assigned to distinct domain layers. * This alignment, reflecting Conway’s Law, ensures that changes to external consumer platforms have minimal impact on the stable core domain logic. Successful DDD adoption requires moving beyond technical patterns like hexagonal architecture and focusing on establishing a shared understanding of roles across the organization. By structuring teams to match domain boundaries, companies can build resilient systems where the core business logic remains protected even as the external service ecosystem evolves.

GrpcApache KafkaSoftware DesignEventual Consistency+4

datadog·Jun 17, 2025·3 min readCurated summary

Breaking up a monolith: How we’re unwinding a shared database at scale

Datadog is moving away from a large shared relational database because its benefits eventually give way to coordination costs, schema fragility, noisy-neighbor problems, and scaling limits. Splitting the database is difficult and expensive, but platform investments in service development and managed Postgres can make independently owned databases practical. The key is to establish functional boundaries, provide safe cross-domain access, and automate migrations. ## Why Shared Databases Persist - Shared databases reduce operational overhead for small or fast-moving organizations. - A single database enables simple, low-latency joins across all data. - Workload isolation and access management often matter less when systems are small. - Because the cost of splitting a database is high, organizations commonly keep the shared model longer than they should. ## Signs It Is Time to Split the Database - Data grows beyond the capacity of one machine, or replication becomes too slow. - Noisy-neighbor effects make performance unpredictable. - Schema changes by one team unexpectedly affect others. - Security requirements such as access-control lists are difficult to enforce. - These issues create engineering costs, incidents, and degraded user experiences across teams. ## What Database Decomposition Requires - Identify functional ownership boundaries. - Build services for cross-domain queries where necessary. - Require consumers to use those services instead of querying another domain’s tables directly. - Provision new database instances. - Migrate data and traffic carefully from the shared database to the new instances. Datadog had previously split off large portions of its database into only a few separate databases. The experience showed that finding boundaries, enforcing them, and migrating without incidents is difficult and highly manual. ## Why Teams Resist Leaving Shared Infrastructure - Building a service may jeopardize existing product goals. - Operating a service can introduce significant maintenance and on-call work. - Cross-domain data access may be unclear or cause unacceptable latency or user impact. - Owning a database creates additional operational responsibility. - Migrations are often handcrafted, risky, and difficult to repeat. - Forcing the transition can cost more than tolerating the existing problems and create organizational resistance. ## Platform Investments That Enable Change Datadog addressed these obstacles through two major initiatives: - **Rapid:** An opinionated framework for building and operating API and gRPC services. - **OrgStore:** A managed platform for Postgres databases. Rapid reduces the cost of creating and maintaining services by providing shared configuration, common data-access patterns, and operational support. OrgStore reduces the burden of owning separate database instances. Together, these platforms make it more attractive for new projects to avoid the legacy shared database and allow existing domains to migrate incrementally. The broader lesson is that database decomposition becomes realistic when platform engineering makes service ownership, database operations, cross-domain access, and migrations safe enough to fit into normal product development.

Read original(opens in new tab)

GrpcDatabase DesignObservabilityPostgreSQL+4

datadog·Nov 20, 2024·3 min readCurated summary

How we use formal modeling, lightweight simulations, and chaos testing to design reliable distributed systems

Courier’s design illustrates why distributed systems need more than unit, integration, and chaos testing. Datadog combined formal modeling, lightweight simulation, and conventional testing to uncover system-level risks before implementation. The approach was especially important after the March 8, 2023 outage, which showed how locally reasonable decisions can produce severe global failures. ## Why Distributed Systems Require Additional Analysis - Distributed systems provide greater scale and availability, but introduce concurrency, coordination, and failure modes that are difficult to reason about intuitively. - Traditional tests operate at relatively low levels of detail and may miss high-level design flaws. - Formal models and simulations allow teams to evaluate system behavior during the design phase, before implementation choices become expensive to change. - Model checking exhaustively explores all states permitted by a design and verifies defined correctness properties. ## Formal Modeling and Lightweight Simulation - Formal modeling uses a high-level specification language to describe: - System components and their interactions - Allowed system states - Properties the system must satisfy - Lightweight simulation builds a replica that runs under controlled conditions to study statistical characteristics such as: - Latency - Cost - Scalability - Behavior under realistic workloads - Modeling verifies correctness but cannot fully assess performance-related concerns. - Neither technique validates the final implementation directly. - Keeping models and simulations synchronized with the production design adds maintenance overhead. - Datadog considered the additional effort worthwhile because Courier was foundational, needed strong reliability guarantees, and incorporated lessons from the 2023 outage. ## Courier’s Requirements Courier was created to replace a decade-old Redis-backed queuing system that had begun to face throughput, scaling, and durability limitations. Its main requirements were: - **Multi-tenancy:** Isolate teams and products so one tenant cannot significantly disrupt others. - **At-least-once delivery:** Messages must not be lost; they must be delivered and acknowledged or sent to a dead-letter queue. - **Graceful degradation and high availability:** Throughput should decline roughly linearly as compute capacity is lost, rather than collapsing entirely. - **Horizontal scalability:** Throughput should increase linearly as compute capacity is added. The graceful-degradation requirement directly addressed the March 8 outage, when lost compute capacity caused a disproportionate throughput failure. ## FoundationDB Sharding for Tenant Isolation - Courier uses multiple FoundationDB clusters. - Each tenant is assigned to a subset of clusters. - No two tenants share the exact same cluster subset. - The initial design used: - Eight FoundationDB clusters - Four clusters per tenant - A theoretical maximum of `8 choose 4 = 70` tenant assignments - If one tenant saturated or disabled its four clusters, other tenants would still retain access to at least 25% of the total cluster capacity. - This arrangement provided sufficient isolation for the intended workloads. ## Broker Layer and High Availability - A broker layer exposes gRPC APIs for: - Sending messages - Receiving messages - Deleting messages - Clients connect only to the brokers, which apply the tenant-sharding logic. - Brokers health-check FoundationDB clusters and remove unhealthy clusters from consideration. - Both brokers and FoundationDB clusters are deployed across three availability zones to improve resilience. Courier demonstrates that formal verification and simulation are valuable complements to implementation testing. For mission-critical distributed services, teams should validate both correctness and operational behavior early, while also using unit, integration, and chaos testing to verify the final system.

Read original(opens in new tab)

GrpcDistributed SystemsHigh AvailabilityFoundationdb+4

datadog·May 1, 2024·3 min readCurated summary

How we built the Datadog heatmap to visualize distributions over time at arbitrary scale

Datadog uses DDSketch-powered distribution metrics and heatmaps to reveal performance patterns that percentile lines can hide. Heatmaps preserve the full shape of latency distributions across hosts and time, making distinct behavioral modes, seasonality, and outliers visible. The visualization is designed to remain scalable and readable even with hundreds of trillions of underlying datapoints. ## Why Unaggregated Distributions Matter - Line graphs reduce billions of events to a single value, such as p50, p99, or max. - Multiple percentile lines provide more context, but the selected percentiles remain arbitrary and can obscure important behavior. - Aggregated percentile changes may suggest that all requests are slowing when only one subset of traffic is changing. - Heatmaps expose separate “modes”—distinct groups of measurements with different behavior. - For example, periodic latency spikes may come from a low-latency benchmarking service rather than from a general degradation in the endpoint. - Filtering out an identified mode can reveal other patterns, such as daily seasonality in the remaining traffic. ## Building Heatmaps with DDSketch - DDSketch sacrifices a small amount of precision to represent extremely large numbers of observations efficiently. - Datadog sends histogram bins and counts to the frontend instead of transmitting every individual datapoint. - Limiting the number of bins keeps the payload size constant as traffic volume grows. - Counts use `float32`, supporting values up to approximately `3 × 10^38` per bin—far beyond practical monitoring volumes. - This allows heatmaps to represent massive datasets, including hundreds of trillions of datapoints. ## Preserving Resolution and Avoiding Aliasing - Heatmap requests contain time buckets, distribution bins, and counts. - Since bucket boundaries are shared across a request, Datadog stores those boundaries only once. - Boundaries must be explicit because distributions may use logarithmic rather than linear scales. - Time buckets need to align with the source data intervals. - Misaligned intervals create aliasing artifacts: for example, grouping 10-second data into 7-second buckets produces repeating count patterns such as `[1, 1, 2, 1, 1, 2, …]`. - Careful discretization preserves the resolution available in the original DDSketch data. ## Designing the Color Scale - The default palette begins with light blue, consistent with other single-series Datadog visualizations. - It transitions toward purple to match Datadog’s visual identity. - The scale avoids lingering on red, which can imply negative alerts, and ends in orange for the hottest values. - Color choices must communicate both the volume and structure of the distribution. ## Maintaining Dynamic Range - A few high-count bins can dominate a linear color scale, leaving most of the heatmap visually indistinguishable. - This is especially problematic for power-law distributions with a dense central mode and a long tail. - A linear scale may clearly show the main mode around 20 ms while hiding a smaller mode near 1 second. - Human brightness perception is nonlinear, approximately following a power law described by Stevens’ law. - Applying nonlinear color interpolation improves the visibility of meaningful differences across both dense regions and long tails. - This helps preserve distribution details that would otherwise be lost when the color range is dominated by outliers or highly concentrated buckets. Datadog’s heatmap approach combines DDSketch compression, aligned high-resolution buckets, and perceptually informed color scaling. For systems where averages or a handful of percentiles conceal important subpopulations, distribution heatmaps provide a more reliable way to investigate performance at scale.

Read original(opens in new tab)

GrpcDatabase DesignDatadogData Visualization+4

datadog·Apr 13, 2022·1 min readCurated summary

It's always DNS . . . except when it's not: A deep dive through gRPC, Kubernetes, and AWS networking | Datadog

The supplied content does not include the blog post itself; it mainly contains Datadog navigation links and a promotional banner stating that Datadog was named a Leader in Gartner’s Magic Quadrant for Observability Platforms. The URL suggests the missing article concerns a gRPC, DNS, and load-balancing incident, but no incident details are provided. ## Available Content - Datadog promotes its recognition as a Gartner Magic Quadrant Leader. - The navigation lists products across: - Infrastructure and application monitoring - Logs, databases, and data observability - Security - Digital experience monitoring - Software delivery and service management - AI and platform capabilities - The linked page path references an engineering post titled “gRPC, DNS, and Load Balancing Incident.” ## Missing Technical Details - No description of the incident or its impact - No explanation of the DNS or load-balancing failure - No timeline, root-cause analysis, or remediation steps - No lessons learned or recommendations Please provide the full article text for a substantive technical summary.

Read original(opens in new tab)

GrpcObservabilityDnsIncident Response+1

datadog·Apr 13, 2022·3 min readCurated summary

It's always DNS . . . except when it's not: A deep dive through gRPC, Kubernetes, and AWS networking

A routine update to a critical metrics query service caused intermittent errors and increased latency. Although logs initially pointed to DNS failures, the investigation revealed a deeper networking problem involving dropped packets and saturated AWS VPC connection tracking. The incident highlighted how Kubernetes, Cilium, AWS networking, and DNS behavior can interact in ways that obscure the true cause. ## Initial Symptoms and Apparent DNS Failures - Errors increased whenever the metrics query service was rolled out. - The service retrieves data from metric stores for dashboards and monitor evaluations. - Automatic retries reduced user-facing failures but increased latency. - Service logs showed DNS errors when connecting to dependencies inside Kubernetes. - The investigation therefore began with the cluster’s DNS infrastructure. ## NodeLocal DNSCache Reaches Its Limits - NodeLocal DNSCache runs as a `node-local-dns` DaemonSet on every Kubernetes node. - DNS pods had: - A 64 MB memory limit - A `max_concurrent` limit of 1,000 requests - The pods experienced out-of-memory errors and rejected requests during rollouts. - Increasing memory to 256 MB stopped the OOM errors, but DNS failures continued. - Request volume was far below the expected capacity: - Normally about 400 queries per second - Nearly 2,000 queries per second during rollouts - Expected capacity of at least 200,000 queries per second - Upstream resolvers were marked unhealthy, suggesting that NodeLocal DNSCache could not establish or maintain connections. - Because upstream requests could wait up to five seconds, connection failures consumed concurrency slots and made the cache appear overloaded. ## Evidence of a Network Problem - The instances were below their 5-Gbps sustained throughput limits. - TCP retransmits increased in correlation with service rollouts. - Engineers suspected brief traffic spikes, or microbursts, that were not visible in aggregate throughput metrics. - This shifted the investigation from DNS configuration toward lower-level AWS networking behavior. ## AWS VPC Connection Tracking - ENA metrics revealed a significant increase in `conntrack_allowance_exceeded`. - This metric counts packets dropped when VPC connection tracking becomes saturated. - Connection tracking maintains state for network flows and supports features such as stateful EC2 security groups. - The infrastructure used two tracking layers: - VPC conntrack maintained at the hypervisor level - Linux conntrack inside each instance - VPC conntrack appeared saturated even though Linux conntrack contained fewer than 60,000 entries—well within the observed capacity of similar instances. - AWS Support confirmed that conntrack capacity varies by instance type and that VPC conntrack limits could differ substantially from Linux conntrack limits. - Scaling to larger instances resolved the symptoms, but the engineers wanted to understand the traffic pattern and find a more efficient long-term solution. ## VPC Flow Logs as the Next Investigation Tool - The team turned to Amazon VPC Flow Logs to examine the service’s low-level network behavior. - These logs were expected to clarify why connection tracking filled up and how rollout traffic contributed to the saturation. - The investigation was still ongoing at the point where the provided article excerpt ends.

Read original(opens in new tab)

GrpcDatabase DesignKubernetesPostgreSQL+4