datadog·Jun 23, 2026·3 min readCurated summary

How we migrated a live routing system using AI-assisted refactoring

Stream Router evolved from a small configuration file into a critical control-plane service routing Datadog’s massive metrics workload. Its original FoundationDB key-value model eventually hit transaction-size and performance limits because relational relationships were reconstructed in application code. Datadog redesigned the system around PostgreSQL and DuckDB, using AI-assisted, test-driven refactoring to accelerate the migration without disrupting production traffic. ## Stream Router’s Role in Datadog’s Metrics Pipeline - Datadog processes more than a hundred trillion events per day. - Stream Router determines which Kafka cluster, topic, partitions, and sharding strategy should handle each datapoint. - It serves both producers and queriers but does not process Kafka messages itself. - Routing decisions change frequently as infrastructure evolves, making correctness and historical tracking essential. ## From Configuration File to Control Plane - In 2016, routing was managed through a small configuration file distributed to services. - As the platform grew, the file expanded to thousands of lines and required manual edits and rollouts. - Stream Router replaced this workflow with: - A centralized gRPC service - API-managed routes - Automated, gradual rollouts - The write path used FoundationDB, while the read path served static RocksDB snapshots restored into memory. - This eventually became a bottleneck as routing tables and operational changes grew larger. ## Why the Key-Value Model Stopped Scaling - Routes reference streams and sharding strategies, while rules reference routes. - These relationships are inherently relational and require cross-entity validation. - The KV implementation loaded tens of thousands of records into application processes and reconstructed database-like relationships in code. - Some operations exceeded FoundationDB transaction-size limits. - Moving to PostgreSQL without changing the access patterns would not solve the issue; certain operations were estimated to require 45 minutes because of thousands of sequential database round trips. - The fundamental problem was the data model and application logic, not simply the choice of database. ## Designing the New Storage Architecture - The team redesigned the schema manually before using AI tools. - The relational model introduced explicit foreign keys between: - Streams - Sharding strategies - Routes - Rules - PostgreSQL was selected for the write path because it provided the required relational semantics and transaction model. - DuckDB was selected for the read path because: - It is embeddable and suitable for snapshot-based serving - It supports array columns - Its SQL dialect is closely compatible with PostgreSQL - Shared query logic could therefore work across both storage engines. ## AI-Assisted Refactoring - Claude and Cursor were used to accelerate a systematic, test-driven migration. - For each method, developers supplied: - The old implementation - The new schema - A failing test - AI generated an initial implementation, while tests determined whether it was correct. - The models assisted with method-level refactoring rather than autonomously designing the architecture. - Human expertise remained central to schema design, migration strategy, and evaluating system-level risks. ## Foundations for a Safe Migration - The migration benefited from infrastructure already present at Datadog. - Stream Router’s storage layer was isolated behind an internal `Controller` interface. - This modularity helped contain storage changes and enabled incremental refactoring. - Existing tests and clear boundaries provided confidence in generated implementations while production traffic continued. The central lesson is that AI was most effective as an accelerator inside a disciplined engineering process. A well-designed relational schema, modular storage abstraction, and failing tests provided the safety mechanisms; AI helped implement the resulting changes faster, but did not replace human architectural judgment.

Read original(opens in new tab)

Database DesignPostgreSQLApache KafkaGrpc+4

netflix·Jun 3, 2026·4 min readCurated summary

Dynamic Repartitioning for Time Series Workloads

Netflix’s TimeSeries Abstraction uses Cassandra to ingest and query petabytes of temporal data with millisecond-scale latency, but growing partitions can cause seconds-long reads, timeouts, and resource exhaustion. Its initial time-based partitioning works well when workload estimates are accurate, yet traffic changes and outlier IDs can make partitions too large or too small. Netflix therefore developed automated time-slice repartitioning and, for isolated hot IDs, asynchronous dynamic partitioning at the individual-ID level. ## Cassandra and the Wide-Partition Problem - Cassandra provides: - High-throughput, low-latency reads and writes - Cost-effective operation at scale - Strong operational familiarity within Netflix - TimeSeries datasets accumulate events over time, creating potentially very wide partitions. - Wide partitions can lead to: - Read latencies increasing from milliseconds to seconds - Request timeouts - Garbage-collection pauses - High CPU utilization and thread queueing - Scaling Cassandra clusters can help, but Netflix sought more targeted solutions. ## Initial Time-Based Partitioning - TimeSeries divides data into discrete time slices to keep partitions manageable. - This structure also makes it efficient to: - Query data by time - Drop old data without creating large tombstone problems - At dataset creation, users provide expected workload characteristics. - Netflix’s provisioning pipeline uses those inputs, along with Monte Carlo simulations, to select infrastructure and partition settings. ## Why Static Provisioning Falls Short - Workloads may be unknown or inaccurately estimated during initial provisioning. - Traffic patterns, client behavior, and product needs can change over time. - A small number of TimeSeries IDs may generate far more events than the rest. - Time slices provide a way to change partitioning for future data, but manually updating thousands of datasets is impractical. ## Repartitioning Entire Time Slices - Cassandra introspection tools, such as `nodetool tablehistograms`, expose partition-size distributions. - Netflix added a background worker that: - Monitors partition histograms for time slices - Publishes observations through a Cassandra virtual table - Detects partitions that are too large or too small - Calculates a new partitioning adjustment factor - Target partition density is typically between 2 MiB and 10 MiB, depending on workload. - The worker updates the strategy for future time slices. For example, it may expand a `time_bucket` interval from 60 seconds to 604,800 seconds when partitions are too small. - This approach reduced read latency and timeouts caused by thread queueing. - Its limitation is that it changes partitioning broadly and is ineffective when only a minority of IDs produce oversized partitions. ## Handling Isolated Problem IDs Netflix considers several responses when only some IDs are problematic: - **Do nothing:** Appropriate when wide partitions do not affect application-level metrics. - **Partial returns:** Abort a request after it exceeds a latency SLO while returning data already collected; useful when latency matters more than completeness. - **Block IDs:** Prevent exceptionally bad test, spam, or otherwise harmful IDs from destabilizing the system. - These options are inadequate when valid, important IDs must return all their data despite generating large partitions. ## Dynamic Partitioning per ID Dynamic partitioning addresses outliers by splitting partitions for individual TimeSeries IDs rather than modifying an entire table. The asynchronous pipeline has three stages: - **Detection:** The read path identifies partitions that exceed a configured size threshold. - **Planning and splitting:** The system asynchronously plans and executes splits into appropriately sized partitions. - **Serving reads:** Once splits are available, read requests are transparently rerouted to them. During each read, the server tracks the bytes retrieved for a partition. If usage exceeds the threshold, it emits a detection event to Kafka containing information such as: - The Cassandra time-slice table - The affected TimeSeries ID - The existing time and event bucket - Whether the partition is immutable - A version identifier ## Practical Recommendation Use whole-time-slice repartitioning when an entire dataset is systematically over- or under-partitioned. For isolated but important high-volume IDs, dynamic per-ID partitioning provides a more precise way to control latency without disrupting the rest of the dataset.

Read original(opens in new tab)

Database DesignApache KafkaTime Series DatabaseApache Cassandra+1

netflix·May 1, 2026·3 min readCurated summary

State of Routing in Model Serving

Netflix’s centralized ML serving platform provides a single, domain-independent API for model inference across personalized experiences and other use cases. Rather than exposing individual scoring functions, Netflix packages feature computation, preprocessing, inference, and postprocessing into self-contained model workflows. The core routing challenge is directing each request to the correct model version and serving cluster while keeping client services independent from model changes and infrastructure topology. ## Models as End-to-End Workflows - Netflix distinguishes **model serving** from traditional model inference: - Inference typically means `infer(features) -> score`. - Serving includes preprocessing, feature computation, optional trained components, and postprocessing. - Example workflows include: - Ranking titles for a personalized Continue Watching row using user, country, and device context. - Predicting payment fraud using user, country, and transaction details. - Models declare the facts they need, while the serving platform retrieves those facts from other microservices. - During offline training, Netflix’s ML fact store provides snapshots for bulk feature computation. - Calling services provide standard request context and domain-specific inputs, while the platform handles feature generation, model selection, and execution. ## Platform Design Principles - **Model innovation without client changes** - Client applications integrate with the platform once. - Model versions, A/B tests, additional experimental data, logging, and model selection remain hidden behind the platform API. - **Clients decoupled from model sharding** - Models run across multiple serving cluster shards, each with its own Virtual IP address. - Shard assignments can change based on traffic, SLAs, model architecture, and resource availability. - Clients should not need to track these VIP changes. - **Flexible traffic routing** - Routing must support A/B allocations, gradual traffic shifts, new model versions, new VIPs, and client-specific overrides. - Safe lifecycle management requires support for shadow deployments, canaries, rollbacks, and migrations. ## Switchboard: Context-Aware Routing - Generic API gateways and service-mesh proxies did not satisfy Netflix’s requirements. - Netflix needed: - Native integration with its experimentation platform. - gRPC support. - Routing based on rich, domain-specific request context. - Model-specific rollout and migration controls. - Netflix built **Switchboard**, a custom proxy layer handling more than one million requests per second. - Switchboard is the mandatory entry point for clients and: - Routes requests to the appropriate model based on request context. - Applies configured context enrichment before invoking the model. - Hides model locations and infrastructure changes from client services. ## Objective Abstraction - Every request must provide an **Objective**, an enumeration defined by the serving platform. - The excerpt introduces Objectives as a central abstraction for identifying the business purpose of a serving request, but the supplied text ends before describing its full roles. Netflix’s approach is to centralize routing, experimentation, and model execution behind one stable API. This allows client applications to evolve independently while researchers can iterate on models and safely manage large-scale production rollouts.

Read original(opens in new tab)

Database DesignMachine LearningPostgreSQLMicroservices+4

cloudflare·May 1, 2026·3 min readCurated summary

Introducing Dynamic Workflows: durable execution that follows the tenant

Dynamic Workflows extends Cloudflare’s durable execution system to multi-tenant and dynamically generated applications. While Dynamic Workers provide isolated runtime compute, Durable Object Facets provide tenant-specific storage, and Artifacts provide versioned source control, Dynamic Workflows lets each tenant supply its own long-running workflow code. The result is durable execution that can resume the correct tenant’s workflow after failures, hibernation, or delays of days. ## The Gap Between Durable and Dynamic Execution - Cloudflare Workflows turns a `run(event, step)` function into a durable program. - Workflow steps can: - Survive isolate recycling and failures - Sleep for hours or days - Wait for external events - Resume from the exact point where execution stopped - Workflows V2 supports up to 50,000 concurrent instances and 300 new instances per second per account. - Traditional Workflows assume the workflow class is included in the deployment and statically configured in `wrangler.jsonc`. - That model breaks for: - Multi-tenant SaaS platforms - AI-generated tenant applications - Repository-specific CI/CD pipelines - Agents that create their own durable plans - In these systems, workflow code varies by tenant, agent, repository, or request, so a single statically bound class is insufficient. ## Dynamic Workflows - `@cloudflare/dynamic-workflows` is a roughly 300-line TypeScript library. - It introduces a Worker Loader that: - Loads each tenant’s code dynamically - Routes workflow creation to the appropriate tenant - Ensures later workflow execution returns to that tenant’s code - The Loader creates a dynamic Worker with: - A tenant-specific module - A `TenantWorkflow` entrypoint - A wrapped `WORKFLOWS` binding - The dynamic entrypoint is registered as the workflow class in `wrangler.jsonc`. - Tenant code remains ordinary Cloudflare Workflows code and does not need to know it is being dynamically dispatched. ## Tenant Workflow Behavior - Tenants can use the normal Workflow APIs, including: - `env.WORKFLOWS.create(...)` - Workflow IDs and `.status()` - `.pause()` - Retries and durable steps - `step.sleep('24 hours')` - `step.waitForEvent()` - A tenant can define a standard `WorkflowEntrypoint` with a `run(event, step)` method. - The library’s primary responsibility is preserving the association between a workflow instance and the tenant implementation when the workflow resumes later. ## Three-Layer Execution Model - Dynamic Workflows consists of three layers: - The Cloudflare Workflows engine - The platform’s Worker Loader - The tenant’s dynamically loaded Worker code - A request first enters the Loader, which identifies the tenant and routes execution to its dynamic code. - The workflow engine then persists the workflow state and later invokes `run(event, step)`. - The Loader resolves the correct tenant implementation when execution resumes, even after delays or failures. Dynamic Workflows provides the missing durable-execution counterpart to Cloudflare’s dynamic compute, storage, and source-control primitives. It is particularly suited to platforms where customers or agents generate workflow code at runtime while still requiring standard durable guarantees.

Read original(opens in new tab)

Database DesignCI/CDTypeScriptCloudflare Workers+4

google·May 1, 2026·3 min readCurated summary

Catalyzing scientific impact through global partnerships and open resources

Google Research argues that scientific breakthroughs have the greatest impact when their software, datasets, and methods are openly shared and responsibly maintained through global partnerships. Its open-science efforts span genomics, neuroscience, climate, biodiversity, and healthcare, reaching more than 250,000 researchers and developers. The post concludes that collaboration and open resources can turn individual discoveries into tools for broader scientific progress and real-world benefits. ## Partnerships Across the Scientific Ecosystem - Google Research works with organizations including UCSC’s Genomics Institute, Janelia Research Campus, ISTA, CSIRO, AIIMS, and the Centre for Population Genomics. - It supports major international initiatives such as: - The Human Pangenome Research Consortium - The Earth BioGenome Project - The NIH BRAIN Initiative - Google is also developing communities of practice for scientific developers, beginning in India, Korea, Japan, and Australia. ## Open-Source Tools and Datasets - **Genomics** - DeepVariant, DeepConsensus, and DeepPolisher support DNA analysis from sequencing through genome assembly. - These tools have helped process exomes and whole genomes from 2.5 million people. - **Neuroscience** - Flood-filling networks, Neuroglancer, and TensorStore enable analysis and visualization of petascale brain reconstructions. - The public H01 dataset contains 1.4 petabytes of human brain tissue data and has been accessed more than 200,000 times. - MICrONS provides a large wiring and functional map of the mouse visual cortex. - **Earth and Atmospheric Science** - Open Buildings contains 1.8 billion building detections across 58 million square kilometers. - Caravan supports large-scale hydrology and flood forecasting in 150 countries, covering roughly 2 billion people. - Groundsource includes 2.6 million historical urban flood events from more than 150 countries. - NeuralGCM is a differentiable hybrid atmospheric model, while FireBench supports wildfire research with high-resolution synthetic data. - **Biodiversity** - SpeciesNet classifies 2,498 animal categories in wildlife-camera images. - **Healthcare** - HAI-DEF provides open-weight medical foundation models, including MedGemma, with more than 4.8 million downloads. - Open Health Stack offers secure, offline-capable tools based on modern healthcare standards. - OHS-powered applications have reached more than 65 million people across over 10 countries. ## Scientific and Humanitarian Impact - **Genomics** - Work with UCSC improved pangenome references and reduced genetic-variant identification errors by 50%. - The research contributes to more representative genomic resources through the Human Pangenome Research Consortium. - **Weather and Agriculture** - The University of Chicago’s Human-Centered Weather Forecasts Initiative used NeuralGCM and ECMWF systems to predict India’s monsoon onset up to a month ahead. - Forecasts, including an unusual dry spell, were delivered by SMS to 38 million Indian farmers to support planting decisions. - **Disaster Response** - UNHCR and other organizations use Open Buildings to improve survey sampling for displaced populations. - The dataset also supports research into building vulnerability to sea-level rise in the Global South. - Sunbird AI uses the data to assess energy needs in urban and rural communities. - **Neuroscience and Medicine** - Johns Hopkins researchers used the H01 brain dataset to identify a possible new form of neuronal communication, suggesting that current models of brain organization may be incomplete. - The finding could have implications for understanding conditions such as Alzheimer’s disease. - Google also partnered with Stanford Medicine and UCSC to accelerate genome analysis in urgent cases of suspected genetic disease. ## Practical Conclusion The post presents open-source scientific infrastructure, accessible datasets, and cross-border partnerships as essential to accelerating discovery. Researchers and institutions can maximize impact by sharing reproducible tools, maintaining resources collaboratively, and applying them to urgent global challenges.

Read original(opens in new tab)

Database DesignDeep LearningGenomicsHealthcare Ai+4

cloudflare·Apr 30, 2026·3 min readCurated summary

Agents can now create Cloudflare accounts, buy domains, and deploy

Agents can now take an application from development to production by creating Cloudflare accounts, obtaining API tokens, purchasing domains, and deploying code. Cloudflare’s integration with Stripe Projects removes most manual setup while keeping humans involved for permissions, terms acceptance, and payment approvals. The underlying protocol combines service discovery, authorization, and tokenized payments so agents can provision infrastructure on a user’s behalf. ## Zero-to-production deployment - Users install the Stripe CLI, authenticate, and run: ```bash stripe projects init ``` - An agent can then build an application and deploy it to a new domain. - If no Cloudflare account exists, one is provisioned automatically. - If an account already exists, the user authorizes access through OAuth. - The agent can: - Create a Cloudflare account - Obtain an API token - Register a domain - Deploy the application to production - Humans are prompted only when approval, terms acceptance, or payment setup is required. ## The protocol: discovery, authorization, and payment - **Discovery:** Agents query a catalog of available provider services and select the resources needed for the user’s request. - **Authorization:** The orchestrating platform verifies the user’s identity and enables providers to create accounts, connect existing accounts, and issue credentials securely. - **Payment:** Tokenized payment credentials let providers charge the user without exposing raw card details to the agent. - The approach builds on OAuth, OIDC, and payment-tokenization standards. ## Service discovery through a catalog - Agents can inspect available services with: ```bash stripe projects catalog ``` - They can select Cloudflare Registrar with: ```bash stripe projects add cloudflare/registrar:domain ``` - Providers expose service catalogs through REST APIs returning JSON. - This gives agents the context to choose appropriate products without requiring users to know which provider offers them. ## Automatic account creation and authorization - Stripe acts as the identity provider and attests to the user’s identity. - Cloudflare creates a new account automatically when the user has none. - Credentials are securely stored by the Stripe Projects CLI but made available to the agent for authenticated Cloudflare API requests. - Existing Cloudflare users authorize the integration through a conventional OAuth flow. ## Controlled agent spending - Agents never receive the user’s raw credit card information. - Stripe supplies Cloudflare with a payment token for subscriptions and purchases. - Spending is initially capped at $100 per month per provider. - Users can raise the limit and configure Cloudflare Budget Alerts as needed. ## Broader platform integration - The protocol is not limited to Stripe Projects. - Any platform with signed-in users can act as the orchestrator and integrate with Cloudflare. - This enables coding-agent platforms to let users deploy directly to production without requiring separate dashboard logins, token copying, or manual account setup. - Cloudflare is also enhancing the experience through its Code Mode MCP server and Agent Skills. Cloudflare and Stripe’s integration makes infrastructure provisioning an agent-driven workflow while retaining safeguards around identity, consent, and spending. For platforms building coding agents, adopting the protocol could provide a frictionless path from generated code to a live, paid production deployment.

Read original(opens in new tab)

Database DesignCloudflareStripeOauth+4

grammarly·Apr 30, 2026·2 min readCurated summary

Educator of the Year

Grammarly’s inaugural Educator of the Year Award honors teachers nominated directly by their students. The first winner, Dr. Humberto López Castillo of the University of Central Florida, is recognized for teaching precise, accessible communication and applying it to public health, technology, and community engagement. His approach combines audience-aware writing, responsible AI use, and hands-on research. ## Student-Led Recognition - Students nominate educators through short videos describing their impact on academic and professional development. - UCF student Vardhan Avaradi nominated Dr. López Castillo for encouraging students to make their language “precise yet accessible.” - López Castillo is a pediatrician, public health researcher, translator, and four-language polyglot from Panama. - His teaching emphasizes collaboration and the connection between individual health and broader communities. ## Communicating With Different Audiences - Students translate complex public health topics for audiences outside academia. - Assignments have included: - Storybooks about mosquitoes for kindergarteners - Monopoly-style games about living with HIV - Rap songs explaining tuberculosis - Podcasts that personalize epidemiology - His medical experience informs this approach: communication must change depending on whether the audience is a child, parent, or professional researcher. ## AI Requires Human Judgment - López Castillo permits students to use AI for drafting but expects them to verify and critically evaluate its output. - When AI-generated citations referenced nonexistent research, he treated the error as a lesson rather than a punishment. - He compares AI to a calculator: useful and powerful, but dependent on the judgment of the person using it. - He and Vardhan are developing a machine learning project using the NIH All of Us dataset, which contains nearly one million de-identified health records. - Their research explores using AI to classify populations and predict health risks. ## Preparing Students for Broader Impact - Students leave with stronger writing, critical-thinking, collaboration, and communication skills. - López Castillo’s teaching focuses not just on adopting new tools, but on using them responsibly and communicating with purpose. - His students learn to reach people beyond academic audiences while keeping human needs at the center of technology and research. The post’s central recommendation is to pair emerging technologies with critical thinking, audience awareness, and a strong sense of social responsibility.

Read original(opens in new tab)

Database DesignAIMachine LearningPublic Health+3

line·Apr 30, 2026·4 min readCurated summary

ID-JAG, a next-generation standard candidate for solving authentication challenges in the AI era

ID-JAG extends enterprise SSO trust to API access between AI agents, applications, and services. It uses an enterprise IdP to centrally evaluate permissions and issue a signed JWT that can be exchanged for a resource-specific access token. This can reduce consent prompts, improve auditing, and limit token sprawl, but organizations should adopt it cautiously while the specification remains an Internet-Draft. ## The Authentication Challenge in the AI Era - AI agents increasingly perform real work, including: - Searching systems - Querying databases - Sending messages - Creating tickets - As the number of connected services grows, authentication and authorization become more complex. - Poorly coordinated integrations can turn AI from a productivity tool into an operational bottleneck. - ID-JAG is being discussed by the IETF OAuth Working Group as a potential solution. ## What ID-JAG Is - ID-JAG, or Identity Assertion JWT Authorization Grant, extends the enterprise IdP’s SSO trust relationship to API access. - The IdP centrally determines: - Which application or agent may access an API - Which user or identity it acts for - Which scopes or permissions are allowed - It combines: - OAuth 2.0 Token Exchange (RFC 8693) - JWT Profile for OAuth 2.0 Authorization Grants (RFC 7523) - The IdP issues a cryptographically verifiable JWT as an “introduction” or authorization assertion. - The target authorization server validates that assertion and issues the final access token. ## The ID-JAG Participants and Flow The model involves four main parties: - **Requesting Agent:** An AI agent or application calling another service’s API - **Enterprise IdP:** Provides SSO and enforces centralized organizational policies - **Authorization Server:** Issues tokens for the target application - **Resource Server:** Hosts the API being accessed The basic five-step flow is: 1. The user signs in to the requesting agent, which obtains an ID token from the IdP. 2. The agent presents the ID token to the IdP and requests an ID-JAG through token exchange. 3. The IdP evaluates organizational policy and issues the ID-JAG if access is allowed. 4. The agent presents the ID-JAG to the target authorization server and receives an access token. 5. The agent uses the access token to call the resource server. The key architectural shift is that authorization decisions move from isolated agent-to-service relationships toward a centrally governed relationship between the enterprise IdP and target authorization servers. ## Benefits for User Experience and Auditing - Centralized IdP policies can reduce repeated consent screens. - This is especially useful when AI agents connect to many tools and services. - ID-JAG claims can record important context, such as: - The user whose authority is being delegated (`sub`) - The requesting agent (`client_id`) - The target authorization server (`aud`) - Approved scopes (`scp`) - Issuer, issue time, expiration, and unique token ID - Centralized issuance logs provide a clearer view of service-to-service relationships. - Security teams can more easily determine which agent accessed which service, on whose behalf, and with what permissions. - The same records can support incident investigation, compliance audits, and accountability. ## Centralized Control and Reduced Token Sprawl - The IdP can help detect and control unauthorized “shadow AI” integrations. - It can evaluate every token exchange using consistent organizational policies. - Requested scopes can be narrowed or overridden according to enterprise security requirements. - Blocking future access can be handled centrally instead of by changing policies across every endpoint. - ID-JAG may reduce token sprawl by avoiding additional long-lived refresh tokens. - The draft recommends that resource authorization servers generally not issue refresh tokens when an ID-JAG is exchanged. - Agents can instead submit a new ID-JAG to obtain another access token, replacing scattered API keys, service credentials, and refresh tokens with dynamic, policy-based trust. ## Adoption Requirements and Risks - ID-JAG is still an IETF Internet-Draft, not a finalized RFC. - Its behavior may change, so systems should avoid tightly coupling their core architecture to the current draft. - Before implementation, organizations need to verify that: - The requesting agent is registered as an OAuth client with both the enterprise IdP and the target authorization server. - Explicit trust relationships exist between the IdP and agent, and between the IdP and authorization server. - The IdP has pre-authorized the agent to act on users’ behalf for the relevant services and scopes. - Deployment also requires coordinated support from agents, enterprise IdPs, authorization servers, and resource servers. Organizations should treat ID-JAG as a promising architectural direction for governing AI-agent access, while isolating its implementation behind adaptable interfaces until the standard stabilizes. Pilot deployments should focus on centralized policy enforcement, detailed audit logging, strict scope control, and minimizing long-lived credentials.

Read original(opens in new tab)

Database DesignOauthJwtIdentity Provider+4

gitlab·Apr 30, 2026·3 min readCurated summary

Automate detection testing with GitLab CI/CD and Duo

GitLab’s WATCH framework continuously tests whether security detections still work in real conditions, rather than only verifying that detection rules deploy successfully. It runs simulated attacks in staging, checks alert propagation through logging, SIEM, and SOAR systems, and reports failures automatically. The framework uses GitLab CI/CD to schedule randomized tests, correlate expected alerts, and publish detection-health results. ## The Detection-Validation Gap - Security detections can silently fail because of: - Log schema changes - SIEM updates - Ingestion or pipeline misconfigurations - Other changes between the log source and alerting systems - Reinjecting synthetic logs into a SIEM can test rule logic, but it does not validate real-world behavior or the log-ingestion path. - GitLab’s detections-as-code pipelines confirm that rules can be created and deployed, but not that they fire when the targeted activity occurs. - WATCH fills this gap by validating detections end to end. ## WATCH’s Testing Lifecycle - **Scheduling:** A weekly GitLab CI/CD pipeline discovers active tests and assigns them randomized execution times. - **Heads-up notification:** WATCH creates a dedicated “WATCH Heads Up” SOAR record containing the detections expected to fire. - **Execution:** Scripts perform simulated malicious actions in staging, such as resetting an administrator password or making suspicious API calls. - **Detection:** Activity logs flow through ingestion into the SIEM, where detection rules process them. - **Correlation:** SOAR matches alerts to registered WATCH tests using: - The time window between execution and alerting - Actor identity, such as an IP address or username - The detection rule ID - **Verification:** A follow-up job confirms that all expected detections fired, updates detection metadata, and publishes results to a GitLab Pages dashboard. - Failed tests generate notifications in the team’s Slack channel. - Correlation prevents test alerts from being escalated as genuine incidents while still validating the complete alerting pipeline. ## GitLab CI/CD Implementation WATCH is organized into three pipeline stages: - **`schedule_pipelines`:** - Runs weekly. - Finds active tests and groups them into scheduled pipelines. - Passes the selected tests through the `TESTS_TO_RUN` variable. - **`run_tests`:** - Executes the assigned attack simulations. - Saves execution results in `detection_status.json`. - Records SOAR identifiers needed for later alert correlation. - **`pages`:** - Queries the SOAR to verify alert generation and routing. - Updates `detection_status.json` with test results. - Deploys the latest status data and dashboard assets to GitLab Pages. The example configuration uses Python 3.12, pipeline inputs to enable weekly scheduling or dashboard updates, conditional `rules`, and GitLab Pages artifacts. Scheduled execution is randomized to avoid predictable test patterns and to expose timing-related problems. ## Practical Recommendation Organizations with critical security detections should add continuous behavioral testing alongside detections-as-code validation. A framework like WATCH can provide earlier warning of broken ingestion, rules, or routing while reducing the cost and generic limitations of commercial breach-and-attack simulation tools.

Read original(opens in new tab)

Database DesignPythonSecurity MonitoringSiem+4

stripe·Apr 29, 2026·2 min readCurated summary

Giving agents the ability to pay

Agents are increasingly capable, but making purchases still requires access to today’s payment systems. Stripe is addressing this with Link’s wallet for agents, which lets users authorize purchases without exposing raw payment credentials. The system uses one-time cards or Shared Payment Tokens (SPTs), with users reviewing each request before approval. ## Link’s Wallet for Agents - Consumers connect an agent to their Link wallet through OAuth. - Agents can request: - One-time-use virtual cards - Shared Payment Tokens backed by cards or bank accounts in Link - Credentials can be restricted by amount, currency, and merchant. - Users approve requests on the web or through Link’s iOS and Android apps. - Users can track spending and manage connected agents in Link. - Stablecoins, agentic tokens, and additional payment methods are planned. ## Approval and Spending Controls - Each spend request currently requires explicit user review. - Link provides transaction context so users can understand what they are approving. - Future controls will support spending limits and allow agents to act without approval in predefined situations. - Agents never receive users’ underlying payment credentials. ## Stripe Issuing for Agents - Link’s wallet is built on Stripe Issuing infrastructure. - Businesses can use Issuing APIs to create customized agent wallets and card experiences. - Developers can control: - Onboarding and fund flows - Card-level permissions - Transaction authorization and fraud checks - Real-time and historical spending visibility - The infrastructure includes virtual cards, fund storage, spending controls, transaction monitoring, and fraud prevention tools. ## Potential Use Cases - Developers can automate business purchases and recurring spend. - Fintech companies can issue cards for real-time expense management and reconciliation. - Vertical SaaS platforms can let SMB agents make purchases under the platform’s brand. - Marketplaces can enable supplier payments, logistics, and fulfillment purchases through agent-issued cards. Stripe’s offering gives agents a practical way to transact through existing payment networks while preserving user oversight and credential security. Developers can use Link for a ready-made wallet or Stripe Issuing to build customized agentic payment workflows.

Read original(opens in new tab)

Database DesignOauthStablecoinsLink Wallet+3

gitlab·Apr 29, 2026·2 min readCurated summary

Teaching software development the easy way using GitLab

GitLab for Education can turn the administrative work of teaching software development into a scalable, professional workflow. University of Washington lecturer Stephen G. Dame uses GitLab groups, controlled permissions, merge requests, and inline comments to distribute materials, protect solutions, and provide contextual feedback. The approach helps students build real-world version-control and code-review habits while reducing instructor overhead. ## Building a Course Structure with Groups - Dame organizes the university in a root group such as `UWTeaching`, with one subgroup per course, such as `css430`. - Course subgroups contain: - Private lecture materials and code repositories - Student subgroups - Grader subgroups - Permissions inherit through the hierarchy, allowing instructors to control access centrally. - Students receive Reporter access with an expiration date tied to the academic quarter. - They can clone and pull assignment repositories but cannot push to instructor-controlled repositories. - Students use SSH keys across local machines, cloud shells, and virtual machines, then copy code into private repositories for their own version history. ## Automating Enrollment for Large Classes - Manually creating student accounts and permissions becomes impractical for large cohorts. - GitLab’s REST API can automate: - Creating personal subgroups for students - Looking up GitLab users - Assigning Reporter permissions - Setting membership expiration dates - GitLab also provides an open source class-management project with additional automation tools. ## Feedback Through Merge Requests - Students submit assignments by opening merge requests in their repositories. - Instructors immediately see a complete diff of the student’s work. - Comments can be attached directly to individual lines of code. - Inline feedback lets instructors explain both what is wrong and why, while directing students toward the next step. - Because feedback appears beside the relevant code, it is more actionable than comments on a separate document. ## Starting with GitLab for Education - The initial setup requires planning, but the workflow becomes largely self-sustaining once established. - GitLab for Education provides qualifying institutions with GitLab Ultimate features, including expanded storage, compute minutes, and merge-request capabilities. - Instructors are advised to begin with one course group, one assignment template, and a basic pipeline before expanding. A simple GitLab structure can make course administration more efficient while giving students practical experience with the collaborative development tools used in industry.

Read original(opens in new tab)

Database DesignGitlabOpen SourcePython+3

toss·Apr 27, 2026·3 min readCurated summary

Why We Adopted Post-Quantum Cryptography a Decade Before Quantum Computers Arrive

Toss Payments’ biggest legacy-overhaul challenge was not the technology itself, but improving security without disrupting tens of thousands of merchants using decades-old integrations. Because payment systems depend on outdated client environments and small businesses with limited technical resources, security upgrades had to be gradual and carefully communicated. The effort ultimately led from modernizing transport security to adopting post-quantum cryptography in 2026. ## The Challenge of Changing a Legacy Payment Network - Toss Payments supports merchants integrated with its PG system for many years, sometimes decades. - Server-side clients are harder to update than browsers, which update automatically to support new standards. - Security changes such as upgrading TLS, removing weak ciphers, or changing encryption can affect every API call, payment window, and server connection. - Many merchants are small businesses without dedicated developers, making complex security requirements difficult to understand and implement. - As a result, security is a shared responsibility: Toss Payments can strengthen its systems, but legacy merchant environments may still leave connections partially exposed. ## Why Existing Encryption Is Becoming Unsafe - Modern HTTPS and payment systems commonly rely on public-key algorithms such as RSA and ECDSA. - These algorithms are considered secure because conventional computers cannot practically factor enormous numbers or solve elliptic-curve problems. - Quantum algorithms have been mathematically shown to solve these problems efficiently once sufficiently powerful quantum computers exist. - This would make current encryption systems vulnerable, undermining decades of digital-security assumptions. ## Q-Day and “Harvest Now, Decrypt Later” - “Q-Day” refers to the point when quantum computers can break today’s widely used encryption. - Attackers can already intercept and store encrypted payment communications that they cannot currently decrypt. - Once quantum computers become practical, the stored data could be decrypted in bulk. - Payment information is especially valuable because it can remain sensitive for years; data transmitted today could be exposed in the 2030s. - The threat therefore requires action before quantum computers are fully operational. ## A Four-Year Security Upgrade Toss Payments chose a phased approach rather than replacing its security stack all at once: - **2022:** Became the first payment gateway in Korea’s PG industry to implement HTTP/3. - **2022–2025:** Removed weak TLS cipher suites. - **2022–2025:** Completed the rollout of TLS 1.3. - **April 2026:** Implemented post-quantum cryptography (PQC). Each stage balanced stronger protection against the risk of disrupting merchant payments. The gradual rollout gave merchants time to update their systems while ensuring that security improvements continued instead of being postponed indefinitely. ## Starting with HTTP/3 - HTTP/3 is a newer web-transport protocol designed to improve speed and stability, especially on unreliable networks. - It requires TLS 1.3, meaning that adopting HTTP/3 also enforces the use of a modern security protocol. - Toss Payments began with HTTP/3 because it offered both performance improvements and a relatively direct path toward stronger encryption. The broader lesson is that legacy security cannot be improved through a single disruptive upgrade. A phased migration, combined with clear communication and preparation for post-quantum cryptography, allows payment providers to raise security standards while keeping existing merchants operational.

Read original(opens in new tab)

Database DesignApi IntegrationPost Quantum CryptographyQuantum Computing+4

netflix·Apr 24, 2026·3 min readCurated summary

Scaling Camera File Processing at Netflix

Netflix built its Media Production Suite (MPS) to automate repetitive media workflows, improve consistency, and give filmmakers more time for creative work. Rather than develop an image-processing engine internally, Netflix partnered with FilmLight and integrated its FilmLight API (FLAPI) into Netflix’s cloud infrastructure. This combination provides reliable, camera-aware processing at global scale while supporting open standards, auditability, and rapid turnaround. ## Why Netflix Built MPS - Netflix productions use a wide range of cameras, formats, workflows, regions, and vendors. - File-based workflows created recurring problems: - Manual file wrangling reduced creative time. - Media handling varied between productions. - Human-driven processes were difficult to audit. - Teams repeatedly rebuilt similar workflows. - MPS aims to: - Standardize media management and movement from production through post-production. - Improve efficiency, consistency, and quality control. - Reduce errors and non-creative administrative work. ## Choosing FilmLight’s Processing Engine - Building a complete image-processing engine would require long-term collaboration with camera manufacturers and the broader industry. - Netflix needed a system that could: - Inspect, trim, and transcode camera-original files. - Preserve trusted color science and metadata. - Support many current and future camera formats. - Run within Netflix’s scalable, observable encoding infrastructure. - FilmLight’s Baselight and Daylight products already serve professional color grading, dailies, and transcoding workflows. - FLAPI allowed Netflix to use this proven processing technology as a backend API instead of duplicating it internally. ## Camera Metadata Inspection - Productions upload media with ASC Media Hash List (MHL) files to verify ingest completeness and integrity. - During the subsequent inspection phase, FLAPI: - Extracts metadata from original camera files. - Maps critical fields into Netflix’s normalized schema. - Makes the metadata searchable and reusable. - The metadata supports: - Matching footage by timing and reel name. - Automated retrieval. - Pipeline validation and troubleshooting. - Investigating why footage appears a certain way after processing. - Packaging FLAPI in Docker allows nearly identical deployments across Netflix’s cloud and global production compute environments. ## VFX Plates and Media Deliverables - MPS generates VFX plates and other outputs while preserving framing, color management, and camera-specific decoding behavior. - FLAPI is used to: - Debayer original camera files with format-appropriate parameters. - Crop and de-squeeze images according to ASC Framing Decision Lists. - Apply ACES Metadata Files for repeatable color workflows. - Produce deliverables in multiple formats. - The workflows are automated, repeatable, and auditable. - AMF files accompany OpenEXR outputs so recipients can identify which color transformations have already been applied. - Because the backend uses FilmLight technology, Netflix specialists can validate automated decisions in Baselight before production begins. ## Cloud-Native Media Processing - Traditional facilities often rely on powerful GPU systems and specialized high-performance storage. - Netflix instead designed its processing around the Cosmos compute and storage platform. - Cloud-compatible tools must: - Run as short-lived serverless functions in Linux Docker containers. - Operate effectively on CPU-only instances. - Support headless execution through Java, Python, or command-line interfaces. - Remain stateless so failed workers can be terminated and relaunched. - This model favors parallel processing across many workers rather than maximizing the power of one machine. - It improves cost and performance efficiency while maintaining production turnaround targets. - FLAPI’s API-driven, container-friendly, and low-state architecture made it straightforward for Netflix to integrate and operate reliably. Netflix’s approach demonstrates the value of combining established industry expertise with cloud-scale orchestration. By using FLAPI for specialized media processing and Cosmos for elastic execution, MPS can deliver consistent, traceable camera-file workflows without requiring Netflix to build and maintain every component itself.

Read original(opens in new tab)

Database DesignCloud ComputingDocker & ContainersImage Processing+4

toss·Apr 23, 2026·5 min readCurated summary

Introducing Toss Place's Data Bot 'PANDA': How every team member works like a data expert

PANDA, short for Place Analytics & Data, is Toss Place’s AI data-analysis assistant, designed to let employees retrieve and interpret approved data without waiting for analysts. It was created after the team found that 70% of data requests involved simple metric lookups rather than complex analysis. The project’s main conclusion is that reliable AI analytics depends less on prompting alone and more on standardized data, business definitions, controlled table selection, and iterative validation. ## Why Toss Place Built PANDA - Employees previously relied on analysts to search dashboards, write SQL, or manually investigate data requests. - PANDA provides self-service access within each employee’s security permissions. - It reduces routine extraction work for analysts, allowing them to focus on deeper analysis. - The goal is to establish a stronger culture of “data democracy,” where employees can access and use data immediately. ## Challenges with a Simple AI Chatbot Early experiments showed that asking an AI model to search all company data produced unreliable and expensive results: - Referencing thousands of tables and internal documents consumed excessive tokens. - The model sometimes selected different tables for identical questions, producing inconsistent answers. - It often misunderstood business definitions. For example, “active stores” could mean stores with completed installations or stores that had processed payments. - Inefficient SQL caused unnecessary Snowflake data scans and higher warehouse costs. ## Standardized Data Marts as a Single Source of Truth Toss Place collaborated across its Data Analysis and Data Platform teams to establish reliable standard data marts. - Core concepts, such as store information, were consolidated into standardized tables. - Naming conventions made table and column purposes easier for both people and AI to understand: - Tables follow `{mart_type}_{domain}_{subject}`, such as `fact_device_error_log`. - Columns follow `{prefix}_{entity}_{attribute}_{suffix}`, such as `is_merchant_active`. - Table and column descriptions were documented comprehensively. - The standardization effort reduced ambiguity by ensuring the same business concepts were represented consistently. ## Connecting Business Language to Data Data structures alone could not answer questions about terms such as “installed store” or “store category.” - Domain-specific terms and metric definitions were documented. - These business definitions were linked to the relevant standard data marts. - Data analysts helped reconcile differing interpretations and establish shared organizational definitions. - This gave PANDA the context needed to apply the correct business logic. ## Scoring and Ranking for Reliable Table Selection PANDA limits its search to well-managed tables and uses dbt tags to import selected metadata into a Manifest file. - Tables are ranked using: - **Similarity score:** Based on relationships between the question and table, including table-name matches and description relevance. - **Hierarchy weight:** Reflecting the reliability of the data layer. - The final score is calculated as: `similarity score × hierarchy weight` - Weights are assigned as follows: - Company-wide SSOT metrics: ×4 - Validated standard marts: ×3 - Domain analysis marts: ×2 - Raw bronze data and logs: ×1 - This improves accuracy, consistency, and trustworthiness while reducing unnecessary warehouse exploration. ## Agentic Loop for Querying and Validation Rather than expecting a correct answer in one attempt, PANDA uses an agentic loop. - It selects appropriate tools based on the question. - It explores tables, generates and executes queries, and reviews the results. - If the result appears inaccurate, it can inspect the schema again, modify the query, and retry. - If necessary, it asks the user for clarification. - This approach allows PANDA to handle exceptions dynamically instead of relying only on predefined rules. ## Answers Designed for Practical Use PANDA structures responses so users can understand and apply the results: - **Result:** The requested data or metric. - **Query criteria:** The period, filters, and aggregation method used. - **Insight:** An interpretation that can support practical decisions. This makes PANDA more than a number-retrieval chatbot; it also exposes part of the reasoning process normally provided by a data analyst. ## Adoption and User Response PANDA quickly became part of everyday work at Toss Place. - One-third of employees used it on its first day. - Half of the organization had tried it within a week. - More than 4,000 messages were exchanged during that period. - Current adoption is approximately 70%. - Employees reported feeling more comfortable asking small questions and using data while away from their desks. - Users particularly valued receiving insights alongside raw figures. - Unexpectedly, developers and even data professionals used PANDA actively, suggesting that its answers achieved a meaningful level of trust. ## Future Development PANDA was developed and launched in just one month, but the team plans further improvements. - Increase data coverage to more than 90%. - Raise answer accuracy above 97%. - Use real user questions, follow-up behavior, and abandonment patterns to identify unmet needs. - Expand beyond basic data retrieval to reduce more of the data team’s workload. PANDA’s central lesson is that effective enterprise AI does not require the most complicated technology. It requires solving a real business pain point with trustworthy data foundations, clear definitions, and a workflow that users can rely on.

Read original(opens in new tab)

Database DesignAIMachine LearningSql+4

toss·Apr 23, 2026·4 min readCurated summary

StarRocks Operations: Isolating Multi-tenant Workloads with Resource Groups

Toss adopted StarRocks as a real-time OLAP engine to consolidate service queries, analytics, validation, and dashboard workloads on one platform. As different workloads began competing within the same clusters, the key operational challenge became deciding which queries to protect during CPU contention. The article describes a gradual strategy: classify workloads, use `cpu_weight` by default, and introduce `exclusive_cpu_cores` only when stronger isolation is required. ## Why StarRocks - Toss previously relied on separate MySQL and Hadoop-based paths for serving, validation, monitoring, and analytics. - StarRocks reduced this duplication by providing: - A MySQL-compatible SQL interface - Large-scale analytical processing - Real-time service-oriented reads - Workloads eventually included: - Advertising and loan-underwriting services - Dashboards and monitoring tools - Kafka Connect ingestion - Batch jobs and backfills - Average traffic varied by cluster: - Service cluster: approximately 69 QPS over 24 hours and 87 QPS over a week - Monitoring and batch cluster: approximately 20 QPS, plus heavier batch workloads - Peak contention between different workloads mattered more than average QPS. ## Workload Classification Toss prioritized workloads in the following order: 1. Service queries 2. Server-side batch jobs 3. Large-scale ingestion and backfills 4. Monitoring and user query tools such as Grafana, Tableau, and Redash - Service queries required strict SLA protection. - Batch jobs needed to finish reliably but did not require real-time responses. - Ingestion and backfills could overwhelm the cluster and therefore needed explicit limits. - Monitoring queries received the lowest priority. ## Using `cpu_weight` for Shared Capacity - `cpu_weight` distributes CPU proportionally when workloads compete. - Higher-weight groups receive more CPU during contention. - When the cluster is idle, all groups can use available CPU regardless of weight. - Toss used this as the default mechanism for multi-tenant workload control. - Example priorities: - `service_wg`: weight 50 - `batch_wg`: weight 10 - `dashboard_wg`: weight 5 - Resource groups could also specify `mem_limit` and `concurrency_limit`. - StarRocks uses a scheduler inspired by Linux CFS, with pipeline drivers yielding in roughly 100 ms time slices. ## Using `exclusive_cpu_cores` for Strong Isolation - `exclusive_cpu_cores` reserves physical CPU cores for a resource group. - StarRocks binds worker threads to those cores using `pthread_setaffinity_np`. - The group receives separate pools for: - `DriverExecutor` - `ScanExecutor` - `ConnectorScanExecutor` - This prevents the protected workload from competing with shared thread pools. - `exclusive_cpu_cores` and `cpu_weight` cannot be used together within the same resource group, although both types can coexist in one cluster. - The setting is limited to `(0, min_be_cpu_cores - 1]`. - Because it is more rigid and consumes dedicated capacity, Toss recommends using it only when relative priority is insufficient. ## Toss Shopping Case - A cluster handled both real-time queries from `shopping_service` and heavy workloads from `commerce_batch`. - Initially, both workloads had similar priority, allowing large batch queries to degrade service latency. - First adjustment: - Increase `shopping_service`’s `cpu_weight` - Lower `commerce_batch`’s weight - This improved prioritization but did not eliminate latency spikes when heavy batch work overlapped with roughly 1,500 service requests per minute. - Second adjustment: - Place `shopping_service` in its own resource group - Assign dedicated CPU cores with `exclusive_cpu_cores` - Afterward, service latency remained stable even during heavy batch execution. - The operational approach was therefore incremental: begin with weights and escalate to dedicated cores only when necessary. ## Classifier Design and Resource Controls - Resource Groups control how resources are allocated; Classifiers determine which queries enter each group. - Classifiers can match attributes such as: - User - Role - Query type - Source IP - Database - The article recommends using stable identifiers such as `user` or `db` for reliable production behavior. - Examples include mapping service `SELECT` queries by service account and assigning server-side batch queries according to their dedicated user. - CPU isolation alone is insufficient for memory-heavy full scans or sudden spikes involving hundreds of concurrent queries, so memory and concurrency limits are also important. Toss’s practical recommendation is to start with clear workload classification and `cpu_weight`, then add memory and concurrency limits. Use `exclusive_cpu_cores` selectively for latency-sensitive workloads whose SLAs cannot be protected through proportional CPU scheduling alone.

Read original(opens in new tab)

Database DesignMysqlHadoopStarrocks+4