Security Insights needed a 10x throughput increase to scan all customers more frequently and detect risks sooner. The existing system was overwhelmed by Kafka backlogs, slow processing, database inefficiencies, and API timeouts. Cloudflare improved capacity by introducing parallel and lane-based processing, optimizing bulk database writes, and addressing regional latency between its API and database.
## Scaling Kafka Processing
- Scans are scheduled and published to Apache Kafka.
- Go-based checker services consume these messages, inspect accounts, zones, and DNS records, and send findings to an internal API.
- Kafka’s partition ordering limits each consumer group to one active consumer per partition.
- Slow messages could block all subsequent messages in the same partition.
- Adding partitions was avoided because it would increase resource usage for shared Kafka brokers.
## Introducing Parallel Processing
- Checkers were changed to consume messages in batches.
- Each message in a batch is processed concurrently in its own goroutine.
- This increased throughput without requiring additional Kafka partitions.
- The trade-offs were higher memory usage and potentially more work to repeat after a process crash.
## Separating Slow and Fast Work
- Some scans took seconds or milliseconds, while unusually large accounts or zones could take minutes or hours.
- These slow messages caused head-of-line blocking for faster work.
- Consumer groups and checkers were split into:
- A fast lane for predictable, short-running scans
- A slow lane for messages expected to require substantially more time
- Fast-lane consumers skipped slow messages, allowing normal scans to continue without delay.
## Optimizing Postgres Writes
- The API originally executed one insert/upsert transaction per insight.
- A request containing up to 500,000 insights could therefore generate hundreds of thousands of database round trips.
- Bulk insertion with `COPY` into a temporary table was tested but caused bloat in Postgres system tables.
- The final hybrid approach used:
- `UNNEST` for smaller batches
- `COPY` for batches above a configured threshold
- This delivered millisecond-level performance for small writes and completion within seconds for very large writes.
## Diagnosing API Timeouts
- Client-side timeouts increased as scan volume grew.
- Checkers sometimes spent 20–90% of their processing time waiting on a single API call.
- Throughput initially rose but then deteriorated under heavy load.
- The root cause was network latency:
- Postgres was hosted in Portland, Oregon.
- The API ran active-active in Portland and Amsterdam.
- Requests routed to Amsterdam incurred roughly 50 milliseconds of network round-trip latency.
- Amsterdam database queries held client connection-pool connections much longer—nearly three seconds on average versus about 10 milliseconds in Portland.
- The connection pool became exhausted, causing requests to wait for available connections and creating uneven Kafka lag across partitions.
Cloudflare’s results came from improving the full processing pipeline rather than relying on a single infrastructure change. Parallelize message handling, isolate slow workloads, batch database writes, and place latency-sensitive services close to their databases to achieve large throughput gains and more frequent security scanning.
Netflix built Service Topology to give engineers a real-time, unified view of dependencies across its thousands of microservices. Traditional metrics, logs, and traces provide isolated signals but do not reveal the broader service relationships needed to diagnose failures or assess blast radius. The system combines multiple dependency sources into a living map that supports fast, context-rich troubleshooting.
## The Observability Problem
- Netflix’s distributed architecture involves thousands of services and complex chains of calls for actions such as playback, authentication, recommendations, and optimization.
- During incidents, engineers need to determine:
- Which services depend on one another
- What the potential blast radius is
- Whether a failure originates locally or upstream
- Existing observability tools show symptoms, logs, or individual request paths, but not the complete steady-state topology.
- Manually combining information from different tools is slow and error-prone, especially during urgent incidents.
## Why Real-Time Service Mapping Matters
- Frequent deployments and changing traffic patterns make static architecture diagrams quickly obsolete.
- Netflix’s Live programming and advertising-supported plans increase the need for rapid diagnosis and operational awareness.
- Engineers repeatedly asked about dependencies, failures, maintenance impact, unknown metrics, and recent call-path changes.
- These recurring questions demonstrated the need for accurate, near-real-time dependency information.
## Lessons from Earlier Approaches
- Netflix evaluated vendor platforms, graph databases, and internal prototypes before developing Service Topology.
- Key lessons included:
- Dependency data must update in near real time.
- Storage and query systems must operate at Netflix’s scale.
- The solution should integrate with existing observability workflows.
- Incorrect or incomplete topology data can mislead engineers during incidents.
- No single data source captures every aspect of service relationships.
## Requirements for a Living Map
Service Topology was designed to provide:
- Real-time updates as services deploy and dependencies change
- Sub-second queries for traversing service call graphs
- Both network-level and application-level views
- Context such as health, availability tiers, ownership, and business domains
- A visual interface for engineers and programmatic APIs for automation, resilience systems, and blast-radius analysis
## Combining Multiple Sources of Truth
Netflix separates dependency information into physically distinct graphs so each layer can evolve and be queried independently. When a unified view is requested, the system traverses the layers in parallel and merges the results to maintain fast response times.
### eBPF Network Flows
- eBPF captures network activity at the kernel level, recording which services communicate over the network.
- This provides broad coverage, including services that lack application instrumentation.
- It supports both cluster-level and application-level topology.
- Its limitation is that network traffic alone does not provide application-specific context, such as the APIs or endpoints involved.
Netflix’s approach is to combine complementary perspectives rather than rely on a single imperfect dependency source, producing a more complete and actionable service map.
SilverTorch is a unified, GPU-based recommendation retrieval system designed to replace fragmented microservices with one integrated neural network. Its “Index as Model” architecture represents retrieval components—including item indices, filtering, reranking, and user modeling—as PyTorch modules. The system reportedly delivers up to 23.7× higher throughput and 20.9× better compute-cost efficiency than comparable CPU-based or traditional multi-service systems, while improving recommendation quality.
## Limits of Microservice-Based Retrieval
- Traditional retrieval pipelines use separate services for:
- Computing user embeddings
- Finding similar content
- Applying eligibility rules
- Scoring and reranking candidates
- An orchestrator coordinates these services before passing thousands of candidates to downstream ranking, all within roughly 100 milliseconds.
- This architecture creates several structural problems:
- **Data movement:** Network calls, serialization, and service coordination consume latency that could otherwise support more computation.
- **Version inconsistency:** User models, item indices, and filtering rules may be updated independently, causing mismatches between user and item representations.
- **Siloed engineering:** ML teams typically work in PyTorch while infrastructure teams work in C++, making improvements difficult to translate, test, and deploy.
- GPU optimizations such as Faiss-GPU can accelerate individual services but do not eliminate the architectural overhead or enable deep coordination between components.
## Index as Model
- SilverTorch replaces the service mesh with a single neural network.
- Its central design principle, **Index as Model**, turns traditional retrieval artifacts into model components:
- Item indices become tensors.
- Eligibility filters become operators.
- User towers, scoring layers, and rerankers become modules.
- A single request passes through the integrated model, which:
- Finds content relevant to the user’s interests
- Applies language, geography, and policy constraints
- Predicts multiple engagement outcomes
- Produces a combined score for the final candidate set
- This integration enables more complex models and larger candidate evaluations without exceeding the sub-100-millisecond latency target.
## Unified Retrieval Components
- SilverTorch incorporates multiple functional regions within one model:
- Approximate nearest-neighbor search identifies relevant items efficiently.
- Eligibility filtering removes content that cannot be shown to a user.
- Multi-task reranking predicts actions such as likes, shares, and comments.
- Composite scoring combines these predictions into a final ranking signal.
- Some components are hand-engineered, while others can be trained end-to-end through backpropagation.
- From the runtime’s perspective, every component is a standard PyTorch `nn.Module`, regardless of whether it performs search, filtering, or learned prediction.
## Pure PyTorch Implementation
- SilverTorch reimplements ANN search, Bloom-filter indexing, eligibility checks, neural reranking, and composite scoring as pure PyTorch modules.
- The unified design requires:
- Tensor-based data representation
- Tensor-in, tensor-out operations
- A consistent `nn.Module` interface
- This allows modules to share memory, execution graphs, and compilation steps.
- Engineers can co-design stages—for example, selecting promising clusters, filtering within them, and scoring only surviving candidates—instead of treating each operation as an isolated service.
- The approach reduces the separation between ML and infrastructure engineering, allowing both groups to work within the same programmable layer.
## Performance and Scale
- In an 80-million-item end-to-end evaluation, SilverTorch achieved:
- **23.7× higher requests per second** than a strong traditional multi-service baseline using the same model architecture.
- **20.9× better estimated total-cost-of-ownership efficiency** than a CPU-based solution.
- The system is intended to support retrieval across multiple applications and large-scale feeds and video products.
- Its increased efficiency makes neural reranking and multi-task engagement scoring practical within strict production latency budgets.
SilverTorch’s main recommendation is architectural: consolidate retrieval into a single, composable model rather than optimizing disconnected services. Representing every retrieval stage as a PyTorch module can reduce overhead, improve consistency, enable deeper cross-stage optimization, and make more sophisticated recommendations feasible at scale.
GitLab Duo Agent Platform can automate the complex, repetitive work of onboarding a microservice into an established GitOps workflow. By analyzing an application’s repositories and configuration, a custom agent can generate manifests, update pipelines, configure image automation, and follow organization-specific conventions. The approach combines AI-driven speed with GitLab-managed versioning, governance, and enterprise security.
## TanukiBank’s GitOps Use Case
- The fictional TanukiBank application needs a new `intra-account-transfers` microservice for its Quick Transfer feature.
- Its deployment architecture includes:
- Individual service projects with container registries and build pipelines.
- **Tanuki Bank - Delivery**, which stores deployment manifests and delivery pipelines.
- **Flux Config**, which contains Flux manifests for Kubernetes.
- Flux Image Automation watches service registries and updates corresponding delivery manifests.
- A delivery pipeline then builds and signs the image, while Flux CD synchronizes it to the Kubernetes cluster.
- Adding a service manually requires coordinated changes across all these components.
## Generating the Custom Agent’s System Prompt
- GitLab Duo Agentic Chat examines the TanukiBank group, subgroups, source files, Dockerfiles, manifests, configuration, and dependencies.
- It generates a detailed system prompt describing:
- The existing GitOps workflow.
- Required operating rules.
- Reporting instructions.
- Recommended tools.
- The prompt is specific to the workflow at the time it is generated.
- If the application’s GitOps process changes, the prompt should be regenerated.
## Creating and Configuring the Agent
- A new `application-agents` project manages custom agents, their administrators, and where they can run.
- A managed agent named **TanukiBank Microservice Onboarder** is created with:
- A description.
- The generated system prompt.
- Tools recommended by GitLab Duo.
- The agent is enabled in both **Tanuki Bank - Delivery** and **Flux Config**.
- Its presence in each project’s Agentic Chat agent selector confirms that it is available.
## Creating the Microservice
- A new `services/intra-account-transfers` project is created.
- GitLab Duo’s **Developer** foundational flow implements the service from an issue specification.
- The flow:
- Reads the requirements.
- Writes the implementation.
- Creates a branch and merge request.
- Links the merge request to the issue.
- After local verification with `curl`, the merge request is merged and the project pipeline publishes container images.
- At this stage, the service exists, but the GitOps system has not been updated:
- `manifests/dev` has no service manifests.
- The delivery pipeline does not reference the service.
- `Flux Config` lacks an `image-update-automation.yaml` entry.
## Using the Custom Onboarding Agent
- The custom agent is enabled in the new service project.
- From **Tanuki Bank - Delivery**, the user selects **TanukiBank Microservice Onboarder** in Agentic Chat and provides the service name and hostname.
- The agent begins onboarding by:
- Finding and reading the service’s Dockerfile.
- Determining the application port.
- Generating the required Kubernetes manifests.
- Updating the relevant delivery pipelines.
- This automates the coordinated repository changes normally required for a new microservice.
## Practical Takeaway
A custom GitLab Duo agent is most valuable when it is grounded in an organization’s real repositories and deployment conventions. Generate its prompt from the current system, keep the agent centrally governed, and regenerate the prompt whenever the GitOps workflow changes.
Netflix’s centralized ML serving platform provides a single, domain-independent API for model inference across personalized experiences and other use cases. Rather than exposing individual scoring functions, Netflix packages feature computation, preprocessing, inference, and postprocessing into self-contained model workflows. The core routing challenge is directing each request to the correct model version and serving cluster while keeping client services independent from model changes and infrastructure topology.
## Models as End-to-End Workflows
- Netflix distinguishes **model serving** from traditional model inference:
- Inference typically means `infer(features) -> score`.
- Serving includes preprocessing, feature computation, optional trained components, and postprocessing.
- Example workflows include:
- Ranking titles for a personalized Continue Watching row using user, country, and device context.
- Predicting payment fraud using user, country, and transaction details.
- Models declare the facts they need, while the serving platform retrieves those facts from other microservices.
- During offline training, Netflix’s ML fact store provides snapshots for bulk feature computation.
- Calling services provide standard request context and domain-specific inputs, while the platform handles feature generation, model selection, and execution.
## Platform Design Principles
- **Model innovation without client changes**
- Client applications integrate with the platform once.
- Model versions, A/B tests, additional experimental data, logging, and model selection remain hidden behind the platform API.
- **Clients decoupled from model sharding**
- Models run across multiple serving cluster shards, each with its own Virtual IP address.
- Shard assignments can change based on traffic, SLAs, model architecture, and resource availability.
- Clients should not need to track these VIP changes.
- **Flexible traffic routing**
- Routing must support A/B allocations, gradual traffic shifts, new model versions, new VIPs, and client-specific overrides.
- Safe lifecycle management requires support for shadow deployments, canaries, rollbacks, and migrations.
## Switchboard: Context-Aware Routing
- Generic API gateways and service-mesh proxies did not satisfy Netflix’s requirements.
- Netflix needed:
- Native integration with its experimentation platform.
- gRPC support.
- Routing based on rich, domain-specific request context.
- Model-specific rollout and migration controls.
- Netflix built **Switchboard**, a custom proxy layer handling more than one million requests per second.
- Switchboard is the mandatory entry point for clients and:
- Routes requests to the appropriate model based on request context.
- Applies configured context enrichment before invoking the model.
- Hides model locations and infrastructure changes from client services.
## Objective Abstraction
- Every request must provide an **Objective**, an enumeration defined by the serving platform.
- The excerpt introduces Objectives as a central abstraction for identifying the business purpose of a serving request, but the supplied text ends before describing its full roles.
Netflix’s approach is to centralize routing, experimentation, and model execution behind one stable API. This allows client applications to evolve independently while researchers can iterate on models and safely manage large-scale production rollouts.
Cloudflare argues that AI agents require a fundamental shift in Internet and cloud infrastructure. Unlike traditional one-to-many applications, agents create unique, ephemeral execution environments for individual users and tasks, making current container-based economics and scaling inadequate. The company positions lightweight V8 isolates, alongside containers and browser support, as the foundation for making agents practical at global scale.
## The Internet Was Built for Applications, Not Agents
- Cloud infrastructure evolved during the smartphone era to serve many users through a finite number of application instances.
- Microservices, containers, Kubernetes, load balancing, and replication all support this one-to-many model.
- Agents differ because an LLM dynamically determines code paths, tool usage, and task duration.
## One User, One Agent, One Task
- Each agent may need its own execution environment, filesystem, tools, and state.
- Coding agents currently use containers with access to Git, Bash, filesystems, and arbitrary binaries.
- As agents spread to assistants, analysts, customer service, and planning tasks, the number of simultaneous environments could grow dramatically.
## The Scale Challenge
- If 100 million US knowledge workers used agents at 15% concurrency, infrastructure would need about 24 million simultaneous sessions.
- At 25–50 users per CPU, that implies roughly 500,000 to 1 million server CPUs in the US alone.
- Multiple agents per person and global adoption would increase demand by orders of magnitude.
## Isolates as Agent Infrastructure
- Cloudflare’s Workers platform uses V8 isolates instead of containers.
- Isolates start in milliseconds, use only a few megabytes of memory, and provide secure sandboxing.
- They can be up to 100 times faster to start and up to 100 times more memory-efficient than containers.
- Dynamic Workers can create execution environments on demand, run code, and discard them at a scale of millions per second.
- This efficiency could make one-agent-per-user economics viable beyond expensive coding assistants.
## The “Horseless Carriage” Phase
- Early agent infrastructure often adapts existing systems instead of using designs built specifically for agents.
- Agents use headless browsers to navigate human-oriented websites, though structured protocols such as MCP could provide direct service access.
- Many MCP servers simply wrap REST APIs, despite LLMs often being better at writing and executing code than making long sequences of tool calls.
- CAPTCHAs and behavioral fingerprinting ask whether a requester is human, while agent systems need identity, authorization, and permission controls.
- Full containers are frequently used for tasks that require only a few API calls and a response.
## Supporting Both Old and New Models
- Infrastructure transitions rarely happen all at once; technologies such as IPv4/IPv6, HTTP/2/HTTP/3, and TLS 1.2/1.3 coexist.
- Cloudflare plans to support existing agent workloads while developing more efficient primitives.
- Containers remain important for coding agents that need filesystems, Git, Bash, and arbitrary binaries.
- Cloudflare is also expanding container-based sandbox environments and browser-rendering capabilities for services that do not yet support agent-native protocols.
Cloudflare’s broader recommendation is to build infrastructure that can serve today’s container-based agents while moving toward lightweight, ephemeral isolates designed for billions of specialized agent sessions.
GitLab’s pipeline model addresses complex CI/CD needs by combining composable features rather than relying on a single linear workflow. Parent-child pipelines, DAG execution, and multi-project triggers help teams scale monorepos and coordinate services across repositories while preserving clear ownership and failure visibility. The article argues that these patterns make pipelines both faster and easier to maintain.
## Monorepos: Parent-child pipelines and DAG execution
- A monorepo containing frontend, backend, and documentation projects should not rebuild everything for every change.
- Parent pipelines can trigger child pipelines for individual services using `trigger: include`.
- Multiple included files are merged into one child pipeline, allowing jobs across files to share context and reference one another with `needs:`.
- `strategy: depend` makes the parent wait for child pipelines and report one overall success or failure while retaining detailed drill-down.
- Each service can own its pipeline configuration, reducing the risk that changes in one service break another.
- DAG execution with `needs:` allows dependent jobs to start as soon as their prerequisites finish instead of waiting for an entire stage.
- For example, API tests can begin immediately after the API build completes, without waiting for unrelated jobs.
## Microservices: Cross-repository pipelines
- When frontend and backend services live in separate repositories, independent pipelines may miss integration failures.
- GitLab multi-project pipelines allow one repository to trigger and await a pipeline in another project.
- The frontend can generate an API contract artifact, publish it, and trigger the backend pipeline with `strategy: depend`.
- The backend downloads the artifact through the GitLab Jobs API using `CI_JOB_TOKEN`.
- An integration test can reject breaking API changes and propagate the failure back to the frontend pipeline.
- The backend job uses `CI_PIPELINE_SOURCE == "pipeline"` so the contract validation runs only when initiated by the frontend, not during ordinary backend pushes.
- The frontend project identifier is supplied through a CI/CD variable such as `FRONTEND_PROJECT_ID`.
These patterns let teams reduce unnecessary work, preserve service-level ownership, and make cross-service compatibility checks part of the delivery process.
MessagingHub turns chat into a reusable platform rather than rebuilding it for each product domain. It separates domain-specific authentication and business context from common chat capabilities, allowing chatbot, customer-support, direct, and group conversations to share the same infrastructure. Its policy-driven design, modular architecture, and configurable metadata aim to reduce integration complexity while preserving flexibility.
## Why MessagingHub Was Introduced
- Chat requirements vary across chatbots, customer support, one-to-one conversations, and group chats.
- Building each implementation independently increases integration points, system complexity, development cost, and the impact of small changes.
- MessagingHub is designed as a domain-independent platform that can be adopted by multiple services.
- The platform focuses on chat itself while absorbing external requirements through generalized, reusable structures.
- It is currently used by a Japanese food-delivery service for users, drivers, customer-service agents, and restaurants.
## Supported Chat Types
- **Chatbots:** Delivered through a public web URL embedded in a partner service’s webview. Scenarios are created and deployed through an administrative console.
- **Inquiry chat:** A user is matched with a customer-service agent. The partner domain supplies contextual information such as user details and previous consultation history.
- The platform is also structured to support direct one-to-one and group conversations.
## Core Platform Policies
### Authentication and User Identification
- MessagingHub does not manage user accounts or domain authentication.
- Partner systems handle login, registration, permissions, and the decision of whether a user may access chat.
- After authenticating a user, the partner requests a connection token and passes it to the client.
- The client uses the token to establish a WebSocket connection; unauthenticated direct access is not allowed.
- A user is identified by a `client_id`, combining the partner domain identifier with the partner’s user identifier.
- Display names, profile images, and `pushToken` values are supplied and updated by the partner system.
### Service Contexts and Room Types
- A **service context** defines which roles may communicate, such as:
- `Driver2CS`
- `Consumer2CS`
- A **chat room type** defines the conversation structure, such as:
- `USER_DIRECT`
- `USER_GROUP`
- `INQUIRY_CHATBOT`
- `INQUIRY_CHAT`
- The combination of service context and room type controls room creation, participation, and message permissions.
### Room Lifecycle and Data Retention
- General room states progress from `WAIT` or `PENDING`, to `SERVICE`, and eventually to `DISABLE` or `BLOCK`, where sending messages is prohibited.
- Messages and potentially identifying data are encrypted at rest.
- Data can be deleted immediately when all participants leave a room.
- Partners can also configure retention periods for automatic deletion of older data.
## Modular Architecture
MessagingHub is not a monolithic chat server. Its components have clearly separated responsibilities and communicate through loosely coupled events.
- **`connection-manager`**
- Manages WebSocket connections and validates connection tokens.
- Tracks user connection status.
- Helps identify active chatbot scenario connections during `SOFT STOP` processing.
- **`chat-app`**
- Implements core chat logic, including message delivery, room creation, state transitions, and read status.
- Exposes functionality as commands that can be combined for different chat types.
- **`message-router`**
- Determines where recipients are connected.
- Routes messages from the chat server to the appropriate connection-management component.
- **`notification-app`**
- Sends push notifications when recipients are offline or the application is in the background.
- Uses partner-provided `pushToken` values and room-level notification settings.
- **`admin-hub`**
- Manages chatbot scenario editing and deployment.
- Handles agent accounts, roles, service contexts, events, webhooks, monitoring, and statistics.
## Command-Based Chat Flows
- Chat behavior is modeled as composable commands.
- Common commands provide functionality shared across chat types.
- Chatbot and inquiry-chat features add more specialized commands.
- This “building block” approach allows business requirements to be assembled without creating a separate chat implementation for every domain.
## Data Model
MessagingHub separates operational data from core chat data:
- **`chat` database:** Stores users, rooms, participants, metadata, and messages.
- **`chat_operation` database:** Stores operational and administrative information.
Important entities include:
- `chat_user`: Uniquely identifies users by `client_id`.
- `chat_room`: Represents rooms and enforces room uniqueness at the schema level.
- `chat_member`: Connects users to rooms.
- `chat_room_meta`: Stores participant-specific state, including read position, push settings, input restrictions, and room status.
- `chat_log`: Stores encrypted messages in a one-to-many relationship with rooms.
- `prev_chat_log_id` preserves message ordering.
- Room-level first and last message IDs, together with participant read positions, support unread-count calculation.
- Partner metadata such as `system_data`, `search_data`, `user_details`, and `descriptions` is stored as JSON. MessagingHub preserves and forwards it without interpreting its domain meaning.
- Scheduling, event, and webhook history are tracked through tables such as `chat_schedule`, `chat_event_record`, and `webhook_event_record`.
- `service_context`, `chat_event`, and `webhook` configure allowed role relationships, event-message policies, and webhook behavior.
## Chatbot Scenario Management
### Flexible Scenario Structure
- Administrators manage multiple chatbot scenarios through an editing tool.
- Scenarios define messages, selectable options, and answers.
- Webhooks can dynamically generate response content.
- The hierarchical data model supports a broad range of chatbot flows.
### Version Deployment and `SOFT STOP`
Chatbot scenarios transition through:
`WAIT → SERVICE → SOFT STOP → DISABLE`
- A newly deployed scenario becomes `SERVICE`.
- The previous scenario moves to `SOFT STOP`.
- Existing users can finish conversations using the previous version.
- New users are directed to the latest scenario.
- A scheduler periodically checks whether any users still have active connections to the old scenario.
- Connection information is collected from connection-management servers and stored in a shared resource.
- Once no active users remain, the old scenario is disabled and the scheduler stops.
- This provides backward compatibility without disrupting users during deployment.
## Inquiry Chat Metadata and Lifecycle
### Partner-Defined Metadata
Inquiry chat allows partner domains to provide information that helps agents handle cases effectively:
- Search data for finding conversations
- User details shown to agents
- Custom display data
- Event data for surveys or webhooks
- Basic consultation descriptions
- Room settings such as room names and push-notification titles
- Tracking data for identifying and mapping rooms in partner systems
### Room Lifecycle
Inquiry rooms generally move through:
`PENDING → SERVICE → DISABLE → BLOCK`
- `PENDING` represents the period while the user waits for an agent match.
- `SERVICE` is the active consultation period.
- `DISABLE` indicates that the consultation has ended.
- `BLOCK` prevents further messaging after closure.
MessagingHub’s overall approach is to keep the platform’s responsibilities narrow and reusable while allowing partner domains to own authentication, user meaning, and business-specific metadata. For organizations supporting multiple chat scenarios, a policy-driven, command-based platform with separated components and explicit data ownership can significantly reduce duplication and integration risk.
Amazon S3 began in 2006 as a simple web service for storing and retrieving objects, but its emphasis on security, durability, availability, performance, and elasticity enabled it to become foundational infrastructure. Over two decades, it scaled from roughly one petabyte to hundreds of exabytes while preserving API compatibility, reducing prices, and expanding beyond object storage. Amazon’s long-term vision is for S3 to serve as a universal foundation for data, analytics, and AI workloads.
## The Original S3 Philosophy
- S3 introduced two basic operations:
- `PUT` to store an object
- `GET` to retrieve it
- The service abstracted away complex infrastructure so developers could focus on applications.
- Its five enduring design principles are:
- **Security:** Data is protected by default.
- **Durability:** Designed for 11 nines of durability, with a lossless operating model.
- **Availability:** Failure is assumed and handled throughout the system.
- **Performance:** Storage capacity can grow without degrading performance.
- **Elasticity:** Capacity expands and contracts automatically.
## From One Petabyte to Hundreds of Exabytes
- At launch, S3 had approximately:
- One petabyte of capacity
- 400 storage nodes across 15 racks and three data centers
- 15 Gbps of bandwidth
- A maximum object size of 5 GB
- A price of $0.15 per GB
- Today, S3:
- Stores more than 500 trillion objects.
- Serves over 200 million requests per second.
- Operates across 123 Availability Zones in 39 AWS Regions.
- Supports objects up to 50 TB—10,000 times larger than the original limit.
- Storage prices have fallen by roughly 85%, to slightly above 2 cents per GB.
- S3 Intelligent-Tiering has saved customers more than $6 billion in storage costs.
- The S3 API has become an industry standard, with many other storage systems offering compatible interfaces.
## Backward Compatibility and Long-Term Reliability
- Code written against S3 in 2006 still works without modification.
- AWS has repeatedly replaced disks, storage systems, and request-processing code while preserving access to older data.
- This compatibility reflects S3’s goal of remaining infrastructure that “just works” despite continuous internal change.
## Engineering for Durability and Scale
- Microservices continuously inspect every byte across the fleet.
- Auditor services detect degradation and automatically trigger repair and re-replication.
- Automated formal methods mathematically verify correctness in areas such as:
- The index subsystem
- Cross-Region replication
- Access policies
- AWS has progressively rewritten performance-critical components in Rust over the past eight years.
- Rust improves performance while preventing memory-safety bugs and other classes of errors at compile time.
- S3 follows the principle that scale should improve the service: larger, more distributed workloads become increasingly decorrelated, improving reliability for all customers.
## S3 as a Foundation for Data and AI
Amazon’s future vision is for customers to store data once in S3 and work with it directly, avoiding costly copies and specialized systems.
- **S3 Tables** provides managed Apache Iceberg tables with automated maintenance to improve query performance and reduce storage costs.
- **S3 Vectors** supports semantic search and retrieval-augmented generation, with up to 2 billion vectors per index and sub-100 ms query latency.
- Within five months of launch, customers created over 250,000 indexes, ingested more than 40 billion vectors, and executed over 1 billion queries.
- **S3 Metadata** enables centralized, faster data discovery without recursively listing large buckets.
These additions extend S3 from inexpensive object storage into a broader platform for analytics, search, and AI while retaining its scale and cost advantages.
Cloudflare’s revamped Security Overview dashboard is designed to turn overwhelming security data into prioritized, actionable work. It combines ranked Security Action Items, security-tool status, and deep links into Security Analytics so teams can identify and investigate risks without switching between tools. Behind it is a checker-based system that processes more than 10 million insights daily through scheduled scans and real-time event handlers.
## From Visibility to Action
- Security Action Items focus analysts on what needs to be fixed now rather than displaying every available event.
- Issues are ranked by severity:
- **Critical:** Immediate risks that could be exploited.
- **Moderate:** Issues requiring attention to maintain security posture.
- **Low:** Hardening recommendations and best-practice improvements.
- Analysts can filter items by insight type, including suspicious activity and insecure configuration.
## Closing Configuration Gaps
- The Detection Tools module shows whether Cloudflare protections are actively operating.
- It highlights issues such as:
- Security tools running in “Log Only” mode instead of blocking threats.
- Shadow API discovery being disabled or unavailable.
- This shifts the focus from whether a security feature exists to whether it is correctly configured and protecting traffic.
## Connected Investigation Workflows
- Suspicious Activity cards appear both on the Security Overview and Security Analytics pages.
- Selecting a card deep-links into Analytics with relevant filters already applied.
- This removes repetitive navigation and filter recreation, helping teams investigate incidents faster.
## Checker-Based Insight Generation
- Cloudflare generates and refreshes more than 10 million actionable insights each day.
- Specialized microservices called **checkers** handle different areas, such as DNS, SSL certificates, and AI bot configurations.
- Checkers can scale independently and operate through:
- **Scheduled checks** for comprehensive configuration inspections.
- **Real-time listeners** that respond immediately to control-plane events.
## Scheduled Checks and Insight Lifecycles
- A scheduler distributes inspection tasks across checkers, such as scanning all DNS records for a zone.
- A checker:
- Receives a task.
- Collects relevant assets and configurations.
- Applies specialized validation rules.
- Creates an insight when a configuration fails its required threshold.
- Updates the insight timestamp if the issue persists.
- Removes the insight once the issue is fixed.
## Real-Time Ruleset Handlers
- Event handlers listen continuously for configuration changes.
- For example, when a WAF ruleset is changed, a handler can immediately detect that it is enabled only in “Log Only” mode.
- The handler determines that attacks are being recorded but not blocked, registers an insight, and displays it on the dashboard.
- Once the configuration is secured, the insight is cleared automatically.
The dashboard’s main benefit is its combination of prioritization, configuration awareness, and immediate investigation paths. By pairing scheduled validation with real-time detection, it helps security teams move from passive monitoring to faster, more proactive remediation.
Anna Sulkina’s career journey moved from hardware diagnostics and frontend development into backend infrastructure and engineering leadership. Her experiences at Twitter taught her to design distributed systems for failure and to build consensus around transformative technologies like GraphQL. She joined Airbnb in 2022 because it aligned her passion for travel with an opportunity to strengthen developer infrastructure, organizational strategy, and engineering collaboration.
## Discovering Technology in Post-Soviet Ukraine
- Sulkina grew up in Eastern Ukraine as the Soviet Union collapsed.
- Her older brother introduced her to computers by bringing home hardware components and assembling a machine that loaded programs from a cassette player.
- Seeing how individual components formed a working system inspired her to pursue technology.
## Learning English While Building Technical Skills
- She studied programming at a Ukrainian university before immigrating to the United States.
- Although she understood written English and knew how to program, communicating in English was initially more difficult than learning programming languages.
- She took ESL classes while studying C++ and Java through Berkeley Extension.
- Her first job was in hardware diagnostics at a five-person company.
- A language barrier caused her to run out of time on a technical interview, but an interviewer familiar with her Berkeley class gave her another opportunity.
- She eventually transitioned from C++ to Java, which became her primary language for many years.
## Moving Down the Stack and Into Leadership
- Sulkina’s career progressed from hardware diagnostics to frontend, backend, and infrastructure engineering.
- At the same time, she increasingly took on leadership responsibilities.
- At Caymas Systems, her manager recognized her leadership potential and showed her the difference effective leadership makes.
- At Comcast, she moved from individual contributor to engineering manager.
- Coaching engineers, building software collaboratively, and developing high-performing teams convinced her that leadership was the right path.
## Lessons from Twitter’s Distributed Systems
- During nearly nine years at Twitter, Sulkina advanced from first-line manager to director.
- She worked through major operational events, including the “fail whale” period and the tweetstorm surrounding Ellen DeGeneres’s viral selfie.
- Twitter’s transition from a monolith to microservices taught her that failure is inevitable in complex systems.
- Resilient distributed systems must be designed to handle failures rather than assuming failures can be prevented.
- Her cultural lesson involved turning promising ideas into adopted technologies.
- She helped bootstrap Twitter’s GraphQL API, replacing legacy REST services.
- The effort required leadership support, cross-team consensus, and stakeholder alignment, but ultimately improved product teams’ development velocity.
## Choosing Airbnb
- Airbnb contacted Sulkina in 2022, when she felt ready to move beyond a well-established organization at Twitter.
- The company appealed to her because it combined her professional interests with her personal passion for travel; she had been an Airbnb guest since 2013.
- Airbnb’s Developer Platform organization had strong work happening in separate silos but needed clearer strategy, direction, and trust across engineering.
- Sulkina began by clarifying the organization’s purpose and future direction.
- Her early priorities included strengthening the organization, coaching leaders, and creating alignment within the team and with the teams it supported.
- Over the following years, this work produced a high-performing organization with clearer strategy, stronger execution, and a focus on delivering business value.
Sulkina’s story emphasizes that technical growth, organizational leadership, and personal motivation can reinforce one another. Her experience suggests that successful engineering leaders design for failure, invest in alignment, and use clear strategy to turn fragmented efforts into meaningful platform-wide impact.
Airbnb launched more than 20 locally preferred payment methods across global markets in just over 14 months. The initiative aimed to improve checkout conversion, reach customers with limited access to cards, and provide familiar payment options. Airbnb achieved this by combining a replatformed, domain-oriented payments architecture with reusable PSP connectors and standardized payment-flow patterns.
## Why Local Payment Methods Matter
- Local payment methods (LPMs) include:
- Digital wallets such as M-Pesa and MTN MoMo
- Online bank transfers
- Real-time payment systems such as PIX and UPI
- Regional payment schemes such as EFTPOS and Cartes Bancaires
- They help Airbnb:
- Increase conversion by offering trusted local options
- Enter markets where card usage is limited
- Serve customers without credit cards or traditional banking access
- Airbnb identified more than 300 payment options worldwide.
- For the initial rollout, it evaluated the top 75 travel markets and selected one or two methods per market, producing a shortlist of just over 20 integrations.
## Payments Platform Modernization
- Airbnb separated payment capabilities from its core stays, experiences, and services businesses.
- Its Payments LTA modernization replaced a monolith with domain-oriented services.
- Core payment subdomains include:
- Pay-in and payout
- Transaction fulfillment and processing
- Wallets and payment instruments
- Ledger
- Incentives and stored value
- Issuing
- Settlement and reconciliation
- This structure improved reuse, extensibility, time to market, and team autonomy.
## Connector Architecture and Multi-Step Transactions
- The processing domain uses connector and plugin-based integrations for payment service providers (PSPs).
- Plugins support:
- API- and file-based integrations
- Payment routing and switching
- Market-specific PSP behavior
- Airbnb also introduced Multi-Step Transactions (MST), a PSP-agnostic framework for payments requiring multiple stages.
- MST represents intermediate operations as Actions, including:
- Redirects to external apps or websites
- Strong customer authentication challenges
- Payment-method-specific interactions
- PSP plugins normalize these requirements into an `ActionPayload` and return an `ACTION_REQUIRED` transaction status.
## Three Standardized LPM Flow Types
Airbnb analyzed its payment methods and grouped them into three reusable archetypes:
- **Redirect flow:** The guest is sent to an external site or app, then returned to Airbnb. Examples include Naver Pay, GoPay, and FPX.
- **Async flow:** The guest completes payment later through a QR code, push notification, or wallet app, while Airbnb receives confirmation through a webhook. Examples include Pix, MB Way, and Blik.
- **Direct flow:** Payment credentials are entered within Airbnb and processed immediately, similar to card payments. Examples include Cartes Bancaires and Apple Pay.
This classification reduced duplicate engineering work and made new integrations more predictable.
## Orchestrating External Payment Actions
- For redirect payments:
- Airbnb sends a charge request to the local vendor.
- The vendor returns a `redirectUrl`.
- The guest completes payment externally.
- Airbnb receives a result token and uses it to confirm the transaction securely.
- For asynchronous payments:
- Airbnb sends a charge request and receives `qrCodeData`.
- The checkout displays the QR code.
- The guest pays in an external wallet.
- The vendor sends a webhook, allowing Airbnb to mark the payment successful and confirm the order.
- These flows required careful handling of app switching, session handoff, delayed confirmation, and synchronization between Airbnb and external providers.
## Outcome
Airbnb’s rollout demonstrates that broad local-payment coverage depends less on building every integration independently and more on creating reusable abstractions. A modular payments platform, standardized flow archetypes, normalized PSP actions, and plugin-based connectors enabled the company to support diverse regional payment behaviors at global scale.
Netflix has significantly enhanced the reliability of its global continuous delivery platform, Spinnaker, by adopting Temporal for durable execution of cloud operations. By migrating away from a fragile, polling-based orchestration model between its internal services, the engineering team successfully reduced transient deployment failures from 4% to a remarkable 0.0001%. This shift has allowed developers to write complex, long-running operational logic as standard code while the underlying platform handles state persistence and fault recovery.
### Limitations of Legacy Orchestration
* **The Polling Bottleneck:** Originally, Netflix's orchestration engine (Orca) communicated with its cloud interface (Clouddriver) via a synchronous POST request followed by continuous polling of a GET endpoint to track task status.
* **State Fragility:** Clouddriver utilized an internal orchestration engine that relied on in-memory state or volatile Redis storage, meaning if a Clouddriver instance crashed mid-operation, the deployment state was often lost, leading to "zombie" tasks or failed deployments.
* **Manual Error Handling:** Developers had to manually implement complex retry logic, exponential backoffs, and state checkpointing for every cloud operation, which was both error-prone and difficult to maintain.
### Transitioning to Durable Execution with Temporal
* **Abstraction of Failures:** Temporal provides a "Durable Execution" platform where the state of a workflow—including local variables and thread stacks—is automatically persisted. This allows code to run "as if failures don’t exist," as the system can resume exactly where it left off after a process crash or network interruption.
* **Workflows and Activities:** Netflix re-architected cloud operations into Temporal Workflows (orchestration logic) and Activities (idempotent units of work like calling an AWS API). This separation ensures that the orchestration logic remains deterministic while external side effects are handled reliably.
* **Eliminating Polling:** By using Temporal’s signaling and long-running execution capabilities, Netflix moved away from the heavy overhead of thousands of services polling for status updates, replacing them with a push-based, event-driven model.
### Impact on Cloud Operations
* **Dramatic Reliability Gains:** The most significant outcome was the near-elimination of transient failures, moving from a 4% failure rate to 0.0001%, ensuring that critical updates to the Open Connect CDN and Live streaming infrastructure are executed with high confidence.
* **Developer Productivity:** Using Temporal’s SDKs, Netflix engineers can now write standard Java or Go code to define complex deployment strategies (like canary releases or blue-green deployments) without building custom state machines or management layers.
* **Operational Visibility:** Temporal provides a native UI and history audit log for every workflow, giving operators deep visibility into exactly which step of a deployment failed and why, along with the ability to retry specific failed steps manually if necessary.
For organizations managing complex, distributed cloud infrastructure, adopting a durable execution framework like Temporal is highly recommended. It moves the burden of state management and fault tolerance from the application layer to the platform, allowing engineers to focus on business logic rather than the mechanics of distributed systems failure.
The Netflix Live Origin is a specialized, multi-tenant microservice designed to bridge the gap between cloud-based live streaming pipelines and the Open Connect content delivery network. By operating as an intelligent broker, it manages content selection across redundant regional pipelines to ensure that only valid, high-quality segments are distributed to client devices. This architecture allows Netflix to achieve high resilience and stream integrity through server-side failover and deterministic segment selection.
### Multi-Pipeline and Multi-Region Awareness
* The origin server mitigates common live streaming defects, such as missing segments, timing discontinuities, and short segments containing missing video or audio samples.
* It leverages independent, redundant streaming pipelines across different AWS regions to ensure high availability; if one pipeline fails or produces a defective segment, the origin selects a valid candidate from an alternate path.
* Implementation of epoch locking at the cloud encoder level allows the origin to interchangeably select segments from various pipelines.
* The system uses lightweight media inspection at the packager level to generate metadata, which the origin then uses to perform deterministic candidate selection.
### Stream Distribution and Protocol Integration
* The service operates on AWS EC2 instances and utilizes standard HTTP protocol features for communication.
* Upstream packagers use HTTP PUT requests to push segments into storage at specific URLs, while the downstream Open Connect network retrieves them via GET requests.
* The architecture is optimized for a manifest design that uses segment templates and constant segment durations, which reduces the need for frequent manifest refreshes.
### Open Connect Streaming Optimization
* While Netflix’s Open Connect Appliances (OCAs) were originally optimized for VOD, the Live Origin extends nginx proxy-caching functionality to meet live-specific requirements.
* OCAs are provided with Live Event Configuration data, including Availability Start Times and initial segment numbers, to determine the legitimate range of segments for an event.
* This predictive modeling allows the CDN to reject requests for objects outside the valid range immediately, reducing unnecessary traffic and load on the origin.
By decoupling the live streaming pipeline from the distribution network through this specialized origin layer, Netflix can maintain a high level of fault tolerance and stream stability. This approach minimizes client-side complexity by handling failovers and segment selection on the server side, ensuring a seamless experience for viewers of live events.
Security platform engineer Jung-woo Kim details his transition from a specialized Athenz developer to a "Kubestronaut," a prestigious CNCF designation awarded to those who master the entire Kubernetes ecosystem. By systematically obtaining five distinct certifications, he argues that deep, practical knowledge of container orchestration is essential for building secure, scalable access control systems in private cloud environments. His journey demonstrates that moving beyond application-level expertise to master cluster administration and security directly improves architectural design and operational troubleshooting.
## The Kubestronaut Framework
* The title is awarded by the Cloud Native Computing Foundation (CNCF) to individuals who pass five specific certification exams: CKA, CKAD, CKS, KCNA, and KCSA.
* The CKA (Administrator), CKAD (Application Developer), and CKS (Security Specialist) exams are performance-based, requiring candidates to solve real-world technical problems in a live terminal environment rather than answering multiple-choice questions.
* Success in these exams demands a combination of deep technical knowledge, speed, and accuracy, as practitioners must configure clusters and resolve failures under strict time constraints.
* The remaining Associate-level exams (KCNA and KCSA) provide a theoretical foundation in cloud-native security and ecosystem standards.
## A Progressive Path to Technical Mastery
* **CKAD (Application Developer):** The initial focus was on mastering the deployment of Athenz—an open-source auth system—ensuring it runs efficiently from a developer's perspective. Preparation involved rigorous use of tools like killer.sh to simulate high-pressure environments.
* **CKA (Administrator):** To manage multi-cluster environments and understand the underlying components that make Kubernetes function, the author moved to the administrator level, gaining insight into how various services interact within the cluster.
* **CKS (Security Specialist):** Given his background in security, this was the most critical and difficult stage, focusing on cluster hardening, vulnerability analysis, and implementing strict network policies to ensure the entire infrastructure remains resilient.
## Organizational Impact and Open Source Governance
* Obtaining these certifications provided a clearer understanding of open-source governance, specifically how Special Interest Groups (SIGs) and pull request (PR) workflows drive massive projects like Kubernetes.
* This technical depth was applied to a high-stakes project providing Athenz services in a Bare Metal as a Service (BMaaS) environment, allowing for more stable and efficient architecture design.
* The learning process was supported by corporate initiatives, including access to Udemy Business for technical training and a hybrid work culture that allowed for consistent, early-morning study habits.
To achieve expert-level proficiency in complex systems like Kubernetes, engineers should adopt the "Ubo-cheonri" philosophy—making slow but steady progress. Starting with even one minute of study or a single GitHub commit per day can eventually lead to mastering the highest levels of cloud-native architecture. For those managing enterprise-grade infrastructure, pursuing the Kubestronaut path is highly recommended as it transforms theoretical knowledge into a broad, practical vision for system design.
