AI Gateway and Workers AI are converging into a unified control plane for accessing models across Cloudflare and external providers. A single Workers binding or REST API can now provide inference, observability, logging, security, and billing without requiring users to choose a product upfront. Cloudflare’s longer-term goal is model-first routing, where applications request capabilities or models while the gateway handles provider selection, failover, and load balancing.
## Unified Bindings and API
- The Workers AI binding and AI Gateway now share the same entrypoint.
- Requests can use the built-in `default` gateway or a named gateway for separate applications and customized policies.
- The unified REST API routes requests through `/ai/` endpoints, using the `cf-aig-gateway-id` header.
- This removes the need to decide between Workers AI and AI Gateway before building an application.
## Automatic Observability for Workers AI
- Passing `default` as the gateway ID automatically creates an AI Gateway on the first authenticated request.
- Requests receive built-in:
- Full request and response logging
- Token tracking by model
- Cost attribution
- Latency and error metrics
- Developers can begin with the default gateway and later switch to a named gateway for features such as custom caching or application-specific traffic separation.
- The AI Gateway dashboard provides detailed visibility into prompts, responses, latency, token usage, and failures.
## Unified Billing with AI Gateway Credits
- AI Gateway credits can now pay for Workers AI usage in addition to providers such as OpenAI and Anthropic.
- Users can maintain one prepaid credit balance across supported providers.
- Workers AI users who use unified billing receive elevated rate limits, subject to current Cloudflare policies and documentation.
## Model-First Routing
- Cloudflare plans to route requests based on the desired model rather than requiring users to select a specific provider.
- The gateway could handle:
- Provider selection
- Failover
- Load balancing
- Capacity management
- For example, a request for a model such as Kimi K2.7 Code could be served by Workers AI, the model’s original provider, or another vetted provider hosting the same weights.
- Applications could remain available if one provider is overloaded or unavailable.
- Users will still be able to restrict traffic to a single provider when necessary.
- Routing is intended to preserve requirements such as Zero Data Retention and maintain model quality.
Cloudflare recommends using the unified binding or REST API with the default gateway to gain observability and centralized billing immediately. As model-first routing develops, applications can rely less on provider-specific infrastructure and gain greater resilience through automatic provider management.
Cloudflare now enables `wrangler dev` and `vite dev` to automatically capture OpenTelemetry traces for local Worker requests. Coding agents can discover the Local Explorer API, query traces and logs, inspect local resources, and debug failures without extra SDKs, configuration, or temporary logging. This lets agents diagnose and verify fixes locally before deployment.
## Automatic Tracing for Local Development
- Wrangler and the Cloudflare Vite plugin capture traces for local Worker invocations.
- Tracing requires no application code changes, SDK installation, or observability setup.
- Instrumentation covers:
- Outbound `fetch` requests
- KV, R2, D1, Durable Objects, Queues, and other bindings
- Fetch, scheduled, and queue handlers
- Custom application spans
- Miniflare collects runtime events and console output, then stores correlated OpenTelemetry traces and logs in a local SQLite-backed Durable Object.
## Agents Discover the Local Explorer API
- When a supported coding-agent session is detected, the development server displays the Local Explorer API URL and trace-query endpoint.
- The API exposes an OpenAPI schema, allowing agents to discover available operations dynamically.
- Agents can query read-only traces and logs using SQL, then inspect or modify local Worker state and bindings.
- Local resources available for inspection include D1, KV, R2, Durable Objects, and Workflows.
## Diagnosing and Verifying Failures
- In an example `POST /api/orders` request:
- KV successfully retrieves the active cart.
- A D1 insert fails because the `delivery_window` column is missing.
- The Queue is never called.
- Without traces, an agent must add logs around each operation and repeatedly reproduce the request.
- With traces, it immediately identifies the failed D1 operation, checks the local schema, applies the existing migration, reruns the request, and confirms success through a new trace.
- The entire debugging cycle happens locally, without deployment or temporary instrumentation.
## Local Explorer for Human Developers
- The browser-based Local Explorer displays the same telemetry available to agents.
- Developers can inspect request spans, timing, attributes, errors, and correlated console logs.
- It runs on the same localhost origin as the Worker.
- Open it by pressing `e` in Wrangler or visiting `/cdn-cgi/explorer`.
## Getting Started
- Update the relevant dependency:
- `wrangler@latest`
- `@cloudflare/vite-plugin@latest`
- Continue asking agents to debug Workers locally as usual; trace access is provided automatically.
Cloudflare’s recommendation is to use local tracing as part of the normal agent-driven development loop, giving agents structured runtime evidence to diagnose problems and validate fixes before deployment.
SRELens is a Grafana-based natural-language observability assistant created by LY Corporation’s Home SRE team. It connects metrics, logs, traces, and profiles so engineers can investigate incidents without switching between tools or manually transferring context. The project’s central conclusion is that production reliability depends less on natural-language querying itself and more on controlling the LLM’s tools, prompts, permissions, cost, and failure behavior through backend code and policy.
## The Observability Analysis Problem
- Incident investigation traditionally requires moving among:
- Grafana or IMON for metrics
- LaaS or IU for logs
- IMON Trace or Tempo for traces
- A separate profiling system
- Engineers must manually connect:
- Error-rate increases
- Error messages
- Trace IDs and slow requests
- Relevant time ranges, services, and labels
- This context switching is especially costly during outages.
- The team first consolidated data with a self-hosted LGTM-P stack:
- Mimir for metrics
- Loki for logs
- Tempo for traces
- Pyroscope for profiles
- OpenTelemetry Collector as the ingestion layer
- Centralizing the data helped, but engineers still needed to know the correct datasource, labels, query syntax, and relationships between signals.
## Why an Existing Open-Source PoC Was Not Enough
The team initially evaluated an open-source Grafana LLM plugin, but identified several production limitations:
- It could not reliably propagate Grafana-authenticated user context for chat history, permissions, and usage limits.
- System prompts could not be controlled strongly enough to enforce organizational policies.
- Short tool-call limits interrupted multi-step investigations.
- Datasource-specific naming differences often produced empty results:
- Metrics might use `service_name`
- Tempo might require `resource.service.name`
- Loki might require JSON parsing or structured metadata filters
- Modifying and deploying the solution internally raised operational and licensing concerns.
The PoC showed that the key requirement was not merely asking questions in natural language, but retaining control over how the agent operates.
## SRELens Architecture
- SRELens runs as a Grafana application plugin.
- The frontend provides the chat interface.
- The backend handles:
- LLM requests
- Tool orchestration
- Prompt composition
- Usage and quota enforcement
- Observability queries are executed through an MCP gateway.
- A `CompositeClient` combines:
- Upstream FlavaMCP observability tools
- Local Grafana tools such as `find_grafana_panel` and `render_grafana_panel`
- The backend is an orchestration and policy layer, not just a proxy.
## Three-Layer System Prompt Design
### Base System Prompt
Defines organization-wide behavior and safety rules, including:
- Tool-call ordering
- Safe handling of dashboard creation, modification, and deletion
- Fallback behavior for empty results
- Re-querying with aggregation when results are truncated
- Response structure and evidence requirements
Only administrators can change this layer.
### Datasource Fragment
Encodes environment-specific operational knowledge in YAML:
- Preferred Mimir, Loki, and Tempo datasource UIDs
- Candidate service-name labels
- Loki parsing and filtering rules
This prevents the agent from wasting tool-call rounds discovering basic datasource conventions.
### User Prompt
Stores personal or team-specific context in Redis, such as:
- Owned services
- Preferred response formats
- Frequently used dashboards
User preferences are added as context but cannot override organizational safety policies.
## Backend Tool Orchestration and Guardrails
The backend exclusively assembles system prompts and runs the agent loop:
1. Send the user’s question to the LLM.
2. Execute requested MCP or local tools.
3. Return tool results to the LLM.
4. Repeat until a final answer is produced.
Safety and reliability controls include:
- A default maximum of 10 tool-call rounds
- Duplicate-call prevention using call hashes
- A default retry limit of two attempts per tool
- Per-tool result-size limits
- Trimming older tool results when the request history becomes too large
- Preserving `tool_call_id` relationships when trimming history
- Hints that encourage changing labels, time ranges, or datasources after empty results
These safeguards reduce dependence on the LLM making perfect decisions.
## Usage Limits and Degraded Operation
- Per-user daily token quotas
- Per-user requests-per-minute limits
- HTTP 429 responses after limits are exceeded
- Post-response accounting based on actual prompt and completion tokens returned by OpenAI
- Daily quota reset at midnight in the Asia/Seoul timezone
- Redis stores conversation history, user prompts, and quotas.
- If Redis is unavailable, personalization and history are reduced, but a single chat request can still proceed.
## Incident Analysis Scenario
In one beta service, SRELens was asked to investigate an error spike between 09:50 and 10:05.
- Instead of separately searching alerts, logs, and traces, the agent examined the relevant dashboard and observability data together.
- It narrowed the incident to a surge in `CopyMedia` requests.
- The analysis was intended to connect the request pattern with the underlying errors and supporting telemetry, demonstrating how SRELens can move from an aggregate error spike toward a specific API-level cause.
SRELens demonstrates that an LLM can accelerate incident analysis when it is grounded in an integrated observability stack and constrained by explicit backend policies. For production use, organizations should treat prompt control, tool orchestration, permissions, quotas, retries, and failure handling as core system components rather than leaving them entirely to the model.
GitLab used a three-person engineering pod and AI agents to migrate 121 application-level rate-limit keys into a shared `labkit-ruby` implementation. The migration succeeded because humans retained ownership of architecture, scope, rollout decisions, and final review while agents handled mechanical coding, tests, and reviews. The main lesson was that disciplined workflows and meaningful observability mattered more than the agents themselves.
## Migration Setup
- GitLab was consolidating two production rate-limiting systems:
- `Gitlab::ApplicationRateLimiter`, with 121 keys
- A separate Rack-level implementation
- The target was a single observable, testable, and consistently operated implementation in `labkit-ruby`.
- A three-person pod divided responsibilities across the monolith, the gem, architecture, and project scope.
- AI agents:
- Read project context
- Drafted specifications
- Implemented bounded changes
- Wrote tests
- Pre-reviewed merge requests
- Humans controlled scope, architecture, rollout strategy, and final approvals.
## The Specification and Review Loop
- The team followed a repeatable process:
- Read the epic
- Write a specification
- Conduct adversarial review
- Implement only after blockers were resolved
- Verify with explicit evidence
- Review the merge request adversarially
- Escalate to human review
- Merge
- Adversarial review was limited to two resolution rounds before requiring human involvement.
- The project produced 14 numbered specifications and more than 30 merge requests.
- This structured loop made agents useful on legacy code without allowing them to make high-impact decisions independently.
## Successful Rollouts
- The first cohort covered five heavily used keys, including:
- `pipelines_create`
- `notes_create`
- `user_sign_in`
- Rollout progressed from 1% to 10%, 50%, and finally 100% over two days.
- Engineers compared the old and new implementations during rollout and deliberately generated traffic to test behavior above the configured limits.
- The second cohort consolidated 95 call sites:
- 83 in the monolith
- 12 in Enterprise Edition
- Agents were especially effective at this repetitive, large-scale codebase work, avoiding roughly 95 individual feature-flag changes and 190 YAML edits.
## Observability and Shadow-Mode Failure
- During Cohort 2, an adapter dropped an identifier on an unauthenticated path by incorrectly packing three strings into two primitive slots.
- Some users briefly received generic failures when enforcement began.
- Shadow comparison had detected divergence, but the dashboards did not distinguish structural identifier collisions from ordinary disagreements.
- The team disabled enforcement immediately and shipped a short-term fix two days later.
- The deeper cleanup will replace array-based scopes with named characteristics when calling `ApplicationLimiter`.
- The incident showed that having observability is insufficient if it cannot identify the failure modes that require action.
## Missed Rate Limits and Infrastructure Constraints
- An audit revealed that the original five-cohort plan had missed 17 of the 121 keys.
- The omissions included:
- Enterprise-only limits
- Registry entries
- Webhook keys
- `partner_*` sub-second limits
- Orphaned adapter rows
- The team had not maintained a complete inventory count, making it possible for keys to become effectively invisible.
- A sixth cohort was added to cover the missed cases.
- Redis capacity also became a constraint:
- The rate-limiting service used a four-shard cluster.
- `maxclients` was increased incrementally.
- Rollout stopped at 75,000 connections rather than 100,000 because primary CPU usage approached saturation.
- Redis command execution was limited by one core per primary, leaving no simple vertical scaling solution.
## How AI Changed the Work
- Agents made code generation faster, shifting the bottleneck to:
- Human review capacity
- Rollout judgment
- Operational monitoring
- Reviewer and operator attention
- Agent collaboration was not always efficient; engineers sometimes spent longer guiding agents than they would have spent coding directly.
- Engineers also had to develop new skills for specifying, reviewing, and correcting agent-generated work.
- Agents could execute a request mechanically—such as creating dozens of feature flags—but could not decide whether that design was appropriate.
- Human judgment remained essential for simplifying the rollout and avoiding unnecessary per-key flags.
## Outcome
- By mid-June, all six cohorts had reached 100%.
- All 121 application rate-limit keys were running through the new framework.
- The migration demonstrated that AI agents can safely support complex legacy-system changes when paired with bounded tasks, adversarial review, gradual rollouts, complete inventories, and failure-specific observability.
A practical recommendation is to use agents for repetitive implementation and verification, but keep architecture, risk assessment, rollout control, and operational decisions firmly with experienced humans.
Datadog built a real-time data-completeness system to ensure that every customer’s telemetry is available for dashboards, alerts, queries, and AI-driven decisions. Because ingestion spans hundreds of distributed paths and customers may send delayed or retried data, global or watermark-based tracking is unreliable. The system instead tracks payloads segment by segment, using idempotent create and acknowledgment events to identify losses and calculate end-to-end completeness.
## Defining Completeness at Datadog’s Scale
- Completeness means every ingested payload—metrics, logs, spans, or other telemetry—is ultimately available to customers.
- The system must measure completeness:
- Across hundreds of services and ingestion paths
- For each individual customer
- In real time
- With enough detail to identify where degradation occurred
- Customer traffic may take different routes because of partitioning, isolation, and traffic patterns.
- Metrics and APM pipelines can each involve hundreds or tens of distinct paths, creating a large number of possible failure points.
- The completeness system must remain independent of the services it monitors so it can provide trustworthy diagnostics during incidents.
## Tracking Completeness by Pipeline Segment
- Datadog considered watermark-based tracking, but delayed customer data, replayed traffic, and pipeline loops made predictable watermarks impractical.
- Pipelines are divided into segments representing steps within or between services.
- For example, intake-in to intake-out is one segment.
- Intake-out to processing-in is another.
- Each segment is measured independently, allowing engineers to locate degradation within a service or between services.
- Segment-level tracking also adapts to pipelines whose branches appear or disappear over time.
## Counting Creates and Acknowledgments
- When a payload enters a segment, the system records a create event.
- When it exits, the system records an acknowledgment using the payload’s unique identifier.
- Comparing creates with acknowledgments reveals whether payloads were lost in that segment.
- Events are organized into time buckets based on when the payload first entered Datadog, using a Datadog-controlled timestamp rather than the customer’s clock.
- Each identifier has a state per segment:
- Created
- Acknowledged
- Acknowledged before the create event arrived
- Duplicate create or acknowledgment events are ignored, making the system idempotent despite retries and event reordering.
## Calculating End-to-End Completeness
- Segment completeness is the ratio of payloads exiting a segment to those entering it.
- For sequential services, overall completeness is calculated by multiplying segment ratios.
- Parallel branches require a different approach:
- Treating branches as one pipeline would make completeness wait for the slowest branch.
- Instead, Datadog uses a weighted average, giving each branch influence proportional to the volume it processes.
- In the example, one branch reaches 94% completeness by multiplying 98% and 96% across two sequential services, while another branch reaches 100%.
- Combining these branch measurements produces a more accurate view of currently available data without incorrectly marking all data incomplete because one branch is slower.
## Practical Conclusion
Segment-level, identifier-based tracking gives Datadog a real-time and customer-specific view of data completeness. It both supports reliable end-to-end calculations and helps humans or automated systems quickly determine where ingestion problems are occurring.
Discord’s API runs from a shared Python codebase with more than 1,700 endpoints and 700 background tasks across hundreds of Kubernetes deployments. While existing observability tracks performance and reliability, Discord lacked a way to understand hosting costs by product feature or endpoint. Because deployments share code and workers handle multiple features concurrently, the solution was to extend application profiling to allocate deployment costs according to the time spent serving each feature.
## A Large, Continuously Deployed API
- Discord operates a unified Python codebase containing:
- Over 1,700 API endpoints
- Around 700 background tasks
- Engineers deploy changes daily to several hundred Kubernetes deployments.
- Phased rollouts and instrumentation help monitor:
- Latency
- Throughput
- Error rates
- These metrics make it possible to detect regressions affecting users or infrastructure.
## The Missing Cost Dimension
- Discord wanted to determine how hosting costs were distributed across product features.
- Example questions included:
- How much does it cost to send and receive messages?
- What does it cost to start a stream or send a Nitro gift?
- How do feature costs change over time?
- Did a recent code change materially affect a team’s hosting spend?
- The goal was to measure costs at both:
- Individual endpoint level
- Broader feature level, such as chat
## Why Kubernetes Deployment Costs Were Insufficient
- Cloud providers can generally report costs by Kubernetes deployment.
- However, Discord’s deployments do not map cleanly to product features:
- The same codebase runs across all deployments.
- Each deployment handles a particular subset of HTTP traffic or background tasks.
- Splitting deployments further would make the system impractical to operate.
- Discord therefore needed cost attribution without changing its deployment topology.
## Allocating Costs Through Profiling
- API worker processes handle multiple tasks concurrently.
- A single worker may simultaneously perform work for many different features.
- Existing traffic isolation was not detailed enough for feature-level cost analysis.
- Discord’s approach was to allocate a deployment’s cost based on the amount of time spent executing code associated with each feature.
- By extending its application profiling tools, Discord could track this execution time and use it to estimate feature and endpoint hosting costs.
In practice, the profiling-based approach provides a way to analyze infrastructure spending within shared deployments, without requiring separate services or Kubernetes environments for every product feature.
AWS DevOps Agent’s new preview release-management features extend its role from post-deployment incident response to pre-production review and testing. It evaluates code changes against production requirements, organizational standards, dependency risks, and access-control practices, then performs targeted tests in isolated or production-like environments. The goal is to help teams safely handle the growing volume of AI-generated code without sacrificing review quality or delivery speed.
## Release Readiness Reviews
- Reviews changes for:
- Production and dependency risks
- Cross-repository impacts
- AWS access-control changes and Well-Architected best practices
- Compliance with organization-specific standards
- Teams can provide standards in plain English, such as:
- Encryption and network-access rules
- Logging and observability requirements
- Sensitive-data classification practices
- Without custom instructions, the agent applies general best practices.
- It runs lightweight user-journey tests in an AWS-managed isolated environment to confirm that the software builds, runs, and passes basic functional checks.
- Findings are available in:
- The AWS DevOps Agent console
- GitHub or GitLab pull-request comments
- IDE workflows through the Kiro power or Claude Code plugin
## Autonomous Release Testing
- Generates test plans based on the specific code change rather than relying only on static test suites.
- Tests web and API applications in customer-provisioned, production-like environments before merging.
- Covers:
- Functional correctness
- Behavioral regressions
- Integration scenarios
- Produces structured artifacts for every run, including metrics, logs, traces, and execution summaries.
## Configuring and Running Reviews
- At least one GitHub or GitLab repository must be connected to an AWS DevOps Agent Space.
- The agent indexes connected code and builds a knowledge graph of cloud and cross-repository dependencies.
- Reviews can be triggered by:
- Submitting a pull request
- Starting an on-demand chat request, such as “Perform a production risk analysis on my repository branch”
- The target can be specified using a branch name, pull-request number, or commit SHA.
- Reviews can also be initiated from supported development environments.
## Reviewing Results
- The **Changes** section lists review executions and supports filtering by category or status.
- The **Timeline** records the agent’s tools, consulted dependencies, observations, and timestamped reasoning steps.
- The **Report** includes:
- Recommended action: **BLOCK**, **Proceed with Caution**, or **Safe to Release**
- Number of critical issues
- Commit revision and changed-file count
- Evidence supporting the recommendation
- Severity-ranked findings
- Actionable remediation steps
- A file-by-file summary of modifications
- Developers can ask follow-up questions about affected downstream consumers, impacted files and line numbers, and recommended fixes.
AWS DevOps Agent’s preview release-management capabilities provide an automated layer of change analysis and targeted testing before production. Teams should configure organization-specific instructions, connect their repositories, and use the generated reports and test artifacts as an additional safety gate for AI-assisted development.
Netflix built Service Topology to give engineers a real-time, unified view of dependencies across its thousands of microservices. Traditional metrics, logs, and traces provide isolated signals but do not reveal the broader service relationships needed to diagnose failures or assess blast radius. The system combines multiple dependency sources into a living map that supports fast, context-rich troubleshooting.
## The Observability Problem
- Netflix’s distributed architecture involves thousands of services and complex chains of calls for actions such as playback, authentication, recommendations, and optimization.
- During incidents, engineers need to determine:
- Which services depend on one another
- What the potential blast radius is
- Whether a failure originates locally or upstream
- Existing observability tools show symptoms, logs, or individual request paths, but not the complete steady-state topology.
- Manually combining information from different tools is slow and error-prone, especially during urgent incidents.
## Why Real-Time Service Mapping Matters
- Frequent deployments and changing traffic patterns make static architecture diagrams quickly obsolete.
- Netflix’s Live programming and advertising-supported plans increase the need for rapid diagnosis and operational awareness.
- Engineers repeatedly asked about dependencies, failures, maintenance impact, unknown metrics, and recent call-path changes.
- These recurring questions demonstrated the need for accurate, near-real-time dependency information.
## Lessons from Earlier Approaches
- Netflix evaluated vendor platforms, graph databases, and internal prototypes before developing Service Topology.
- Key lessons included:
- Dependency data must update in near real time.
- Storage and query systems must operate at Netflix’s scale.
- The solution should integrate with existing observability workflows.
- Incorrect or incomplete topology data can mislead engineers during incidents.
- No single data source captures every aspect of service relationships.
## Requirements for a Living Map
Service Topology was designed to provide:
- Real-time updates as services deploy and dependencies change
- Sub-second queries for traversing service call graphs
- Both network-level and application-level views
- Context such as health, availability tiers, ownership, and business domains
- A visual interface for engineers and programmatic APIs for automation, resilience systems, and blast-radius analysis
## Combining Multiple Sources of Truth
Netflix separates dependency information into physically distinct graphs so each layer can evolve and be queried independently. When a unified view is requested, the system traverses the layers in parallel and merges the results to maintain fast response times.
### eBPF Network Flows
- eBPF captures network activity at the kernel level, recording which services communicate over the network.
- This provides broad coverage, including services that lack application instrumentation.
- It supports both cluster-level and application-level topology.
- Its limitation is that network traffic alone does not provide application-specific context, such as the APIs or endpoints involved.
Netflix’s approach is to combine complementary perspectives rather than rely on a single imperfect dependency source, producing a more complete and actionable service map.
Nova is Dropbox’s internal cloud platform for running coding agents across the software development lifecycle. Rather than building separate tools for coding, CI debugging, migrations, and operational tasks, Dropbox created a shared platform that supports interactive sessions and autonomous workflows within its monorepo and infrastructure. The platform grounds agent changes in real builds and tests, making AI assistance more reliable and easier to integrate into engineering workflows.
## The Case for a Shared Platform
- Engineering work includes repetitive but important tasks such as:
- Debugging CI failures
- Updating dependencies
- Improving test coverage
- Fixing flaky tests
- Managing migrations and operational work
- Different tasks require different interaction models:
- Interactive chat for developer-driven work
- Asynchronous workflows for long-running remediation and automation
- Dropbox’s environment has specialized requirements:
- A large monorepo
- Bazel for builds and tests
- Caching and remote execution
- On-premises infrastructure
- Dropbox-specific validation workflows
- Off-the-shelf coding agents were designed primarily for local development and did not naturally fit this environment.
## How Nova Runs Coding Sessions
- Each session runs in an isolated environment using a specific snapshot of the codebase.
- Callers provide:
- The repository commit
- A task description
- Optional validation commands
- Iteration limits and branch settings
- Nova can run builds and tests after an agent proposes a change.
- If validation fails, the results are sent back to the agent so it can continue troubleshooting.
- This creates a feedback loop of:
- Propose a change
- Validate it in the real environment
- Correct failures
- Repeat as needed
- Nova supports multiple coding agents behind a common interface.
- Engineers can access it through:
- A web interface
- A command-line client
- An API
- Internal scripts and services
- The platform also provides prompt evaluation, observability, feedback collection, skills, plugins, and MCP integrations for accessing systems such as logs and monitoring tools.
## Deterministic Code Publication
- Nova keeps code publication outside the agent.
- Each session is limited to a single branch.
- This makes active work and publication status predictable.
- It avoids the complexity of agents creating and managing multiple branches.
- The deterministic model simplifies automation such as:
- Running tests
- Rebasing onto the main branch
- Tracking which changes belong to each session
## Engineering Workflows Using Nova
### Developer-Driven Sessions
- Engineers use Nova’s web interface for quick fixes and prototypes without disrupting local work.
- Validation commands can use Bazel selectivity tools to target the relevant compile and test dependencies.
- Slack discussions can be carried into Nova sessions, preserving context and reducing manual setup.
### Flaky Test Remediation
- Dropbox built Deflaker, a durable workflow connected to Athena, its flaky-test detection system.
- Deflaker gathers examples of a test passing and failing.
- It sends the associated logs to Nova.
- The agent analyzes the evidence, identifies a likely cause, and proposes a fix.
- This demonstrates how Nova can combine investigation, context gathering, and code changes in a longer-running automated process.
## Practical Takeaway
Dropbox’s experience suggests that coding agents are most useful when embedded in existing engineering systems rather than treated as isolated code-generation tools. A shared platform like Nova can support many workflows while preserving consistent execution, validation, context, and observability.
Kakao’s messaging platform team uses a continuously available stress-testing environment to identify scalability limits, failure points, and recovery behavior before production incidents occur. The setup mirrors production hardware, generates realistic traffic patterns with Locust, and tests both routine and extreme scenarios. The central lesson is that performance testing must examine not only application throughput, but also observability, infrastructure, framework choices, and domain-specific traffic behavior.
## Continuous Stress-Testing Environment
- The environment has two main components:
- Target servers using the same JVM heap, CPU, memory, and network specifications as production.
- Load-generating clients built primarily with Locust, with workers scaled to hundreds of pods when necessary.
- Client capacity is deliberately oversized so that the load generators do not become the bottleneck.
- JMH may also be used for focused benchmarking.
- Traffic scenarios are maintained according to realistic production ratios rather than simply generating large volumes of identical requests.
- Typical scenarios include:
- Normal midday traffic.
- New Year’s midnight bursts, when message sending increases sharply.
- Scenarios are built from configurable settings, allowing new traffic patterns to be created without rewriting load-generation code.
## What the Team Stress-Tests
### Observability and Logging Infrastructure
- New or modified components such as Logstash, Fluent Bit, OpenTelemetry, and Vector are tested under production-like load.
- The team checks:
- Application throughput and elapsed time.
- CPU, memory, and network overhead.
- Delays in metrics collection and alerting.
- Previous increases in application load caused by metric collection intervals demonstrated why monitoring infrastructure must also be performance-tested.
### Protocol and Framework Benchmarks
- Server protocols and frameworks are benchmarked before changing business logic.
- Tests isolate I/O behavior and compare alternatives such as WebFlux or virtual threads using real worker-count changes and system metrics.
- CPU-bound work and I/O wait are increased separately to understand how each affects:
- Requests per second.
- Latency.
- CPU utilization and other system resources.
- During the C++-to-Kotlin migration, stress tests exposed system-metric differences and supported additional garbage-collection tuning.
### Operating-System and Security Changes
- Host OS migrations and the addition of antivirus, monitoring, or security agents are tested under high load.
- Stress tests have revealed issues such as slab-memory leaks and resource spikes caused by security software.
- Components that appear harmless under normal traffic can materially affect high-throughput applications.
### Domain-Specific User Scenarios
- Messaging systems have distinctive worst-case patterns, including:
- Many users writing simultaneously in one chat room.
- Midnight message bursts.
- Entering group chats with hundreds of members.
- These cases are reproduced by adjusting configurable load settings.
- New features are stress-tested to locate bottlenecks before launch.
## Interpreting Test Metrics
### Endpoint-Level Metrics
- **RPS:** Increase workers gradually to find saturation, or hold worker count constant to verify that throughput remains stable.
- Unexpectedly low saturation points or sharply fluctuating RPS indicate a problem requiring deeper investigation.
- **Latency:** P50 represents typical user experience, while P95 and P99 expose worst-case behavior.
- Sudden P95/P99 increases may indicate internal capacity limits.
- A degraded P50 can signal broader performance regression.
- **Error rate:** Analyze 5xx errors, timeouts, and business errors separately.
- 5xx responses may indicate server capacity exhaustion.
- Timeouts may result from insufficient client resources.
- 400-level errors can indicate broken test data or business logic.
- Nonlinear changes in RPS or latency, or any unexpected errors, are signals to investigate lower-level system metrics.
## Practical Recommendation
Maintain a production-like, always-available stress-testing environment with configurable realistic scenarios. Validate every major application, framework, observability, infrastructure, and feature change under both normal and worst-case traffic, then diagnose problems from endpoint metrics down through system resources.
GitLab released patch versions 18.11.2 and 18.10.5 on April 29, 2026, for Community and Enterprise Editions. The releases address an observability gap affecting disaster recovery RTO/RPO commitments for GitLab Dedicated and fix several regressions and bugs. No security fixes are included.
## Changes in GitLab 18.11.2
- Reverts the `ia-refactor-role-permission-enablement` merge.
- Adds Code Suggestions to DAP-supported features for self-hosted models.
- Preserves DAP code review access for Duo Core users.
- Clears persisted filters when loading the `/work_items` page.
- Adds a GraphQL mutation for retrying failed reassignment operations.
- Resolves Sidekiq spikes when users are banned.
- Fixes MCP OAuth discovery for installations using relative URLs.
- Adds the `*_oldest_unsynced_time` metric.
- Includes additional changes related to disaster recovery observability.
## Changes in GitLab 18.10.5
- Adds Code Suggestions support for self-hosted models through DAP.
- Updates the Duo CLI version used for remote flows.
- Skips three migrations that reference dropped tables.
- Preserves DAP code review access for Duo Core users.
- Resolves Sidekiq spikes caused when users are banned.
- Fixes missing `model_definitions` in self-hosted feature settings.
- Prevents `CreateOrUpdateDefaultTrackedContextWorker` from running on Geo secondaries.
- Adds the `*_oldest_unsynced_time` metric.
## Upgrade and migration impact
- **Single-node installations:** Expect downtime because migrations must finish before GitLab starts.
- **Multi-node installations:** Zero-downtime procedures can allow upgrades without downtime.
- **Regular migrations:** Included in version 18.10.5.
- **Post-deploy migrations:** Included in both 18.11.2 and 18.10.5.
Administrators should follow GitLab’s standard upgrade guidance for single-node systems and zero-downtime procedures for multi-node deployments before updating.
CI/CD observability is essential for improving pipeline performance at enterprise scale, particularly in self-managed GitLab environments. The post presents a containerized solution built with `gitlab-ci-pipelines-exporter`, Prometheus, Grafana, and Node Exporter to turn pipeline and infrastructure data into actionable insights. Its conclusion is that centralized dashboards help teams identify bottlenecks, plan runner capacity, and measure delivery performance.
## Defining CI/CD Performance
- Teams should first determine:
- Which metrics matter, such as pipeline duration, job success rates, queue times, and runner utilization.
- Who needs access, including developers, DevOps engineers, platform teams, and leadership.
- Which decisions the data will support, such as infrastructure investment, bottleneck remediation, and capacity planning.
## Observability Architecture
- The solution uses two exporters:
- **Pipeline Exporter:** Collects pipeline duration, job status, and deployment metrics through the GitLab API.
- **Node Exporter:** Collects host CPU, memory, and disk metrics for infrastructure correlation.
- Prometheus gathers and stores the metrics.
- Grafana provides real-time and historical dashboards.
- Dashboards are provisioned automatically through Grafana’s file-based provisioning and can be filtered by project, branch, or time range.
## Grafana Dashboards
- **Pipeline Overview:** Displays pipeline volume, success and failure rates, cancelled runs, and average duration trends.
- **Job Performance:** Shows job-duration histograms, the ten slowest jobs, and failure heatmaps by project and stage.
- **Runner & Infrastructure:** Correlates runner queue times with CPU, memory, and disk usage to support capacity planning.
- **Deployment Frequency:** Tracks deployment counts and durations by environment, supporting DORA-style delivery analysis and detection of environment drift.
## Kubernetes Deployment
- The recommended enterprise deployment runs each component as a separate workload in a dedicated `gitlab-observability` namespace.
- A Kubernetes secret stores the GitLab personal access token, which requires the `read_api` scope.
- The Pipeline Exporter runs as a Deployment with a service on port `8080`.
- Node Exporter runs as a DaemonSet so each node can expose host metrics on port `9100`.
- Prometheus and Grafana are deployed alongside the exporters and configured to scrape and visualize their metrics.
- Kubernetes deployment supports existing cluster infrastructure, secrets managers, network policies, and scalable operations.
## Prerequisites
- GitLab Self-Managed 18.1 or later.
- Kubernetes for enterprise deployments, or Docker/Podman for smaller environments and proof-of-concept testing.
- A GitLab personal access token with `read_api` permissions.
- Secure secret-management practices, preferably using external secret operators in production.
The practical recommendation is to begin with clearly defined performance questions, then deploy the exporter–Prometheus–Grafana stack in a controlled namespace. Combining pipeline data with host metrics provides the context needed to distinguish inefficient jobs from infrastructure capacity problems.
SLI/SLO adoption is not merely a matter of choosing metrics; it requires redefining how a service is understood from the user’s perspective. LINE’s SRE team applies this approach by identifying critical user journeys, measuring reliability with clear criteria, and setting realistic objectives. The resulting data helps teams balance reliability, engineering costs, feature delivery, and incident response.
## The Mindset Behind SLI/SLO
### Understanding the Service and Users
- Begin by identifying the services and features users depend on.
- Map user journeys and select critical user journeys (CUJs) based on:
- How frequently users use a feature
- Whether the feature is essential to the service
- Its relationship to business objectives
- Aligning SLOs with business goals ensures that reliability work supports organizational priorities.
### Communication and Collaboration
- SLI/SLOs should be defined and managed collaboratively rather than by a single team.
- Product or service owners define CUJs because they understand the user experience best.
- Infrastructure teams provide scalable systems for collecting and managing metrics.
- SREs build the measurement tools and processes used to monitor and improve reliability.
- Shared ownership is essential so SLOs can guide both daily operations and new feature launches.
## Implementing SLI/SLOs
### Analyze Critical User Journeys
- List the services and functions provided to users.
- Ask:
- Which features are used most often?
- Which features are indispensable?
- LINE examples include:
- Account registration
- Sending and receiving messages
- User authentication and encryption
- LINE Login
- Profile information
- The goal is not to include every feature, but to select the most important ones from the user’s perspective.
### Define Service Level Indicators
For each CUJ, determine:
- **Measurement location:** Choose the point that best represents the user experience, such as a gateway, frontend, or backend.
- **Measurement API:** Select a representative API to avoid unnecessarily complex calculations.
- **Success criteria:** Establish clear boundaries between successful and failed requests.
Common SLI criteria include:
- **Latency:** Define a percentile, such as the 99.9th percentile, and the maximum acceptable response time.
- **Success rate:** Define the required percentage of successful responses during the measurement period.
For example, a messaging service might require 99.9% of requests to complete within 500 milliseconds and 99.999% of all requests to receive successful responses. If a CUJ cannot be measured reliably or its success criteria cannot be defined clearly, it may be excluded or supported with a dedicated measurement metric.
### Set SLO Targets
- Define the reliability level the service must maintain over a specific period.
- An example target is achieving the defined latency and success-rate criteria for 99.9% of a 28-day period.
- Targets must be realistic:
- Excessively high targets increase operational and infrastructure costs.
- Excessively low targets can result in poor user experiences.
- SLOs should balance reliability requirements with available resources.
### Visualize Reliability
- Provide dashboards that allow all stakeholders to understand the current SLO status quickly.
- Show overall SLO performance and error-budget consumption, with detailed dashboards for individual CUJs.
- Keep dashboards simple and easy to scan rather than displaying excessive information.
- Use visual indicators such as:
- Green for healthy performance
- Orange for warning conditions
- Red for missed objectives
## How SLI/SLOs Are Used
### Quantifying Reliability
- Replace vague descriptions such as “the service is slow” with measurable statements.
- Teams can identify issues such as latency exceeding a 400-millisecond SLI threshold or success rates falling below 99.99%.
- Dashboards also help correlate periods of poor performance with incidents or operational changes.
### Guiding Resource Allocation
- SLOs show whether reliability targets are being met.
- Error budgets indicate how much additional failure or downtime is acceptable.
- When performance exceeds the SLO and the error budget is healthy, teams can invest more aggressively in:
- New features
- Faster release cycles
- Product experimentation
- When little error budget remains, resources can instead focus on prevention, remediation, and reliability improvements.
### Supporting On-Call Operations
- LINE uses alerts triggered by changes in error-budget status to help on-call teams recognize and respond to service issues.
- SLO reviews are also incorporated into regular meetings and preventive reliability work.
SLI/SLO implementation works best as a shared, user-focused operating model. By combining clear CUJs, measurable criteria, realistic targets, and actionable dashboards, teams can make informed decisions about when to prioritize innovation and when to prioritize stability.
The provided content does not include the blog post itself. It contains Datadog navigation links and a page title announcing that Datadog was named a Leader in the 2026 Gartner® Magic Quadrant™ for Observability Platforms, but no substantive discussion of the evaluation platform or its conclusions.
## Available Information
- Datadog’s page promotes its recognition as a Gartner Magic Quadrant Leader.
- The navigation lists products across:
- Infrastructure and application monitoring
- Logs, databases, and data observability
- Security
- Digital experience monitoring
- CI/CD and software delivery
- Incident and service management
- AI capabilities, including Bits AI Agents and Bits Investigation
- The referenced URL path suggests the intended article may concern Datadog’s “Bits AI eval platform,” but the article text is not included.
## Conclusion
Please provide the full blog post content for a meaningful section-by-section summary.
The April 6, 2026 AWS Weekly Roundup highlights the general availability of AWS DevOps Agent and AWS Security Agent, autonomous “frontier agents” designed to handle complex operational and security tasks. It also reviews AWS service lifecycle changes and summarizes notable product launches and technical updates from the previous week. The overall message is that AWS is expanding agentic automation while helping customers manage service transitions and adopt new capabilities.
## AWS DevOps Agent and Security Agent Reach GA
- **AWS DevOps Agent**
- Investigates incidents, accelerates resolution, and helps prevent recurring problems.
- Works continuously across multiple steps until an operational goal is complete.
- Customers report up to **75% lower mean time to resolution (MTTR)** and **3–5 times faster incident resolution**.
- Western Governors University reduced resolution times from hours to minutes.
- **AWS Security Agent**
- Provides continuous, context-aware penetration testing during the software development lifecycle.
- Operates similarly to a human penetration tester.
- LG CNS reported testing that was more than **50% faster**, approximately **30% less expensive**, and produced fewer false positives.
- **Deployment flexibility**
- Both agents support AWS, multicloud, and on-premises environments.
- They are intended to automate repetitive investigative and testing work while allowing teams to focus on higher-value activities.
## AWS Service Lifecycle Changes
AWS updated its Product Lifecycle Changes guidance on March 31, 2026, including migration recommendations and alternative services.
- Services with availability changes or maintenance guidance include:
- AWS App Runner
- AWS Audit Manager
- AWS CloudTrail Lake
- AWS Glue Ray jobs
- AWS IoT FleetWise
- Amazon Application Recovery Controller Readiness Check
- Amazon Comprehend features such as Topic Modeling and Prompt Safety Classification
- Amazon Rekognition streaming and batch moderation features
- Amazon SNS Message Data Protection
- Services listed as entering sunset include:
- AWS Service Management Connector
- Amazon RDS Custom for Oracle
- Amazon WorkMail
- Amazon WorkSpaces Thin Client
- **Amazon Chime SDK Proxy Sessions** is reaching sunset.
AWS recommends reviewing the relevant service documentation or contacting Support to reduce operational disruption.
## Notable AWS Launches
- Amazon ECS introduced **Managed Daemons for ECS Managed Instances**.
- The AWS Sustainability console now consolidates **Scope 1–3 emissions reporting**.
- **Amazon Bedrock AgentCore Evaluations** became generally available.
- AWS Transform added generally available automated codebase analysis.
- CloudWatch introduced OpenTelemetry Container Insights for Amazon EKS in preview.
- Amazon Lightsail added compute-optimized bundles with up to **72 vCPUs**.
- Amazon CloudFront added **SHA-256 support** for signed URLs and signed cookies.
## Additional AWS Resources
The roundup also points readers to material on:
- Architecting agentic AI applications on AWS.
- Reducing data-transfer costs with Network Load Balancers.
- Preventing hallucinations in production AI agents.
- The AWS World Sports Innovation Cup.
- Exploring AWS communities through an interactive 3D globe.
AWS also encourages readers to participate in Builder Center discussions, community events, AWS Summits, and developer-focused programs.
AWS teams should review the lifecycle notices for services they depend on, while developers and operations groups may benefit from evaluating the new agents and launches for automation, security testing, and observability improvements.
