Opentelemetry | Techlist.io

cloudflareAug 4, 20263 min readCurated summary

The Agent Development Lifecycle has arrived on Cloudflare

AI has made software implementation dramatically faster, exposing bottlenecks in testing, deployment, maintenance, and operations. Cloudflare argues that software development must evolve from the human-centered SDLC to an Agent Development Lifecycle (ADLC), where agents manage the full process rather than merely generate code. This requires software factories with APIs, observability, scalability, permissions, reproducibility, and self-improvement built in from the start. ## AI Has Overloaded the Traditional SDLC - The SDLC typically covers planning, design, implementation, testing, deployment, maintenance, and retirement. - AI has made implementation cheaper and faster, overwhelming downstream activities: - Open-source maintainers face huge volumes of issues and pull requests. - Production teams must handle software delivery at unprecedented speed. - Many organizations still use agents only for coding while humans perform validation, review, deployment, monitoring, and bug triage. ## From Software Teams to Software Factories - A software factory accepts inputs such as: - Production errors - Customer bug reports - Feature ideas - Agents then autonomously build, improve, deploy, and manage software. - The goal is to reduce human involvement in routine coordination and reserve human time for design, customer understanding, judgment, and creativity. - Cloudflare treats agents as customers and provides APIs that let them interact with its products and services across the SDLC. ## Requirements for Agent-Driven Development Software factories must redesign human-oriented processes so agents can operate safely and independently: - **Programmatic:** Every operation needs a reliable API; manual “ClickOps” cannot support agents. - **Horizontally scalable:** Each agent should receive isolated, production-like preview environments. - **Reproducible:** Systems must reproduce complex conditions, such as device, network, or geographic variations. - **Real-time and push-based:** Events should trigger agents instead of relying on humans to inspect dashboards. - **Atomic:** Changes must be independently testable, releasable, observable, and reversible. - **Permissioned:** Agents need controlled access and mechanisms to safely escalate permissions when necessary. - **Self-improving:** Agents must learn from prior work and operational experience. ## Cloudflare’s Initial ADLC Tools Cloudflare describes several projects intended to extend agents beyond code generation: - `@cloudflare/ci`: CI/CD infrastructure designed to operate across millions of repositories, self-heal, and spawn agents for complex tasks using Cloudflare Workflows. - OpenTelemetry traces in local development: Gives agents production-like observability through Wrangler and the Cloudflare Vite plugin. - Cloudflare Agents and Agent Traces: Provides tools for observing, maintaining, and improving agents. - AI-enforced engineering standards: Applies best practices across products, systems, and specifications. - An Astro software factory: Automatically triages, reproduces, verifies, and fixes GitHub issues to reduce the project’s issue backlog. ## Autonomous Software Requires Purpose-Built Infrastructure - Traditional SDLC tools and linear GitHub Actions workflows are designed around human decision-making and do not cover the complexity of autonomous software delivery. - Agents must handle subjective requirements, cross-functional dependencies, production risks, and operational feedback—not just run tests and open pull requests. - Like autonomous vehicles, software agents need specialized sensors, feedback systems, controls, and remote intervention capabilities rather than being placed in systems designed for humans. - The relevant standard is not whether an agent succeeds most of the time, but whether it can achieve the reliability and safety required for production. The practical recommendation is to treat agents as participants in the entire software lifecycle. Organizations adopting AI at scale should build the APIs, observability, isolation, permissions, event systems, and feedback loops needed for safe software factories—not simply add agents to existing human workflows.

Read original

Opentelemetry AI Agents CI/CD Cloudflare+3

cloudflareAug 4, 20262 min readCurated summary

Your agent can now debug Workers with local tracing

Cloudflare now enables `wrangler dev` and `vite dev` to automatically capture OpenTelemetry traces for local Worker requests. Coding agents can discover the Local Explorer API, query traces and logs, inspect local resources, and debug failures without extra SDKs, configuration, or temporary logging. This lets agents diagnose and verify fixes locally before deployment. ## Automatic Tracing for Local Development - Wrangler and the Cloudflare Vite plugin capture traces for local Worker invocations. - Tracing requires no application code changes, SDK installation, or observability setup. - Instrumentation covers: - Outbound `fetch` requests - KV, R2, D1, Durable Objects, Queues, and other bindings - Fetch, scheduled, and queue handlers - Custom application spans - Miniflare collects runtime events and console output, then stores correlated OpenTelemetry traces and logs in a local SQLite-backed Durable Object. ## Agents Discover the Local Explorer API - When a supported coding-agent session is detected, the development server displays the Local Explorer API URL and trace-query endpoint. - The API exposes an OpenAPI schema, allowing agents to discover available operations dynamically. - Agents can query read-only traces and logs using SQL, then inspect or modify local Worker state and bindings. - Local resources available for inspection include D1, KV, R2, Durable Objects, and Workflows. ## Diagnosing and Verifying Failures - In an example `POST /api/orders` request: - KV successfully retrieves the active cart. - A D1 insert fails because the `delivery_window` column is missing. - The Queue is never called. - Without traces, an agent must add logs around each operation and repeatedly reproduce the request. - With traces, it immediately identifies the failed D1 operation, checks the local schema, applies the existing migration, reruns the request, and confirms success through a new trace. - The entire debugging cycle happens locally, without deployment or temporary instrumentation. ## Local Explorer for Human Developers - The browser-based Local Explorer displays the same telemetry available to agents. - Developers can inspect request spans, timing, attributes, errors, and correlated console logs. - It runs on the same localhost origin as the Worker. - Open it by pressing `e` in Wrangler or visiting `/cdn-cgi/explorer`. ## Getting Started - Update the relevant dependency: - `wrangler@latest` - `@cloudflare/vite-plugin@latest` - Continue asking agents to debug Workers locally as usual; trace access is provided automatically. Cloudflare’s recommendation is to use local tracing as part of the normal agent-driven development loop, giving agents structured runtime evidence to diagnose problems and validate fixes before deployment.

Read original

Opentelemetry Observability Cloudflare Workers Durable Objects+4

cloudflareAug 4, 20263 min readCurated summary

Introducing: Cloudflare Agents

Cloudflare is introducing Agents, a unified platform for deploying, observing, and improving hosted AI agents. Its first major feature is agent tracing, which exposes model calls, tool execution, token usage, approvals, subagents, and underlying Cloudflare infrastructure in one view. The goal is to help developers diagnose agent failures, understand costs and latency, and use operational data to continuously improve agent behavior. ## Agent Tracing Adds Visibility - Traditional telemetry can show that an HTTP request succeeded while hiding agent-level failures, such as: - Choosing the wrong tool - Passing stale context to a subagent - Entering a token-consuming retry loop - Cloudflare’s agent-aware traces capture: - Agent invocations - Model calls and token usage - Tool executions and results - Approval or pause events - Supported subagent calls - These agent spans appear alongside existing Workers telemetry for fetches, KV, D1, Durable Objects, and other infrastructure. - Initial integrations support Think, Flue, and AI SDK through OpenTelemetry-compatible tooling. ## Reviewing Agents in the Cloudflare Dashboard - A new Agents view lists observed agents, traces, sessions, instances, runs, and token usage. - Developers can inspect agent behavior through: - **Session replay**, which reconstructs recorded conversations - **Trace waterfalls**, which show execution timing and nested operations ## Session Replay - The Messages tab displays: - System instructions - User messages - Model reasoning - Tool calls, arguments, and results - Final responses - Replay is based on captured data and does not re-execute the agent. - It can reveal malformed tool arguments, inappropriate tool choices, subagent handoffs, retries, and context that influenced later decisions. - Think, Flue, and AI SDK provide `storeMessages` and `storeTools` controls to determine whether message and tool payloads are recorded. - Payload capture can be disabled when data may contain personal information, secrets, or other sensitive content. ## Trace Waterfalls Connect Agent and Infrastructure Activity - Traces show how much time each part of a turn consumed and how operations relate to one another. - A parent agent can be connected to nested subagents, model calls, tools, and Cloudflare resources. - Example operations include: - A parent `TravelPlanner` invocation lasting 2.72 minutes - An `itinerary_builder` subagent using 1.83 minutes - Model calls with duration and provider-reported token usage - Tool executions - D1 queries and KV writes triggered by those tools - Nested tracing makes it possible to follow work from the original agent through delegated tasks and the infrastructure each task used. ## Enabling Agent Tracing - Enable tracing in `wrangler.jsonc`: ```json { "observability": { "traces": { "enabled": true } } } ``` - Setup then depends on the agent stack: - **Think and Flue:** Emit agent, conversation, turn, model, and tool telemetry through their tracing integrations. - **AI SDK:** Wrap the SDK with Cloudflare’s `wrapAISDK()` adapter. - **Custom harnesses:** Use Cloudflare’s custom spans API and OpenTelemetry’s Generative AI semantic conventions. ## Broader OpenTelemetry Support - Cloudflare plans to support the OpenTelemetry API directly inside Workers. - Frameworks that already emit standard Generative AI spans will eventually work in the Agents view without Cloudflare-specific adapters. - Standard agent and conversation identifiers will allow Cloudflare to group spans into agents and sessions. - This complements Cloudflare’s existing ability to export OpenTelemetry data by allowing Workers to accept standard telemetry directly. ## OpenTelemetry Export - Agent telemetry is not restricted to Cloudflare. - Traces can be exported to OTLP-compatible observability providers by configuring a destination in the Worker’s Wrangler configuration. Cloudflare’s initial Agents release focuses on making AI behavior inspectable rather than treating agents as opaque application requests. Developers should enable tracing, choose payload retention carefully for privacy, and use session replay and nested traces to identify correctness, latency, cost, and orchestration problems.

Read original

Opentelemetry Cloudflare Workers Durable Objects Wrangler+4

awsJul 27, 20263 min readCurated summary

AWS Weekly Roundup: Local Zone in Athens, Claude Opus 5 on AWS, Lambda durable execution for .NET, and more (July 27, 2026) | Amazon Web Services

The week’s AWS updates focus on bringing infrastructure closer to users, expanding AI capabilities, and improving reliability and observability. Highlights include a new Athens Local Zone, Claude Opus 5 availability, durable Lambda workflows for .NET, and unified Amazon Bedrock AgentCore telemetry. The post also points to new data, contact-center, developer-productivity, and community resources. ## AWS Local Zone in Athens - AWS launched its second EMEA Local Zone in Athens, Greece. - It supports: - Amazon EC2 C7i, M7i, and R7i instances - Amazon S3 One Zone-Infrequent Access - Amazon EBS and EBS Local Snapshots - Amazon ECS - Greek organizations can process and store data locally to help satisfy data-residency requirements. - The zone enables single-digit-millisecond latency for workloads such as gaming, media production, and financial services while still connecting to the nearest AWS Region. ## New AI and Serverless Capabilities - **Claude Opus 5 on AWS** is available through Amazon Bedrock and Claude Platform on AWS. - It provides top-tier Opus-level intelligence. - Bedrock enables zero data retention by default, supporting stricter governance requirements. - **Lambda durable execution for .NET** is now generally available. - C# developers can create long-running, multi-step workflows without custom checkpointing or external orchestration. - Workflows can pause for up to one year. - Use cases include payment pipelines, AI agents, and human approvals. ## Improved Agent Observability - Amazon Bedrock AgentCore now stores traces, prompts, inputs, outputs, and application logs in a single CloudWatch log group. - Engineers can investigate individual agent invocations without searching multiple destinations. - Agent-level access controls and customer-managed key encryption are also supported. ## Contact Center, Data, and Developer Updates - Amazon Connect added more natural agentic voice experiences across more than 50 languages. - The service now includes over 100 new voice options and improved handling of tone and sentiment. - SageMaker Unified Studio can query Amazon OpenSearch data alongside Redshift, S3, and relational database assets. - CloudWatch coding agent insights measures the organizational impact of tools such as Claude Code, Codex, and GitHub Copilot using OpenTelemetry telemetry without custom instrumentation. ## Additional Resources and Events - AWS published guidance for evaluating AI agents with Strands Agents and Bedrock AgentCore. - A new guide covers multi-region resiliency for CloudFormation custom resources. - Amazon SES introduced pricing plans intended to make large-scale email costs more predictable. - Upcoming opportunities include AWS Summits and AWS Community Day Belo Horizonte on August 22. Together, these releases suggest AWS is prioritizing lower-latency regional infrastructure, production-ready AI workflows, and better operational visibility. Builders should assess the Athens Local Zone for residency-sensitive workloads, consider durable Lambda execution for complex .NET processes, and adopt the new observability tools as agent deployments grow.

Read original

Opentelemetry AWS Amazon Bedrock Aws Lambda+4

lineMar 19, 20264 min readCurated summary

Utilizing SLI/SLO to Improve Reliability Part 1: SLI/SLO Framework and the Development Story of Service Status Check Tool LINE Status

Repeated SLI/SLO adoption revealed a common process that could be standardized across services. The team turned that process into a reusable framework and built “LINE Status,” an internal tool that automatically presents service health according to user experience rather than raw alerts. Together, these initiatives create a shared organizational language for understanding reliability and its impact on users. ## A Reusable SLI/SLO Framework After applying SLI/SLOs to several platforms and services, the SRE team identified recurring patterns independent of service type. They organized these patterns into a five-stage framework: - **Select critical user journeys (CUJs) and define SLIs** - Identify the experiences most important to users. - Define measurable SLIs that represent those experiences. - **Design instrumentation and metrics** - Build or adapt metrics suitable for each CUJ. - Use standardized naming based on Prometheus or OpenTelemetry. - **Create dashboards and recording rules** - Provide Grafana dashboards for quickly assessing SLO achievement. - Precompute complex PromQL operations to improve query performance. - **Set SLOs and alerts** - Begin with flexible targets, such as 99.9% availability over a 28-day rolling window, allowing roughly 40 minutes of downtime. - Define runbooks for responding to alerts. - Refine targets after operational data and experience accumulate. - **Establish error-budget governance** - Balance release speed against reliability. - Review objectives monthly or quarterly. - Adjust SLOs and processes as needed. The framework is currently distributed as a Confluence template containing guidance and FAQs, reducing the communication effort required from SREs during initial adoption. ## Moving from Alerts to User-Centered Service Status As more services adopted SLI/SLOs, the team wanted a consistent way to understand the health of services they did not directly operate. - The existing public LINE Status API page focused on external users and was updated manually during major incidents. - The new internal tool was intended to: - Represent the status of individual service components. - Update automatically from SLI/SLO alerts and outage data. - Show whether user experience was being affected. - Rather than simply reflecting whether an alert or outage existed, status was based on CUJ-related SLI performance and SLO achievement. - Only representative, high-value CUJs were exposed, avoiding unnecessary technical detail. ## LINE Status Architecture and Interface LINE Status was designed as more than an alert list. It collects events through webhooks, stores them in a separate database, and uses that data to track both current status and historical changes. - Technical SLI/SLO terms are translated into user-facing functions such as “Message Sending” or “Read Receipts.” - Status colors provide an immediate overview: - Green: normal - Yellow: event detected - Red: outage - The main page provides: - An overview of all services. - CUJ status within each service card. - AI-generated one-line summaries. - Service detail pages provide: - Recently affected items near the top. - Timeline-based event displays. - Monthly historical events. - The history page shows: - The scope of impact for each service during an event. - Past events organized by month. The initial implementation took about a month and was refined through colleague feedback. The author also used AI-assisted “vibe coding” for the frontend, emphasizing that clear, detailed requirements were more important than the development tool itself. ## Connecting the Framework and LINE Status Once a service adopts SLI/SLOs through the framework, it can be registered in LINE Status. This connects the definition of reliability objectives with an organization-wide view of service health. - Developers and operators can use the same CUJ-based standards. - Teams can focus on whether users are affected instead of interpreting isolated alerts. - During incidents, the tool helps identify impacted experiences quickly. - Over time, the approach may improve decision-making speed and cross-team communication. The team plans to refine CUJs, SLIs, and status-transition rules through continued operational experience. The practical goal is to make SLI/SLOs a common language for describing service health, enabling reliability practices to scale without depending heavily on individual teams or specialists.

Read original

Opentelemetry Database Design API Prometheus+3

awsDec 2, 2025Original article

Amazon CloudWatch introduces unified data management and analytics for operations, security, and compliance (opens in new tab)

Amazon CloudWatch has evolved into a unified platform for managing operational, security, and compliance log data, significantly reducing the need for redundant data stores and complex ETL pipelines. By standardizing ingestion through industry-standard formats like OCSF and OpenTelemetry, the service enables seamless cross-source analytics while lowering operational overhead and storage costs. This update allows organizations to move away from fragmented data silos toward a centralized, Iceberg-compatible architecture for deeper technical and business insights. **Data Ingestion and Schema Normalization** * Automatically collects AWS-vended logs across accounts and regions via AWS Organizations, including CloudTrail, VPC Flow Logs, WAF access logs, and Route 53 resolver logs. * Includes pre-built connectors for a wide range of third-party sources, such as endpoint security (CrowdStrike, SentinelOne), identity providers (Okta, Entra ID), and network security (Zscaler, Palo Alto Networks). * Utilizes managed Open Cybersecurity Schema Framework (OCSF) and OpenTelemetry (OTel) conversion to ensure data consistency across disparate sources. * Provides built-in processors, such as Grok for custom parsing and field-level operations, to transform and manipulate strings during the ingestion phase. **Unified Architecture and Cost Optimization** * Consolidates log management into a single service with built-in governance, eliminating the need to store and maintain duplicate copies of data across different tools. * Introduces Apache Iceberg-compatible access via Amazon S3 Tables, allowing data to be queried in place by external tools. * Removes the requirement for complex ETL pipelines by providing a unified data store that is accessible to Amazon Athena, Amazon SageMaker Unified Studio, and other Iceberg-compatible analytics engines. **Advanced Analytics and Discovery Tools** * Supports multiple query interfaces, allowing users to interact with logs using natural language, SQL, LogsQL, or PPL (Piped Processing Language). * The new "Facets" interface enables intuitive filtering by application, account, region, and log type, featuring intelligent parameter inference for cross-account queries. * Enables the correlation of operational logs with business data from third-party tools like ServiceNow CMDB or GitHub to provide a more comprehensive view of organizational health. Organizations should leverage these unified management features to consolidate their security and operational monitoring into a single source of truth. By adopting OCSF normalization and the new S3 Tables integration, teams can reduce the technical debt associated with managing multiple log silos while improving their ability to run cross-functional analytics.

Opentelemetry Log Management Amazon S3 Apache Iceberg+4

naverNov 17, 2025Original article

Naver TV (opens in new tab)

NAVER is transitioning its internal search monitoring platform, SEER, to an architecture built on OpenTelemetry and open-source standards to achieve a more scalable and flexible observability environment. By adopting a vendor-agnostic approach, the engineering team aims to unify the collection of metrics, logs, and traces while contributing back to the global OpenTelemetry ecosystem. This shift underscores the importance of standardized telemetry protocols in managing complex, large-scale service infrastructures. ### Standardizing Observability with OTLP * The transition focuses on the OpenTelemetry Protocol (OTLP) as the primary standard for transmitting telemetry data across the platform. * Moving away from proprietary formats allows for a unified data model that encompasses metrics, traces, and logs, ensuring consistency across different services. * A standardized protocol simplifies the integration of various open-source backends, reducing the engineering overhead associated with supporting multiple telemetry formats. ### The OpenTelemetry Collector Pipeline * The Collector acts as a critical intermediary, decoupling the application layer from the storage backend to provide greater architectural flexibility. * **Receivers** are used to ingest data from diverse sources, supporting both OTLP-native applications and legacy systems. * **Processors** enable data transformation, filtering, and metadata enrichment (such as adding resource attributes) before the data reaches its destination. * **Exporters** manage the delivery of processed telemetry to specific backends like Prometheus for metrics or Jaeger for tracing, allowing for easy swaps of infrastructure components. ### Automated Management via OpenTelemetry Operator * The OpenTelemetry Operator is utilized within Kubernetes environments to automate the deployment and lifecycle management of the Collector. * It facilitates auto-instrumentation, allowing developers to collect telemetry from applications without manual code changes for every service. * The Operator ensures that the observability stack scales dynamically alongside the production workloads it monitors. ### Open-Source Contribution and Community * Beyond mere adoption, the NAVER engineering team actively participates in the OpenTelemetry community by sharing bug fixes and feature enhancements discovered during the SEER migration. * This collaborative approach ensures that the specific requirements of high-traffic enterprise environments are reflected in the evolution of the OpenTelemetry project. Adopting OpenTelemetry is a strategic move for organizations looking to avoid vendor lock-in and build a future-proof monitoring stack. For a successful implementation, teams should focus on mastering the Collector's pipeline configuration to balance data granularity with processing performance across distributed systems.

Opentelemetry Observability Monitoring Sre+3