slack·Mar 31, 2026·2 min readCurated summary

From Custom to Open: Scalable Network Probing and HTTP/3 Readiness with Prometheus

Slack needed better client-side observability while migrating edge services to HTTP/3, which uses QUIC over UDP rather than TCP. Existing SaaS tools and Prometheus Blackbox Exporter could not probe HTTP/3 endpoints, so an intern added QUIC support using Go’s `quic-go` library and open-sourced it. The result unified HTTP/1.1, HTTP/2, and HTTP/3 monitoring while making the capability available to the broader Prometheus community. ## Limitations of Legacy Monitoring - Slack used a mix of commercial monitoring services and internal tools for network measurements. - HTTP/3 introduced a major observability gap because it runs over QUIC/UDP. - Existing SaaS solutions lacked built-in HTTP/3 probing. - Prometheus Blackbox Exporter had no native QUIC support. - Without probing at scale, Slack could not reliably measure round-trip times, detect regressions to HTTP/2, or monitor hundreds of thousands of HTTP/3 endpoints. ## Adding QUIC Support to Blackbox Exporter - Intern Sebastian Feliciano selected `quic-go` because of its adoption and first-class Go HTTP client support. - The implementation used an `http3.Transport` with TLS and QUIC configuration: ```go http3Transport := &http3.Transport{ TLSClientConfig: tlsConfig, QUICConfig: &quic.Config{}, } ``` - The new transport was attached to a standard Go `http.Client`. - The implementation preserved Blackbox Exporter’s existing configuration and composability patterns. - Sebastian open-sourced the feature and eventually got it accepted upstream. ## In-House Integration and Operational Benefits - Because upstream review could take longer than the internship timeline, Slack built an internal system around the new functionality. - Grafana now provides a unified view of HTTP/1.1, HTTP/2, and HTTP/3 metrics. - Operators can compare protocol performance and correlate it with other telemetry. - Improved visibility supports more accurate alerts and faster debugging of HTTP/3 issues. ## Future Enhancements - **SNI routing tests:** Verify that shared edge infrastructure routes hostnames to the correct backend and presents the correct TLS certificate. - **End-to-end path visualization:** Map network hops between monitoring agents and endpoints to identify latency spikes or packet loss more precisely. ## Broader Lessons - Observability should be established before a major protocol or infrastructure migration. - Filling gaps through open source can benefit both the organization and the wider engineering community. - Supporting emerging protocols such as QUIC early helps future-proof monitoring systems. Slack recommends trying the new QUIC functionality in Prometheus Blackbox Exporter and contributing to its continued development.

Read original(opens in new tab)

ObservabilityGoPrometheusQuic+4

figma·Mar 25, 2026·3 min readCurated summary

Figma's Next-Generation Data Caching Platform | Figma Blog

Figma built FigCache to address scalability, reliability, and operational weaknesses in its Redis-based caching infrastructure. The stateless proxy provides a unified Redis data plane, decouples Redis connections from volatile client fleets, centralizes routing and security, and standardizes observability. After rollout to Figma’s main API in 2025, the caching layer reached six nines of uptime. ## Growing pains in caching - Redis evolved from a secondary component into a critical dependency for site availability. - Redis clusters were nearing connection limits as Figma’s infrastructure grew. - Rapid client-service scaling caused thundering herds of new connections, creating I/O bottlenecks and reducing availability. - Decentralized traffic management allowed applications to pollute or corrupt data across clusters. - Client libraries provided inconsistent observability, complicating incident diagnosis and mitigation. - A fragmented client ecosystem made it difficult to guarantee correct client-side behavior during failovers and topology changes. - Figma initially reduced Redis dependency in core API subsystems and created service-specific connection pooling, but pursued a broader platform redesign for long-term scalability. ## Design goals for a durable platform Figma defined several objectives for a caching platform capable of supporting future growth: - **Decouple Redis from client volatility:** Redis connection volume should not rise directly with elastic application fleets. - **Provide built-in observability:** Service owners and platform operators should receive consistent, granular visibility across workloads in a multitenant environment. - **Hide Redis Cluster complexity:** Clients should not need to manage topology changes such as scaling, failovers, or shard loss. - **Offer a universal endpoint:** Applications should access multiple Redis clusters through a centralized routing layer rather than managing separate endpoints and clients. - **Enable alternative backends:** New storage technologies, including durable systems, should be usable behind the same protocol and API. - **Remain extensible:** Cross-cutting capabilities such as encryption, guardrails, and traffic backpressure should be implemented centrally rather than repeatedly in applications. ## FigCache’s foundational architecture - Figma identified the need for a caching proxy that would serve as: - A unified Redis data plane. - An ingress layer for applications. - A connection multiplexer shielding Redis from client connection spikes. - A language-agnostic interface that hides cluster routing and management. - The platform was designed to centralize traffic decisions and abstract the underlying Redis topology from application developers. - FigCache is stateless and communicates using the Redis RESP wire protocol, allowing existing Redis-compatible clients and first-party libraries to use it. - Its broader platform role includes centralized security, routing, and end-to-end observability across the caching stack. ## Results - FigCache was rolled out to Figma’s main API service during the second half of 2025. - The caching layer subsequently achieved six nines of uptime. - The system established a foundation for more reliable, scalable, and interchangeable ephemeral storage across Figma. Figma’s approach demonstrates that Redis reliability at large scale requires more than larger clusters: a dedicated platform layer can isolate connection volatility, simplify client behavior, and centralize operational controls.

Read original(opens in new tab)

ObservabilityDistributed SystemsRedisCaching+2

datadog·Mar 23, 2026·1 min readCurated summary

When upserts don't update but still write: Debugging Postgres performance at scale

The provided content does not include the tech blog post itself. It consists primarily of Datadog’s website navigation and a promotional link about its Gartner recognition, so there is not enough article content to summarize reliably. ## Available Information - Datadog was named a Leader in the 2026 Gartner® Magic Quadrant™ for Observability Platforms. - The page navigation lists Datadog products across: - Infrastructure and application monitoring - Database and log management - Security - Digital experience monitoring - Software delivery - Incident and service management - AI-powered observability tools - The URL path references `debugging-postgres-performance`, suggesting the intended article may concern PostgreSQL performance debugging, but the article text is not included. Please provide the blog post’s body or a working text extraction for a substantive summary.

Read original(opens in new tab)

ObservabilityKubernetesDatadogApplication Performance Monitoring+2

datadog·Mar 9, 2026·2 min readCurated summary

When an AI agent came knocking: Catching malicious contributions in Datadog’s open source repos

Datadog announces that Gartner has named it a Leader in the 2026 Magic Quadrant for Observability Platforms. The surrounding product catalog presents Datadog as a broad platform spanning infrastructure, applications, data, logs, security, digital experience, software delivery, service management, and AI. However, the provided content does not include the blog post’s detailed analysis or Gartner’s specific evaluation criteria. ## Gartner Recognition - Datadog highlights its position as a **Leader** in the **Gartner Magic Quadrant for Observability Platforms 2026**. - The announcement links to a Gartner resource but provides no further details about the ranking, strengths, or limitations. ## Broad Observability Platform - **Infrastructure:** Infrastructure and container monitoring, metrics, Kubernetes autoscaling, network monitoring, serverless, cloud cost, storage, GPU monitoring, and Cloudcraft. - **Applications and data:** Application Performance Monitoring, service monitoring, profiling, dynamic instrumentation, database monitoring, data streams, data quality, and jobs monitoring. - **Logs and security:** Log management, sensitive-data scanning, audit trails, observability pipelines, cloud security, SIEM, code security, vulnerability management, and workload protection. - **Digital experience:** Browser and mobile RUM, product analytics, session replay, synthetic monitoring, mobile testing, error tracking, and experiments. - **Software delivery and service management:** CI visibility, test optimization, continuous testing, feature flags, code coverage, event management, SLOs, incident response, workflow automation, and service catalogs. - **AI capabilities:** Agent observability, GPU monitoring, AI integrations, AI agents, investigation tools, security analysis, MCP Server, and agent-building features. Datadog’s positioning is based on consolidating telemetry, security, developer, operations, and AI capabilities into one observability platform. To assess the Gartner recognition fully, readers would need the linked report or the complete article, which is not included here.

Read original(opens in new tab)

ObservabilityKubernetesDatadogApplication Performance Monitoring+2

datadog·Mar 4, 2026·3 min readCurated summary

Designing MCP tools for agents: Lessons from building Datadog's MCP server

Datadog’s initial MCP server simply exposed existing APIs, but real-world agent use revealed major problems with context limits, inaccurate trend analysis, and tool overload. The team redesigned its tools around token efficiency, query-based analysis, and a smaller, more deliberate tool surface. These changes improved both answer quality and cost, though emerging agent features may eventually reduce the need for some optimizations. ## Context Efficiency Matters - Observability results can be extremely large: a log record may range from roughly 100 characters to 1 MB. - CSV or TSV is more token-efficient than JSON for tabular data, often using about half as many tokens per record. - YAML can reduce token usage for nested data by around 20% compared with JSON. - Removing rarely used fields from default responses, while allowing agents to request them when needed, further reduces output size. - Combined formatting and field-trimming improvements allowed some tools to return approximately five times more records within the same token budget. - Pagination by record count is unreliable when records vary greatly in size. Datadog instead paginates by token budget and returns a cursor when the limit is reached. - Tools such as Cursor and Claude Code increasingly write long results to disk, which could make response-format efficiency less important in the future. ## Let Agents Query Data - Retrieval-only tools forced agents to infer trends from incomplete samples, such as guessing which services generated the most errors. - Agents sometimes repeatedly fetched logs to compensate, wasting tokens and producing unreliable answers. - SQL lets agents aggregate and filter data directly: ```sql SELECT service, COUNT(*) AS error_count FROM logs WHERE status = 'error' GROUP BY service ORDER BY error_count DESC LIMIT 10 ``` - Agents can select only necessary fields, limit row counts, and calculate aggregates without loading raw data. - SQL improved correctness and reduced costs; some evaluation scenarios became about 40% cheaper. - Supporting SQL at Datadog’s scale required significant infrastructure work because traditional relational databases were insufficient. ## Tools Are Not Free - Exposing every API endpoint as a separate tool increases tool-selection errors and consumes context through tool descriptions. - Flexible tools can support multiple related workflows through carefully designed schemas, reducing the total tool count. - Toolsets provide a core collection by default while allowing users to opt into specialized capabilities, though users must anticipate their needs. - Layered tools can first explain how to accomplish a task and then execute it, keeping specialized functionality out of the initial context. - Layering introduces additional tool calls and therefore increases latency. - Improving agent context management, including tool search and dynamically loaded skills, may reduce the need for aggressive tool minimization over time. The practical recommendation is to design MCP tools for how agents actually reason: minimize and control output size, provide query and aggregation capabilities instead of raw retrieval alone, and expose a focused set of flexible tools rather than mirroring every API endpoint.

Read original(opens in new tab)

ObservabilityDatabase DesignDatadogAPI Design+4

datadog·Mar 4, 2026·2 min readCurated summary

Designing MCP tools for agents: Lessons from building Datadog's MCP server | Datadog

Datadog is presented as a Leader in the 2026 Gartner Magic Quadrant for Observability Platforms. The provided content, however, consists almost entirely of Datadog’s website navigation rather than the blog post itself, so it does not include Gartner’s evaluation criteria, Datadog’s strengths, or any supporting analysis. ## Gartner Recognition - The page headline announces Datadog’s “Leader” position in the Gartner Magic Quadrant for Observability Platforms. - A link is provided to a Gartner-related resource page. - No ranking details, competitor comparisons, or Gartner commentary are included in the supplied text. ## Datadog’s Product Coverage The navigation indicates that Datadog offers a broad observability and operations platform spanning: - **Infrastructure:** infrastructure, container, network, serverless, GPU, storage, and cloud-cost monitoring. - **Applications:** APM, service monitoring, profiling, dynamic instrumentation, and agent observability. - **Data and logs:** database, data-stream, data-quality, job, log, and sensitive-data monitoring. - **Digital experience:** browser and mobile RUM, session replay, synthetic monitoring, product analytics, and error tracking. - **Security:** code, cloud, workload, vulnerability, compliance, SIEM, and application/API protection. - **Software delivery and service management:** CI visibility, testing, developer portals, incident response, SLOs, workflows, and case management. - **AI:** agent observability, GPU monitoring, AI integrations, Bits AI agents, and an MCP server. ## Limitations of the Provided Content - The actual article body is absent. - The text does not explain why Gartner recognized Datadog as a Leader. - It provides no technical findings, customer examples, methodology, or conclusions beyond the headline. The supplied excerpt supports only the conclusion that Datadog announced Gartner recognition and positions itself as a comprehensive observability platform. A substantive summary would require the full article text.

Read original(opens in new tab)

ObservabilityAI AgentsKubernetesDatadog+3

datadog·Feb 18, 2026·1 min readCurated summary

How we reduced the size of our Agent Go binaries by up to 77% | Datadog

The supplied text does not include the tech blog post itself. It contains Datadog navigation links and a promotional banner announcing its recognition as a Leader in the 2026 Gartner Magic Quadrant for Observability Platforms, but no article body or technical sections. ## Available content - Datadog promotes observability products covering: - Infrastructure and Kubernetes monitoring - Application performance monitoring - Logs and database monitoring - Security - Digital experience monitoring - Software delivery and CI visibility - Service management - AI-powered investigation and monitoring - The page links to an engineering article at: - `/blog/engineering/agent-go-binaries/` - No technical explanation, examples, conclusions, or section content from that article is included. Please provide the blog post’s full text or relevant excerpt for a substantive summary.

Read original(opens in new tab)

ObservabilityKubernetesDatadogApplication Performance Monitoring+3

figma·Feb 17, 2026·3 min readCurated summary

Redefining Impact as a Data Scientist | Figma Blog

Data science impact is not limited to experiments, forecasting, or optimization. In complex, high-stakes systems such as billing, data scientists can create value by making workflows understandable, validating correctness, and improving operational safety. Figma’s experience shows that effective data science may require domain modeling, cross-functional collaboration, instrumentation, and production-quality tools. ## Data Science as a Full-Stack Discipline - The role of data science varies by team: it may involve experimentation, product analysis, data modeling, instrumentation, or operational tooling. - Billing combines a user-facing product with backend infrastructure, so accuracy directly affects customer trust. - Supporting Billing required: - Building deep domain expertise - Partnering with engineers and other functions - Creating tools that explain and verify system behavior - Experimentation and opportunity analysis remained useful, but represented a smaller portion of the actual work. - Figma’s full-stack model encouraged the team to define the right data science support collaboratively rather than follow a fixed playbook. ## Explaining Complex Systems Beyond Charts and Models - Some of the most valuable data science work explains existing or historical outcomes rather than predicting future ones. - A single invoice seat charge may depend on: - Seat assignments and removals - Permission changes - Contract terms - Workspace state - Billing rules - The timing of state transitions - Figma built the **Invoice Seat Report** to reconstruct the complete reasoning behind each charge. - The application combines product events, contract metadata, billing rules, and historical state transitions, presenting the result in plain language. - Building it required: - Reconciling fragmented schemas and inconsistent historical data - Validating assumptions with engineers - Adding instrumentation where logs recorded what happened but not why - Translating billing rules into traceable and debuggable SQL transformations - The team also had to account for legacy multiyear contracts, sparse seat histories, early upgrades, and other cases that could create gaps in the data. ## Shaping Technical Direction Through Data - Data scientists can turn business rules into measurable checks that define expected system behavior. - These validations can detect drift, regressions, and anomalies in both development and production. - For Billing, automated verification is especially important because small errors in seat states or invoice calculations can affect customer charges and trust. - During Figma’s billing-model re-architecture, data science helped verify that: - Data moved correctly through pipelines - New pricing and billing logic produced intended outcomes - Customers did not enter unexpected billing states - The system could be monitored consistently across environments The practical lesson is to look beyond conventional analytics when assessing data science impact. In complex domains, building reliable data foundations, explanatory tools, and correctness checks may be more valuable than running another experiment.

Read original(opens in new tab)

ObservabilityData ScienceSqlData Engineering+3

line·Feb 4, 2026·4 min readCurated summary

Scaling to Infinity: LY Corporation’s

LY Corporation’s observability team evolved its time-series database to handle rapidly growing infrastructure and Kubernetes workloads. After outgrowing MySQL and OpenTSDB, the team built an engine optimized for high-cardinality metrics, low-latency queries, and seamless API compatibility. Its architecture now combines in-memory, Cassandra, and S3-compatible storage, enabling cost-efficient scaling while supporting trillions of daily metrics. ## Why Time-Series Storage Matters - Metrics record system state as timestamped numerical values. - They support dashboards, threshold-based alerts, and predictive analysis using tools such as ARIMA and Prophet. - Even a small metric record can consume about 280 bytes when timestamps, values, and tags are included. - One CPU metric collected every 15 seconds requires roughly 562 MiB per server annually; across 1,000 servers, this grows to about 548 GiB before adding memory, disk, and network metrics. - High-cardinality cloud environments make both storage cost and query latency critical operational concerns. ## Moving Beyond MySQL and OpenTSDB - MySQL initially became inadequate as the organization moved from SOA to MSA: - Write load increased sharply. - Storage costs and capacity requirements grew. - Query latency worsened for large datasets. - Rigid schemas could not easily represent changing cloud resources. - MySQL sharding provided temporary relief but could not support high-resolution metrics collected at intervals under one minute. - OpenTSDB, introduced in 2016 on Apache HBase, improved write performance but had important limitations: - Tag growth harmed UID-table lookup performance. - Metadata was restricted to a narrow character set. - Large queries required cache warm-up procedures. - These constraints led to the development of an internal database beginning in 2018. ## Building the Internal Time-Series Database - The 2019 engine was designed around: - Flexible protocol support independent of a particular agent. - Linear scalability without downtime. - Low-latency processing of high-resolution metrics. - Strong availability during failures. - Inspired by Meta’s Gorilla research, the team used access patterns in which most queries target recent data. - Frequently accessed metrics were kept in an in-memory database, while colder data was stored in Apache Cassandra. - The new engine enabled metric volumes to grow by more than 200 billion records annually while preserving existing APIs. - Users benefited from the new backend without migration work or code changes. ## Scaling for Kubernetes Workloads - Kubernetes introduced rapidly changing pods, dynamically allocated volumes, and much higher metric churn. - Both major storage layers encountered scaling problems: - IMDB initially required adding identical hardware, limiting expansion options. - Cassandra rebalancing could take tens of hours because of its data volume. - The team improved IMDB with weighted load balancing so nodes with different capacities could be used effectively. - Storage was divided into tiers: - Recent 14-day data remained in Cassandra for high-performance access. - Older data was moved to S3-compatible storage. - This reduced Cassandra dependency, lowered costs, simplified operations, and enabled more flexible hardware and Kubernetes-based deployment. ## Writing and Reading Through S3 - The write path separates data processing from long-term storage: - A Dumper reads metric slots from IMDB. - It converts them into internally defined sub-blocks. - A Block Dumper combines sub-blocks into blocks and writes them to S3. - A Storage Gateway reads the blocks for queries and caches them on local disks. - Disk caching initially caused excessive page-cache use and rapid memory exhaustion. - Direct I/O was considered but withdrawn after the cloud storage team warned that it consumed too much shared bandwidth. - Through cross-team collaboration, the team adopted a B+ tree-based cache that made better use of the kernel page cache without overloading infrastructure. ## Future Direction: From Storage to Intelligence - The team aims to move beyond recording metrics toward prediction and AI-assisted operations. - Achieving this requires consolidating time-series data currently scattered across internal systems. - A key requirement is to perform this integration without imposing migration work or breaking changes on users. - The broader goal is an observability platform that turns unified metrics into predictive and intelligent operational capabilities. The main recommendation is to design time-series platforms around real access patterns, tier storage according to data age, and preserve compatibility while evolving the backend. At extreme scale, careful storage architecture and collaboration across infrastructure teams are as important as raw database performance.

Read original(opens in new tab)

ObservabilityAIDatabase DesignKubernetes+4

aws·Feb 2, 2026·3 min readCurated summary

AWS Weekly Roundup: Amazon Bedrock agent workflows, Amazon SageMaker private connectivity, and more (February 2, 2026) | Amazon Web Services

The AWS Weekly Roundup highlights new capabilities for AI agents, private connectivity, encryption management, and resilience testing. Major launches include Bedrock server-side tools and longer prompt caching, SageMaker Unified Studio support for PrivateLink, and S3 encryption changes without data movement. Additional updates strengthen event-driven architectures, observability, zero-trust access, and AI-assisted AWS deployments. ## AI Agents and Developer Workflows - Amazon Bedrock’s Responses API now supports server-side tools such as web search, code execution, and database updates within AWS security boundaries. - Bedrock also offers a one-hour prompt-cache TTL for select Anthropic Claude models, improving performance and reducing costs for long-running, multi-turn agents. - AWS MCP Server deployment SOPs, currently in preview, let agents deploy applications from natural-language prompts using CDK, CloudFormation, and CI/CD workflows. - The deployment preview supports React, Vue.js, Angular, and Next.js through tools such as Kiro, Cursor, and Claude Code. - CloudWatch Application Signals integration with Kiro provides AI-assisted investigation of service health, SLO compliance, and observability issues. ## Private Connectivity and Zero-Trust Security - SageMaker Unified Studio now supports AWS PrivateLink, allowing VPC traffic to remain within the AWS network instead of traversing the public internet. - IAM policies can govern private SageMaker connectivity for stricter security and compliance requirements. - AWS Verified Access guidance demonstrates centralized zero-trust application access across multi-account environments using IAM Identity Center and AWS RAM. - AWS Network Firewall adds predefined web categories for identifying and controlling generative AI application traffic, with full-URL filtering available alongside TLS inspection. ## Storage, Encryption, and Database Performance - Amazon S3’s `UpdateObjectEncryption` API changes encryption for existing objects without moving or re-uploading data. - Supported operations include switching from SSE-S3 to SSE-KMS, rotating customer-managed KMS keys, and standardizing encryption with S3 Batch Operations. - Amazon Keyspaces table pre-warming prepares tables for predictable high-throughput workloads, reducing throttling and cold-start delays during traffic spikes. - Pre-warming works with on-demand and provisioned capacity, including multi-Region tables. - DynamoDB MRSC global tables now integrate with AWS Fault Injection Service, enabling simulated Regional failures and validation of replication and application resilience. ## Event-Driven Systems and Observability - EventBridge’s event payload limit increased from 256 KB to 1 MB, allowing events to carry richer JSON, telemetry, ML, and generative AI data without external storage or fragmentation. - Lambda’s enhanced observability for Kafka event source mappings adds CloudWatch logs and metrics for polling, scaling, processing state, permissions, and failures. - The feature supports both Amazon MSK and self-managed Apache Kafka sources. ## CloudFormation and Community - AWS’s 2025 CloudFormation review covers improved troubleshooting, drift-aware change sets, stack refactoring, StackSets, the CloudFormation language server, and IaC MCP tooling. - AWS Community Day Romania will take place April 23–24, 2026, featuring technical sessions, AWS experts, and networking opportunities. Together, these updates point toward more private, observable, resilient, and AI-assisted AWS operations. Teams should evaluate the new capabilities against their security, scalability, and automation needs, particularly Bedrock agent tooling, S3 encryption updates, PrivateLink connectivity, and resilience testing.

Read original(opens in new tab)

ObservabilityAI AgentsGenerative AIAmazon Bedrock+4

datadog·Jan 7, 2026·1 min readCurated summary

Hardening eBPF for runtime security: Lessons from Datadog Workload Protection | Datadog

The provided text does not include the blog post’s article body. It contains Datadog’s navigation menu and a link to an engineering post titled around “eBPF workload protection lessons,” so there is not enough source material to accurately summarize its technical arguments or conclusions. ## Available information - The page is hosted by Datadog’s engineering blog. - The linked topic concerns workload protection built with eBPF. - Datadog’s broader product areas include infrastructure monitoring, application performance monitoring, security, logs, and AI. - The excerpt itself does not describe: - The eBPF implementation - Design challenges or trade-offs - Performance considerations - Security detection methods - Lessons learned or recommendations Please provide the article text or a fuller extract for a substantive summary.

Read original(opens in new tab)

ObservabilityKubernetesDatadogApplication Performance Monitoring+3

datadog·Jan 7, 2026·3 min readCurated summary

Hardening eBPF for runtime security: Lessons from Datadog Workload Protection

eBPF gives security tools broad, efficient, and relatively safe access to Linux kernel activity, making it well suited for runtime threat detection. Datadog chose it for Workload Protection after comparing kernel modules, tracing interfaces, ptrace, seccomp, Linux Audit, and other approaches. However, five years of production use across diverse kernels showed that eBPF’s safety and performance benefits are not automatic; reliability, compatibility, observability, and operational discipline are essential at scale. ## Why Runtime Workload Protection Is Needed - Static analysis and vulnerability scanning cannot catch every threat. - Zero-days and vulnerable third-party dependencies can remain active while patches are being prepared or deployed. - Workload Protection is intended to: - Monitor known-vulnerable workloads until they can be patched. - Continuously observe all workloads. - Detect and help mitigate previously unknown vulnerabilities during incident response. ## Alternatives Evaluated Datadog evaluated a broad range of Linux monitoring and instrumentation mechanisms: - **Linux kernel modules** - Offer deep access and can hook or replace almost any kernel function. - Are invasive and often considered too risky for production infrastructure. - **Traditional tracing interfaces** - Include inotify, fanotify, kprobes, tracepoints, and perf events. - Provide useful visibility but generally need to be combined for comprehensive coverage. - **ptrace and seccomp-bpf** - Can provide detailed user-space process visibility. - Are less suitable as a unified solution for monitoring the whole system. - **Linux Audit** - Produces configurable streams for process execution, file access, and network activity. - Is widely used by security tooling but has its own performance and operational tradeoffs. - **Other mechanisms** - Netlink, LD_PRELOAD, and binfmt_misc were also considered. - Each involves compromises in reliability, visibility, or system impact. ## Why eBPF Stood Out - **Safety checks** - The kernel statically verifies eBPF bytecode before loading it. - Verification detects issues such as infinite loops and unsafe memory access. - This is safer than deploying custom kernel modules, though eBPF can still cause harm or performance problems. - **Performance** - eBPF generally has lower overhead than approaches such as Linux Audit or ptrace. - Actual impact depends heavily on implementation and workload. - **Unified visibility** - A single mechanism can observe process, filesystem, and network activity. - This avoids assembling multiple specialized tracing systems. - **Container and namespace coverage** - eBPF provides consistent visibility across namespaces, cgroups, and containers. - CO-RE (Compile Once–Run Everywhere) improves portability across Linux distributions and kernel versions. - **Enforcement capabilities** - BPF LSM programs support mandatory access controls. - This gives eBPF enforcement power beyond ordinary tracing mechanisms, which is important for runtime security. ## Lessons from Operating eBPF at Scale After five years of operating an agent that hooks process scheduling, filesystem, and networking internals, Datadog emphasizes that production eBPF is more complicated than its reputation suggests. The six areas of operational experience are: - Ensuring programs load, attach, and continue firing across kernel versions. - Capturing and enriching event data accurately. - Monitoring and auditing eBPF usage to reduce the attack surface. - Coexisting with other eBPF-based tools on the same host. - Measuring and controlling performance overhead. - Shipping changes safely through disciplined rollout practices. The practical recommendation is to treat eBPF as powerful infrastructure rather than a maintenance-free kernel feature: validate behavior across kernels and workloads, monitor its own operation, measure overhead continuously, and use cautious deployment practices.

Read original(opens in new tab)

ObservabilityDatabase DesignEbpfLinux Kernel+4

meta·Dec 19, 2025Original article

DrP: Meta's Root Cause Analysis Platform at Scale (opens in new tab)

DrP is Meta’s programmatic root cause analysis (RCA) platform designed to automate incident investigations and reduce the burden of manual on-call tasks. By codifying investigation playbooks into executable "analyzers," the platform significantly lowers the mean time to resolve (MTTR) by 20% to 80% for over 300 teams. This systematic approach replaces outdated manual scripts with a scalable backend that executes 50,000 automated analyses daily, providing immediate context when alerts fire. ## Architecture and Core Components * **Expressive SDK:** Provides a framework for engineers to codify investigation workflows into "analyzers," utilizing a rich library of helper functions and machine learning algorithms. * **Built-in Analysis Tools:** The platform includes native support for anomaly detection, event isolation, time-series correlation, and dimension analysis to identify specific problem areas. * **Scalable Backend:** A multi-tenant execution environment manages a worker pool that handles thousands of requests securely and asynchronously. * **Workflow Integration:** DrP is integrated directly into Meta’s internal alerting and incident management systems, allowing for automatic triggering without human intervention. ## Authoring and Verification Workflow * **Template Bootstrapping:** Engineers use the SDK to generate boilerplate code that captures required input parameters and context in a type-safe manner. * **Analyzer Chaining:** The system allows for seamless dependency analysis by passing context between different analyzers, enabling investigations to span multiple interconnected services. * **Automated Backtesting:** Before deployment, analyzers undergo automated backtesting integrated into the code review process to ensure accuracy and performance. * **Decision Tree Logic:** Investigation steps are modeled as decision trees within the code, allowing the analyzer to follow different paths based on the data it retrieves. ## Execution and Post-Processing * **Trigger-based Analysis:** When an alert is activated, the backend automatically queues the relevant analyzer, ensuring findings are available as soon as an engineer begins triaging. * **Automated Mitigation:** A post-processing system can take direct action based on investigation results, such as creating tasks or submitting pull requests to resolve identified issues. * **DrP Insights:** This system periodically reviews historical analysis outputs to identify and rank the top causes of alerts, helping teams prioritize long-term reliability fixes. * **Alert Annotation:** Results are presented in both human-readable text and machine-readable formats, directly annotating the incident logs for the on-call responder. ## Practical Conclusion Organizations managing large-scale distributed systems should transition from static markdown playbooks to executable investigation code. By implementing a programmatic RCA framework like DrP, teams can scale their troubleshooting expertise and significantly reduce "on-call fatigue" by automating the repetitive triage steps that typically consume the first hour of an incident.

ObservabilityAIMachine LearningAutomation+4

aws·Dec 2, 2025Original article

Amazon Bedrock AgentCore adds quality evaluations and policy controls for deploying trusted AI agents (opens in new tab)

AWS has introduced several new capabilities to Amazon Bedrock AgentCore designed to remove the trust and quality barriers that often prevent AI agents from moving into production environments. These updates, which include granular policy controls and sophisticated evaluation tools, allow developers to implement strict operational boundaries and monitor real-world performance at scale. By balancing agent autonomy with centralized verification, AgentCore provides a secure framework for deploying highly capable agents across enterprise workflows. **Governance through Policy in AgentCore** * This feature establishes clear boundaries for agent actions by intercepting tool calls via the AgentCore Gateway before they are executed. * By operating outside of the agent’s internal reasoning loop, the policy layer acts as an independent verification system that treats the agent as an autonomous actor requiring permission. * Developers can define fine-grained permissions to ensure agents do not access sensitive data inappropriately or take unauthorized actions within external systems. **Quality Monitoring with AgentCore Evaluations** * The new evaluation framework allows teams to monitor the quality of AI agents based on actual behavior rather than theoretical simulations. * Built-in evaluators provide standardized metrics for critical dimensions such as helpfulness and correctness. * Organizations can also implement custom evaluators to ensure agents meet specific business-logic requirements and industry-specific compliance standards. **Enhanced Memory and Communication Features** * New episodic functionality in AgentCore Memory introduces a long-term strategy that allows agents to learn from past experiences and apply successful solutions to similar future tasks. * Bidirectional streaming in the AgentCore Runtime supports the deployment of advanced voice agents capable of handling natural, simultaneous conversation flows. * These enhancements focus on improving consistency and user experience, enabling agents to handle complex, multi-turn interactions with higher reliability. **Real-World Application and Performance** * The AgentCore SDK has seen rapid adoption with over 2 million downloads, supporting diverse use cases from content generation at the PGA TOUR to financial data analysis at Workday. * Case studies highlight significant operational gains, such as a 1,000 percent increase in content writing speed and a 50 percent reduction in problem resolution time through improved observability. * The platform emphasizes 100 percent traceability of agent decisions, which is critical for organizations transitioning from reactive to proactive AI-driven operations. To successfully scale AI agents, organizations should transition from simple prompt engineering to a robust agentic architecture. Leveraging these new policy and evaluation tools will allow development teams to maintain the necessary control and visibility required for customer-facing and mission-critical deployments.

ObservabilityAIAI AgentsGenerative AI+5

datadog·Nov 18, 2025·2 min readCurated summary

Scaling real-time file monitoring with eBPF: How we filtered billions of kernel events per minute | Datadog

Datadog was named a Leader in the 2026 Gartner® Magic Quadrant™ for Observability Platforms. The announcement positions Datadog as a provider of broad, integrated monitoring across infrastructure, applications, logs, security, digital experiences, software delivery, and AI. The supplied content does not include Gartner’s evaluation details or the blog post’s supporting arguments. ## Recognition and Positioning - Datadog highlights its designation as a Leader in the Gartner Magic Quadrant for Observability Platforms. - The announcement emphasizes Datadog’s unified observability platform rather than a single monitoring product. ## Breadth of the Platform - **Infrastructure:** infrastructure, container, network, serverless, GPU, storage, and cloud-cost monitoring. - **Applications and data:** APM, database monitoring, continuous profiling, data-stream monitoring, and job monitoring. - **Logs and observability operations:** log management, sensitive-data scanning, audit trails, and observability pipelines. - **Security:** cloud security, SIEM, workload protection, code security, vulnerability management, and application/API protection. - **Digital experience:** browser and mobile RUM, session replay, synthetic monitoring, product analytics, and error tracking. - **Software delivery and service management:** CI visibility, testing, feature flags, incident response, SLOs, workflow automation, and case management. - **AI capabilities:** agent observability, GPU monitoring, AI integrations, Bits AI agents, and investigation tools. Overall, the available material presents Datadog’s Gartner Leader recognition and extensive product coverage, but it does not provide enough article text to summarize the specific reasoning behind the designation.

Read original(opens in new tab)

ObservabilityKubernetesDatadogEbpf+2