spotify·Jul 20, 2026·3 min readCurated summary

Content Ingestion & Podcast Video Incident Report | Spotify Engineering

Spotify’s June 24 publishing incident delayed video podcast episodes for several hours after transcoding capacity was overwhelmed. The disruption resulted from insufficient capacity headroom, a concurrent batch job, increased processing costs, and a scheduling bug that reduced throughput by about 10%. Spotify cleared the backlog by the following morning and has since expanded capacity, improved monitoring, and launched a broader publishing reliability initiative. ## What Caused the Delay - A spike in new video podcast submissions pushed transcoding infrastructure close to its maximum capacity. - A scheduled batch job was reprocessing existing episodes, consuming capacity needed for new publications. - Recent video-quality improvements increased the processing time and compute required per episode. - A resource-scheduling bug following an infrastructure migration underused available hardware, reducing throughput by approximately 10%. - Creators re-uploaded missing episodes, unintentionally adding more load because Spotify did not clearly confirm that uploads had been received and queued. ## Incident Timeline and Response - Early monitoring alerts fired at 13:30 UTC, but they were not initially recognized as a broader capacity problem. - At 15:00, a delivery spike brought transcoding close to capacity. - The batch job was stopped at 16:35 to free resources. - Formal incident response began at 17:34 after queue thresholds were exceeded. - A scheduling fix was deployed at 20:49, and an additional processing cluster came online at 00:14 on June 25. - All queues were cleared by 01:02, with normal pipeline operation confirmed at 07:30. - Approximately four hours passed between the first alerts and formal incident response, highlighting a major monitoring and escalation gap. ## Remediation Measures - Spotify increased transcoding capacity by approximately 67% to provide more room for traffic spikes and batch processing. - The resource-scheduling bug was fixed, restoring use of previously underutilized compute capacity. - Monitoring was improved to warn earlier when capacity is approaching operational limits. ## Broader Reliability Program - Capacity planning is being expanded to account for burst traffic, background workloads, and recovery needs—not only typical steady-state demand. - Publishing systems will prioritize real-time creator uploads over background operations. - Spotify is extending rate limiting and backpressure mechanisms across the pipeline so unexpected load can be handled more gracefully. - A dedicated cross-team effort is coordinating these improvements across the publishing infrastructure. Spotify’s main lesson is that reliable publishing requires both additional capacity and better operational safeguards. Earlier detection, clearer upload acknowledgments, stronger workload prioritization, and deliberate burst-capacity planning should reduce the likelihood and impact of similar incidents.

Read original(opens in new tab)

MonitoringIncident ResponseRate LimitingVideo Transcoding+4

line·Mar 13, 2026·4 min readCurated summary

LY Corporation's Cloud Infrastructure Reorganization: Introducing the Architecture of Flava, a Next-Generation Platform Integrating Two Massive Clouds

LY Corporation is consolidating the former LINE “Verda” and Yahoo Japan “YNW” private clouds into Flava, a next-generation platform designed for large-scale, uninterrupted operations. Its approach assumes failures will occur, prioritizing stateless services, application-led availability, rapid IaC-based recovery, and extensive automation. Flava also restructures the architecture around shared resources, upstream OpenStack, default VPC networking, and user-driven cost optimization. ## Failure-Aware Design and Operations - VM root disks are treated as temporary; persistent data is placed in external storage so instance failures have limited service impact. - Availability is achieved through cooperation between infrastructure and applications rather than excessive infrastructure-side guarantees. - Recovery focuses on maintaining service continuity, rebuilding environments quickly with infrastructure as code, and avoiding lengthy root-cause investigations during incidents. - The company promotes KaaS and PaaS to help developers build resilient services without managing low-level infrastructure. - OS configuration, package installation, networking, and other changes are managed as code through CI/CD. - Deployments are performed by availability zone to limit the blast radius of failures. ## Observability from Fleet-Wide Trends to Root Causes - Prometheus, Grafana, and custom dashboards monitor overall cloud health and long-term trends. - When anomalies appear, engineers investigate at a deeper level using kernel traces, packet captures, and other low-level diagnostics. - This combination of broad monitoring and detailed investigation allows teams to move between “forest” and “tree” perspectives. - The operational model depends not only on tools but also on engineers capable of tracing problems down to their fundamental causes. ## OSS, Software-Defined Infrastructure, and Custom Development - The platform relies heavily on OpenStack, Envoy, Linux kernel technologies such as eBPF/XDP, FRR, and Ceph. - LY contributes patches and new capabilities upstream instead of maintaining long-lived private forks. - It has developed SRv6 BGP functionality required for Flava’s VPCs and contributed related work to FRRouting and the Linux kernel. - Compute, VPC, DNS, and load-balancing services run primarily on commodity x86 servers rather than specialized appliances. - XDP-based data planes, hardware offload, and system tuning are used to achieve near-wire-speed throughput and low latency. - Where OSS cannot meet internal requirements, LY builds systems from scratch, including the Dragon object store, SDN control-plane components, load-balancer health agents, and service discovery tools written in Rust, Go, and Python. ## Autonomous Hardware Operations - With tens of thousands of hypervisors and petabyte-scale storage, hardware failures occur continuously. - Failure detection, requests to data-center technicians, hardware replacement, and cluster reintegration are largely automated. - Some exceptional cases still require engineers, but LY plans to use LLMs to automate more of these operational tasks. ## Flava’s Architectural Improvements ### Shared Resource Pools - Older clouds used many dedicated clusters and resource pools, making capacity planning complex and reducing utilization. - Flava consolidates most products and services into one large shared resource pool. - This reduces planning variables, improves resource efficiency, and accelerates provisioning. ### Upstream-Compatible OpenStack - Excessive customization in the legacy environment made upgrades difficult. - Flava minimizes private patches, follows upstream OpenStack, and contributes necessary improvements back to the project. - This enables regular upgrade cycles and keeps security fixes and features current. ### VPC by Default - VPC networking is the standard security model for multi-tenant workloads. - Logical isolation replaces many cases where dedicated VLANs or firewalls previously required months of preparation. - Equivalent security environments can now be provisioned in minutes. - The VPC data plane is being redesigned with XDP to support the reliability and performance required at company-wide scale. ### Built-In Cost Optimization - Development environments require resource lifetimes, allowing unused “zombie” resources to be deleted automatically. - Object storage offers bucket classes such as “High Performance” and “Scalable.” - Users can change storage classes without changing endpoints, adapting cost and performance as access patterns evolve. ## Remaining Challenges - Flava currently offers only a limited set of products and must expand its capabilities while addressing post-launch bugs and overlooked requirements. - The largest challenge is migrating users from the legacy platforms. - LY is working to provide transparent migration tools and reduce manual effort while shortening the period of duplicate investment in old and new infrastructure. ## Team and Engineering Culture - The team includes specialists ranging from kernel developers to web-front-end engineers. - Engineers are expected to understand and control infrastructure rather than treat it as a black box. - Deep source-level expertise enables upstream OSS contributions and informed negotiations with commercial vendors. - This culture of ownership and technical control is presented as a core reason the platform can evolve at LY’s scale. LY’s experience demonstrates that large private clouds can combine OSS, custom software, commodity hardware, and rigorous automation effectively. The practical recommendation is to design for failure, keep infrastructure reproducible through IaC, contribute changes upstream where possible, and use custom development selectively for requirements that general-purpose platforms cannot satisfy.

Read original(opens in new tab)

MonitoringKubernetesInfrastructure As CodeOpenstack+4

woowahan·Dec 5, 2025Original article

아한형제들 기술블로그 (opens in new tab)

The 7th Woowacourse crew has successfully launched three distinct services, demonstrating that modern software engineering requires a synergy of technical mastery and "soft skills" like product planning and team communication. By owning the entire lifecycle from ideation to deployment, these developers moved beyond mere coding to solve real-world problems through agile iterations, user feedback, and robust infrastructure management. The program’s focus on the full stack of development—including monitoring, 2-week sprints, and collaborative design—highlights a shift toward producing well-rounded engineers capable of navigating professional environments. ### The Woowacourse Full-Cycle Philosophy * The 10-month curriculum emphasizes soft skills, including speaking and writing, alongside traditional technical tracks like Web Backend, Frontend, and Mobile Android. * During Level 3 and 4, crews transition from fundamental programming to managing team projects where they must handle everything from initial architecture to UI/UX design. * The process mimics real-world industry standards by implementing 2-week development sprints, establishing monitoring environments, and managing automated deployment pipelines. * The core goal is to shift the developer's mindset from simply writing code to understanding why certain features are planned and how architecture choices impact the final user value. ### Pickeat: Collaborative Dining Decisions * This service addresses "decision fatigue" during group meals by providing a collaborative platform to filter restaurants based on dietary constraints and preferences. * Technical challenges included frequent domain restructuring and UI overhauls as the team pivoted based on real-world user feedback during demo days. * The platform utilizes location data for automatic restaurant lookups and supports real-time voting mechanisms to ensure democratic and efficient group decisions. * Development focused on aligning team judgment standards and iterating quickly to validate product-market fit rather than adhering strictly to initial specifications. ### Bottari: Real-Time Synchronized Checklists * Bottari is a checklist service designed for situations like traveling or moving, focusing on "becoming a companion for the user’s memory." * The service features template-based list generation and a "Team Bottari" function that allows multiple users to collaborate on a single list with real-time synchronization. * A major technical focus was placed on the user experience flow, specifically optimizing notification timing and sync states to provide "peace of mind" for users. * The project demonstrates the principle that technology serves as a tool for solving psychological pain points, such as the anxiety of forgetting essential items. ### Coffee Shout: Real-Time Betting and Mini-Games * Designed to gamify office culture, this service replaces simple "rock-paper-scissors" with interactive mini-games and weighted roulette for coffee bets. * The technical stack involved challenging implementations of WebSockets and distributed environments to handle the concurrency required for real-time gaming. * The team focused on algorithm balancing for the weighted roulette system to ensure fairness and excitement during the betting process. * Refinement of the service was driven by direct feedback from other Woowacourse crews, emphasizing the importance of community testing in the development lifecycle. These projects underscore that the transition from a student to a professional developer is defined by the ability to manage shifting requirements and technical complexity while maintaining a focus on the end-user's experience.

MonitoringAndroidDistributed SystemsSoftware Architecture+4

woowahan·Dec 2, 2025Original article

How Woowa Brothers Detects (opens in new tab)

Woowa Brothers addresses the inevitability of system failures by shifting from traditional resource-based monitoring to a specialized Service Anomaly Detection system. By focusing on high-level service metrics such as order volume and login counts rather than just CPU or memory usage, they can identify incidents that directly impact the user experience. This approach ensures near real-time detection and provides a structured response framework to minimize damage during peak service hours. ### The Shift to Service-Level Monitoring * Traditional monitoring focuses on infrastructure metrics like CPU and memory, but it is impossible to monitor every system variable, leading to "blind spots" in failure detection. * Service metrics, such as real-time login counts and payment success rates, are finite and offer a direct reflection of the actual customer experience. * By monitoring these core indicators, the SRE team can detect anomalies that system-level alerts might overlook, ensuring that no failure goes unnoticed. ### Requirements for Effective Anomaly Detection * **Real-time Performance:** Alerts must be triggered in near-real-time to allow for immediate intervention before the impact scales. * **Explainability:** The system favors transparent logic over "black-box" AI models, allowing developers to quickly understand why an alert was triggered and how to improve the detection logic. * **Integrated Response:** Beyond just detection, the system must provide a clear response process so that any engineer, regardless of experience, can follow a standardized path to resolution. ### Technical Implementation and Logic * The system leverages the predictable, pattern-based nature of delivery service traffic, which typically peaks during lunch and dinner. * The team chose a Median-based approach to generate "Prediction" values from historical data, as it is more robust against outliers and easier to analyze than complex methods like IQR or 2-sigma. * Detection is determined by comparing "Actual" values against "Warning" and "Critical" thresholds derived from the predicted median. * To prevent false positives caused by temporary spikes, the system tracks "threshold reach counts," requiring a metric to stay in an abnormal state for a specific number of consecutive cycles before firing a Slack alert. ### Optimization of Alert Accuracy * Each service metric requires a tailored "settling period" to find the optimal balance between detection speed and accuracy. * Setting a high threshold reach count improves accuracy but slows down detection, while a low count accelerates detection at the risk of increased false positives. * Alerts are delivered via Slack with comprehensive context, including current status and urgency, to facilitate rapid decision-making. For organizations running high-traffic services, prioritizing service-level indicators (SLIs) over infrastructure metrics can significantly reduce the time to detect critical failures. Implementing simple, explainable statistical models like the Median approach allows teams to maintain a reliable monitoring system that evolves alongside the service without the complexity of uninterpretable AI models.

MonitoringIncident ManagementAnomaly DetectionSre+2

naver·Nov 17, 2025Original article

Naver TV (opens in new tab)

This technical session from NAVER ENGINEERING DAY 2025 details the transition from traditional open-source exporters to a Telegraf-based architecture for collecting custom system metrics. By evaluating various monitoring tools through rigorous benchmarking, the developers demonstrate how Telegraf provides a more flexible and high-performance framework for infrastructure observability. The presentation concludes that adopting Telegraf streamlines the metric collection pipeline and offers superior scalability for complex, large-scale service environments. ### Context and Motivation for Open-Source Exporters * The project originated from the need to overcome the limitations of standard open-source exporters that lacked support for specific internal business logic. * Engineers sought a unified way to collect diverse data points without managing dozens of fragmented, single-purpose agents. * The primary goal was to find a solution that could handle high-frequency data ingestion while maintaining low resource overhead on production servers. ### Benchmark Testing for Metric Collection * A comparative analysis was conducted between several open-source monitoring agents to determine their efficiency under load. * Testing focused on critical performance indicators, including CPU and memory footprint during peak metric throughput. * The results highlighted Telegraf's stability and consistent performance compared to other exporter-based alternatives, leading to its selection as the primary collection tool. ### Telegraf Architecture and Customization * Telegraf operates as a plugin-driven agent, utilizing four distinct categories: Input, Processor, Aggregator, and Output plugins. * The development team shared their experience writing custom exporters by leveraging Telegraf’s modular Go-based framework. * This approach allowed for the seamless transformation of raw data into various formats (such as Prometheus or InfluxDB) using a single, unified configuration. ### Operational Gains and Technical Options * Post-implementation, the system saw a significant reduction in operational complexity by consolidating various metric streams into a single agent. * Specific Telegraf options were utilized to fine-tune the collection interval and batch size, optimizing the balance between data granularity and network load. * The migration improved the reliability of metric delivery through built-in retry mechanisms and internal buffers that prevent data loss during transient network failures. For teams currently managing a sprawling array of open-source exporters, migrating to a Telegraf-based architecture is recommended to centralize metric collection. The plugin-based system not only reduces the maintenance burden but also provides the necessary extensibility to support specialized custom metrics as service requirements evolve.

MonitoringOpen SourceMetrics CollectionTelegraf+2

naver·Nov 17, 2025Original article

Naver TV (opens in new tab)

NAVER is transitioning its internal search monitoring platform, SEER, to an architecture built on OpenTelemetry and open-source standards to achieve a more scalable and flexible observability environment. By adopting a vendor-agnostic approach, the engineering team aims to unify the collection of metrics, logs, and traces while contributing back to the global OpenTelemetry ecosystem. This shift underscores the importance of standardized telemetry protocols in managing complex, large-scale service infrastructures. ### Standardizing Observability with OTLP * The transition focuses on the OpenTelemetry Protocol (OTLP) as the primary standard for transmitting telemetry data across the platform. * Moving away from proprietary formats allows for a unified data model that encompasses metrics, traces, and logs, ensuring consistency across different services. * A standardized protocol simplifies the integration of various open-source backends, reducing the engineering overhead associated with supporting multiple telemetry formats. ### The OpenTelemetry Collector Pipeline * The Collector acts as a critical intermediary, decoupling the application layer from the storage backend to provide greater architectural flexibility. * **Receivers** are used to ingest data from diverse sources, supporting both OTLP-native applications and legacy systems. * **Processors** enable data transformation, filtering, and metadata enrichment (such as adding resource attributes) before the data reaches its destination. * **Exporters** manage the delivery of processed telemetry to specific backends like Prometheus for metrics or Jaeger for tracing, allowing for easy swaps of infrastructure components. ### Automated Management via OpenTelemetry Operator * The OpenTelemetry Operator is utilized within Kubernetes environments to automate the deployment and lifecycle management of the Collector. * It facilitates auto-instrumentation, allowing developers to collect telemetry from applications without manual code changes for every service. * The Operator ensures that the observability stack scales dynamically alongside the production workloads it monitors. ### Open-Source Contribution and Community * Beyond mere adoption, the NAVER engineering team actively participates in the OpenTelemetry community by sharing bug fixes and feature enhancements discovered during the SEER migration. * This collaborative approach ensures that the specific requirements of high-traffic enterprise environments are reflected in the evolution of the OpenTelemetry project. Adopting OpenTelemetry is a strategic move for organizations looking to avoid vendor lock-in and build a future-proof monitoring stack. For a successful implementation, teams should focus on mastering the Collector's pipeline configuration to balance data granularity with processing performance across distributed systems.

MonitoringObservabilityOpentelemetrySre+3

line·Oct 17, 2025Original article

Essential Element for App Success: Error Monitoring (opens in new tab)

Effective mobile app management requires proactive outage monitoring to prevent user churn caused by failures in critical flows like registration or payment. Relying on user reports is often too late, so developers must implement systematic event collection and real-time dashboards to identify issues the moment they arise. By integrating tools like Sentry or Firebase, teams can maintain high quality through immediate response and detailed performance analysis. ### Implementing Sentry in Flutter * **Dependency and Initialization**: Integration begins by adding `sentry_flutter` and `sentry_dio` to the project. The initialization process involves setting the Data Source Name (DSN), environment tags (e.g., production vs. staging), and release versions to ensure logs are correctly categorized. * **Performance and Privacy**: Developers should configure `tracesSampleRate` and `profilesSampleRate` to balance monitoring depth with costs. Additionally, the `beforeSend` callback allows for masking sensitive user data like authorization headers or IP addresses before they are transmitted. * **Contextual Tracking**: To aid debugging, the system captures user IDs via `Sentry.configureScope` and tracks user movement using `SentryNavigatorObserver`. Utilizing `SentryInterceptor` with the Dio library allows for automatic tracking of HTTP request performance and API bottlenecks. ### Strategic Log Level Design * **Debug and Info**: Debug logs remain local to the terminal to save resources. Info logs are reserved for significant user actions that change data, such as successful sign-ups or purchases, while high-frequency read actions like "viewing a product list" are excluded to reduce noise and costs. * **Warning**: This level tracks external system failures, such as failed API calls or push notification losses. To prevent "alert fatigue," client-side network issues (e.g., timeouts or offline status) are ignored, and alerts are triggered only when specific thresholds are met, such as 100 failures within 10 minutes. * **Error**: Error logs represent internal logic failures that bypass defensive coding, such as null object errors, parsing failures, or unreachable code branches. These require immediate notification to the development team to facilitate rapid hotfixes. * **Fatal**: This level is dedicated to application crashes and unhandled exceptions. When configured at the app's entry point, the system automatically captures these critical failures to provide a comprehensive "crash-free users" metric. ### Creating Effective Dashboards * **Naming Conventions**: Logs should follow a strict structure, using tags for modules and event names (e.g., `[API] [postLogin] success`). This consistency allows for granular querying and clearer visualization on monitoring dashboards. * **Data Enrichment**: Using the `extra` field in log events provides vital context for troubleshooting, such as including the specific endpoint, request body, and response status code for a failed transaction. * **Actionable Metrics**: Effective monitoring focuses on key performance indicators like API error rates and the failure percentage of core business events (login, registration, payment) rather than just raw crash counts. A robust monitoring strategy shifts the focus from simple crash reporting to comprehensive service health. By standardizing log levels and automating event collection, development teams can distinguish between transient network blips and critical logic errors, ensuring they spend their time fixing high-impact issues.

MonitoringFlutterSentryError Tracking+3

datadog·Jun 1, 2023·3 min readCurated summary

2023-03-08 incident: A deep dive into our incident response

Datadog’s March 8, 2023 global outage tested an incident-response process designed for large-scale failures. The company’s monitoring, on-call structure, training, and blameless culture enabled a coordinated response, but the incident also exposed challenges in diagnosing and managing a rapidly evolving, global outage. Datadog’s central lesson is that effective response depends less on rigid runbooks than on preparation, clear ownership, autonomous decision-making, and continuous learning. ## Datadog’s Incident Response Model - Datadog follows a “you build it, you own it” operating model. - Teams instrument their services extensively and configure monitors to detect problems around the clock. - Independent, out-of-band monitoring checks Datadog’s APIs from outside its infrastructure, ensuring that monitoring still works if Datadog itself becomes unavailable. - Slack channels are automatically created for incidents to provide shared situational awareness and enable additional engineers to contribute. ## Handling High-Severity Incidents - Senior engineers rotate on call for incidents involving substantial customer impact or multiple teams. - The first senior responder becomes the incident commander and retains overall responsibility. - A communications lead may manage internal updates and coordination. - For the most serious incidents, an engineering executive and customer-support manager join to provide leadership, business context, and customer-facing communication. - The incident commander remains accountable for coordinating the overall response. ## Preparation, Training, and Postmortems - Datadog uses a relatively low threshold for declaring incidents, giving engineers frequent practice with its response process. - Engineers complete incident-response training before joining an on-call rotation and repeat refresher training every six months. - Training covers on-call responsibilities, response roles, and blameless investigation practices. - Every high-severity incident receives a detailed postmortem focused on preventing recurrence. - Automation prompts responders to begin postmortems while the incident is still fresh. ## Autonomy and a Blameless Culture - Because large systems change constantly, detailed recovery procedures can quickly become outdated. - Datadog therefore gives engineers authority to choose the best response based on their knowledge of the affected services. - The company treats failures as weaknesses in systems rather than evidence of individual fault. - Blamelessness is intended to encourage creativity, honesty, and effective decision-making under pressure. ## The March 8 Outage - A systemd upgrade began around 06:00 UTC and ultimately triggered the outage. - Monitoring detected the problem within three minutes, and engineering teams were paged shortly afterward. - A high-severity incident was declared at 06:18, with an incident commander joining five minutes later. - The first public status update was posted at 06:31, and the outage was officially diagnosed as global at 06:32. - By 07:20, responders identified a Kubernetes failure and unhealthy intake systems as central problems. - Engineers confirmed by 08:00 that the Kubernetes failure was not spreading to additional or newly provisioned nodes. - A working mitigation for the EU1 region was found by 08:30. - Most US1 compute capacity recovered automatically by 11:00, while teams began organizing a longer recovery effort. - At 11:36, unattended upgrades were identified as the triggering event. - Compute capacity in EU1—the first step toward recovery—was restored by 12:05. ## Practical Lessons Datadog’s experience demonstrates the value of independent monitoring, practiced incident roles, rapid communication, and empowered responders. Organizations operating complex systems should regularly rehearse incident management, invest in resilient observability outside the primary platform, and use blameless postmortems to turn major outages into improvements.

Read original(opens in new tab)

MonitoringKubernetesDatadogAutomation+4

datadog·Jun 1, 2023Original article

2023-03-08 incident: A deep dive into our incident response | Datadog (opens in new tab)

Datadog’s first global outage on March 8, 2023, served as a rigorous stress test for their established incident response framework and "you build it, you own it" philosophy. While the outage was triggered by a systemic failure during a routine systemd upgrade, the company's commitment to blameless culture and decentralized engineering autonomy allowed hundreds of responders to coordinate a complex recovery across multiple regions. Ultimately, the event validated their investment in out-of-band monitoring and rigorous, bi-annual incident training as essential components for managing high-scale system disasters. ## Incident Response Structure and Philosophy * Datadog employs a decentralized "you build it, you own it" model where individual engineering teams are responsible for the 24/7 health and monitoring of the services they build. * For high-severity incidents, a specialized rotation is paged, consisting of an Incident Commander to lead the response, a communications lead, and a customer liaison to manage external messaging. * The organization prioritizes "people over process," empowering engineers to use their judgment to find creative solutions rather than following rigid, pre-written playbooks that may not apply to unprecedented failures. * A blameless culture is strictly maintained across all levels of the company, ensuring that post-incident investigations focus on systemic improvements rather than assigning fault to individuals. ## Multi-Layered Monitoring Strategy * Standard telemetry provides internal visibility, but Datadog also maintains "out-of-band" monitoring that operates completely outside its own infrastructure. * This out-of-band system interacts with Datadog APIs exactly like a customer would, ensuring that engineers are alerted even if the internal monitoring platform itself becomes unavailable. * Communication is streamlined through a dedicated Slack incident app that automatically generates coordination channels, providing situational awareness to any engineer who joins the effort. ## Anatomy of the March 8 Outage * The outage began at 06:00 UTC, triggered by a systemd upgrade that caused widespread Kubernetes failures and prevented pods from restarting correctly. * The global nature of the outage was diagnosed within 32 minutes of the initial monitoring alerts, leading to the activation of executive on-calls and the customer support management team. * Responders identified "unattended upgrades" as the incident trigger approximately five and a half hours after the initial failure. * Recovery was executed in stages: compute capacity was restored first in the EU1 region, followed by the US1 region, with full infrastructure restoration completed by 19:00 UTC. Organizations should treat incident response as a perishable skill that requires constant practice through a low threshold for declaring incidents and regular training. By combining out-of-band monitoring with a culture that empowers individual engineers to act autonomously during a crisis, teams can more effectively navigate the "not if, but when" reality of large-scale system failures.

MonitoringKubernetesDatadogAutomation+4

datadog·Apr 2, 2019·3 min readCurated summary

Building highly reliable data pipelines at Datadog

Datadog’s approach to reliable data pipelines focuses on delivering correct data on time, even when individual jobs fail. Reliability therefore requires fault tolerance, monitoring, and fast recovery rather than eliminating every failure. The company achieves this through isolated, short-lived clusters and pipelines designed to limit the impact of failures. ## Reliability Means Timely, Correct Results - A reliable pipeline is one that consistently produces correct outputs within the required time window. - Occasional crashes do not necessarily make a pipeline unreliable if automatic recovery still delivers the data on schedule. - Pipelines should be designed with the expectation that failures will eventually occur. - Monitoring must detect unexpected failures early, while operational processes should support rapid recovery. ## Architecture for Batch Pipelines - Datadog streams and analyzes live data in real time but uses batch pipelines for features such as optimized long-term storage. - Historical data is stored in object storage. - Cloud Hadoop/Spark services launch and configure processing clusters. - Luigi workers manage tasks and workflows, while Spark workers compile code and submit jobs. - Jobs can be launched through a web interface, command line, or scheduler. ## One Cluster per Pipeline Instead of placing all workloads on one large Hadoop cluster, Datadog gives each pipeline its own cluster. - **Isolation:** Jobs do not compete for resources or interfere with one another, simplifying monitoring and diagnosis. - **Workload-specific hardware:** Clusters can use CPU-optimized or memory-optimized instances depending on the job. - **Elastic scaling:** Clusters can be expanded to catch up with delays or handle growing data volumes without waiting for a shared cluster. - **Safer upgrades:** Hadoop and Spark versions can be upgraded gradually across separate clusters. - Clusters are typically short-lived, averaging about three hours, although dozens may run simultaneously. ## Using Spot Instances to Encourage Fault Tolerance - AWS spot instances can reduce infrastructure costs by as much as 80%, but their nodes may be terminated whenever capacity or demand changes. - Rather than avoiding this failure mode, Datadog designs pipelines to tolerate disappearing clusters. - Long-running jobs are risky because failures discard more work and make recovery slower. - Pipelines are split into smaller jobs: - **Vertically:** Separate transformations into multiple stages, persisting intermediate results in S3. - **Horizontally:** Partition input data so multiple jobs process different portions concurrently. ## Breaking Up the Rollup Pipeline - Datadog’s rollup pipeline generates aggregated time-series data for historical metrics queries. - A single job would take more than 14 hours, making failures costly and difficult to recover from. - The pipeline is divided into two stages: - Aggregate high-resolution data and checkpoint it to S3 as Parquet files. - Convert the intermediate data into a custom format optimized for queries. - As these jobs grew, they were partitioned further using Kafka’s partitioning scheme. - Kafka partitions are grouped into shards, allowing Datadog to: - Adjust how much data each job processes. - Run more or fewer jobs as needed. - Isolate unusually large or sensitive shards. - This decomposition adds overhead because launching jobs and checkpointing to S3 take extra time, but it substantially limits the work lost during failures. ## Practical Recommendation Design pipelines around failure rather than assuming uninterrupted execution. Use isolated, scalable clusters, short jobs, intermediate checkpoints, and partitioned processing so that failures affect only a small portion of the workload and recovery remains fast.

Read original(opens in new tab)

MonitoringDatabase DesignPostgreSQLAWS+4

datadog·Apr 2, 2019·1 min readCurated summary

Building highly reliable data pipelines at Datadog | Datadog

The provided text does not include the blog post’s actual article content. It contains Datadog’s navigation menu and a link titled “Highly Reliable Data Pipelines,” so the post’s argument, architecture, and technical conclusions cannot be summarized reliably. ## Available Information - The page appears to be a Datadog engineering blog post about building highly reliable data pipelines. - The surrounding content is primarily Datadog product navigation. - It also promotes Datadog’s recognition as a Leader in the 2026 Gartner® Magic Quadrant™ for Observability Platforms. Please provide the article body or a complete page extract for a detailed technical summary.

Read original(opens in new tab)

MonitoringKubernetesDatadogObservability+3

datadog·Aug 23, 2016·3 min readCurated summary

Restroom hacks

Datadog built an office bathroom-availability monitor to reduce contention without compromising privacy or existing door functionality. Raspberry Pi 2 devices, GPIO-connected sensors, and simple Unix tools provided a low-maintenance way to report whether bathrooms were occupied. The project showed that the hardest parts were adapting to varied real-world hardware, mounting sensors cleanly, and dealing with unreliable Wi-Fi—not writing software. ## Project Goals - Avoid intrusive monitoring: - No cameras or sensors that could feel invasive. - Provide reliable occupancy information with minimal false positives and negatives. - Use door-lock status where possible as the occupancy signal. - Avoid interfering with existing locks and doors. - Keep devices secure, professional-looking, easy to maintain, and remotely updateable. - Treat the project as a fun hardware experiment. ## Adapting to Different Bathrooms - Bathrooms differed significantly in: - Lock styles, including push-button handles and rotary stall locks. - Number of rooms or stalls. - Availability and location of power outlets. - Wi-Fi quality, especially near concrete walls and older electrical equipment. - These variations required different sensor designs rather than one universal installation. ## Raspberry Pi and Sensor Hardware - Raspberry Pi 2 Model Bs served as the project’s controllers because they: - Ran Linux. - Supported Wi-Fi and SSH administration. - Were compact enough to conceal. - The team used several sensor types: - Magnetic reed switches for detecting door position. - Pin switches for detecting sliding stall-lock positions. - Photoresistors were purchased as a possible way to detect darkness but were not needed in the MVP. - For push-button locks, reed switches detected whether the door was open or closed. Although this could theoretically misreport a closed but unoccupied bathroom, it worked reliably in practice. - Stall-lock sensors were hidden inside hollow metal panels. Automotive-style pin switches were mounted using simple carved wooden blocks that contacted the sliding lock without obstructing it. - Wiring was concealed in wiremolding, with Raspberry Pis placed inside outlet boxes where possible. ## GPIO and Unix-Based Monitoring - Raspberry Pi GPIO pins were accessed through files in `/sys/class/gpio/`. - A Python script read sensor values and translated them into bathroom availability. - Configuration handled differences between normally open and normally closed sensors. - The service was exposed through `tcpserver` and managed with `daemontools`. - A basic command-line client could query status with Netcat, for example: ```sh nc 11.bathrooms.datadog-internal.com 50 ``` ## Making Availability Easy to Use - Employees could check status from the command line. - Some added bathroom availability to TextBar. - Datadog dashboards displayed bathroom status throughout the New York office. - The implementation required very little code; most effort went into sensor selection, physical installation, and network troubleshooting. The project demonstrates that inexpensive, hackable hardware combined with simple Linux tools can solve a practical office problem. For similar systems, prioritize non-intrusive sensors, flexible installation designs, and secure remote management; the resulting software can remain remarkably small.

Read original(opens in new tab)

MonitoringOpen SourcePythonNetwork Security+4

datadog·Aug 11, 2016·3 min readCurated summary

Consul at Datadog

Consul has become a critical part of Datadog’s production infrastructure for distributing configuration and discovering services. After 18 months of use, the main lesson is that Consul requires careful capacity planning, controlled access, efficient query patterns, and continuous monitoring. The recommendations aim to keep clusters stable while supporting frequent configuration updates and high-volume service discovery. ## Consul Server Capacity and CPU Consul servers use Raft consensus to elect a leader and coordinate the cluster. - Followers trigger a leadership transition if they cannot hear from the leader for 500 milliseconds. - Frequent leadership transitions usually indicate insufficient CPU capacity. - Datadog’s approximate sizing guidance: - `m3.large`: about 300 agent nodes - `c3.xlarge`: about 500 agent nodes - `c3.2xlarge`: about 800 agent nodes - If transitions occur hourly or more often, increase server CPU capacity until they happen no more than daily. - Standard monitoring may miss brief 500-millisecond CPU spikes, so reducing CPU pressure is important even when dashboards look normal. ## Auditable Configuration Changes Consul’s key-value store is useful for distributing configuration throughout a cluster. - Configuration can be retrieved through HTTP or delivered through Consul watches. - Direct edits without an audit trail make it difficult to determine who changed a value and when. - `git2consul` distributes configuration from a Git repository, providing version control and accountability. - Datadog uses it for cluster-wide configuration updates roughly every 60 seconds, dozens of times per day. ## Access Control with ACLs Consul ACLs prevent unauthorized processes from modifying or deleting key-value data. - Tokens should be limited to the data and operations each process requires. - Scoped permissions reduce the impact of accidental changes. - ACLs provide an important safety boundary between services and configuration areas. ## Watches Instead of Excessive Polling Consul can handle substantial traffic, but it should not be queried hundreds of thousands of times per second like Redis or Memcached. - Watches notify clients when key-value data changes. - This reduces unnecessary polling and distributes updates efficiently. - Watches can sometimes trigger unexpectedly or too frequently. - Tools such as `sifter` can help protect systems from excessive watch activity. ## Using dnsmasq for Service Discovery Applications using Consul’s DNS interface can reduce load by placing `dnsmasq` between clients and Consul. - Use short DNS TTLs; Datadog commonly uses 10 seconds. - Query `dnsmasq` rather than Consul directly so repeated answers can be cached locally. - At very high request volumes, cache Consul services in an additional hosts file loaded by `dnsmasq`. - This setup served over 100,000 DNS requests per second while sending only about 400 requests per second directly to Consul. - `goshe` can collect `dnsmasq` statistics for monitoring. ## Monitoring Cluster Health Monitoring is essential for operating Consul reliably. - `consul.consul.leader.reconcile.count` should remain stable and indicate that a leader exists. - `consul.serf.events.consul_new_leader` shows leadership transitions; frequent events suggest instability. - `consul.raft.leader.lastContact` measures how recently nodes contacted the leader. - `consul.consul.dns.domain_query.count` reveals how many DNS requests are reaching Consul directly. - Also monitor CPU and network usage on Consul servers. Consul works best when server nodes have sufficient CPU, configuration changes are managed through version control, ACLs restrict access, watches replace aggressive polling, and `dnsmasq` absorbs service-discovery traffic. Continuous monitoring of leadership, Raft connectivity, DNS load, CPU, and networking helps identify failures before they affect production.

Read original(opens in new tab)

MonitoringMicroservicesConfiguration ManagementConsul+4