spotifyJul 20, 2026

Content Ingestion & Podcast Video Incident Report | Spotify Engineering (opens in new tab)

Spotify’s June 24 publishing incident delayed video podcast episodes for several hours after transcoding capacity was overwhelmed. The disruption resulted from insufficient capacity headroom, a concurrent batch job, increased processing costs, and a scheduling bug that reduced throughput by about 10%. Spotify cleared the backlog by the following morning and has since expanded capacity, improved monitoring, and launched a broader publishing reliability initiative. ## What Caused the Delay - A spike in new video podcast submissions pushed transcoding infrastructure close to its maximum capacity. - A scheduled batch job was reprocessing existing episodes, consuming capacity needed for new publications. - Recent video-quality improvements increased the processing time and compute required per episode. - A resource-scheduling bug following an infrastructure migration underused available hardware, reducing throughput by approximately 10%. - Creators re-uploaded missing episodes, unintentionally adding more load because Spotify did not clearly confirm that uploads had been received and queued. ## Incident Timeline and Response - Early monitoring alerts fired at 13:30 UTC, but they were not initially recognized as a broader capacity problem. - At 15:00, a delivery spike brought transcoding close to capacity. - The batch job was stopped at 16:35 to free resources. - Formal incident response began at 17:34 after queue thresholds were exceeded. - A scheduling fix was deployed at 20:49, and an additional processing cluster came online at 00:14 on June 25. - All queues were cleared by 01:02, with normal pipeline operation confirmed at 07:30. - Approximately four hours passed between the first alerts and formal incident response, highlighting a major monitoring and escalation gap. ## Remediation Measures - Spotify increased transcoding capacity by approximately 67% to provide more room for traffic spikes and batch processing. - The resource-scheduling bug was fixed, restoring use of previously underutilized compute capacity. - Monitoring was improved to warn earlier when capacity is approaching operational limits. ## Broader Reliability Program - Capacity planning is being expanded to account for burst traffic, background workloads, and recovery needs—not only typical steady-state demand. - Publishing systems will prioritize real-time creator uploads over background operations. - Spotify is extending rate limiting and backpressure mechanisms across the pipeline so unexpected load can be handled more gracefully. - A dedicated cross-team effort is coordinating these improvements across the publishing infrastructure. Spotify’s main lesson is that reliable publishing requires both additional capacity and better operational safeguards. Earlier detection, clearer upload acknowledgments, stronger workload prioritization, and deliberate burst-capacity planning should reduce the likelihood and impact of similar incidents.

backpressuremonitoringincident-responserate-limiting+4

netflixJul 14, 2026

Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned (opens in new tab)

The post explains how Netflix built a real-time service topology system capable of processing millions of network-flow records per second at production scale. Its core design combines streaming ingestion, reactive backpressure, physically separate data layers, and a distributed aggregation pipeline that resolves network intermediaries into meaningful service dependencies. The system favors slightly delayed but complete updates over stale batch data or incomplete results caused by dropping records. ## The Need for Real-Time Topology - Traditional topology tools rely on hourly or daily batch processing, making their data outdated during incidents. - Netflix combines: - eBPF network flows - IPC metrics delivered through Server-Sent Events - Distributed tracing data - These sources are stored in separate graph or columnar storage layers and can be queried independently or merged. - The goal is near-real-time freshness, faster incident response, blast-radius analysis, and immediate change validation. ## Backpressure for Reliable Streaming - Processing millions of flow records per second creates a risk that downstream systems will become overwhelmed. - Common alternatives are inadequate: - Unbounded queues eventually exhaust memory. - Dropping records produces incomplete topology. - Batch processing introduces unacceptable delays. - Reactive streams propagate slowdown upstream: - A graph database signals Stage 2. - Stage 2 slows Stage 1. - Stage 1 pauses Kafka consumption. - Kafka retains the data until capacity returns. - This allows the system to degrade gracefully during traffic spikes, garbage-collection pauses, or temporary storage slowdowns. - Updates may be delayed by seconds or minutes, but the data remains substantially more complete than a dropped or hourly-processed stream. ## Physically Separate Topology Layers Netflix keeps each data source in storage optimized for its characteristics: - **Network layer:** eBPF flow logs provide broad coverage but limited application context. - **IPC layer:** Application metrics offer detailed endpoint information but cover only instrumented services. - **Tracing layer:** Parquet-based distributed traces show actual request paths but are sampled. - Separate storage enables each layer to evolve and scale independently. - Queries can run in parallel and merge results while preserving sub-second response times. ## Three-Stage Distributed Aggregation The network layer uses a distributed pipeline to transform individual network hops into logical service dependencies. - Cloud traffic commonly passes through load balancers, NAT gateways, API gateways, and proxies. - Flow logs therefore show relationships such as: - `App A → Load Balancer` - `Load Balancer → App B` - The useful topology must infer the logical dependency: `App A → App B`. ### Stage 1: Initial Flow Aggregation - Consumes flow logs from Kafka across four regions. - Filters invalid records. - Groups data into five-minute windows. - Creates initial aggregators for each window. - Uses consistent hashing to distribute aggregators. - Streams the results to Stage 2 through SSE. ### Stage 2: Intermediary Resolution - Receives the initial aggregators from Stage 1. - Groups flows by intermediary components. - Resolves multi-hop network paths into application-level relationships. - This prevents infrastructure components from dominating the resulting service graph. ## Engineering Trade-offs - Streaming provides much fresher data than batch processing but introduces greater operational and conceptual complexity. - Backpressure is essential for stability at Netflix’s scale, even though reactive pipelines are harder to reason about than synchronous systems. - The architecture prioritizes reliable, complete topology updates over perfectly immediate processing. - Production behavior differed substantially from local testing: consumers lagged, memory was exhausted, traffic became unevenly distributed, and garbage collection consumed significant resources. Netflix’s approach demonstrates that large-scale real-time topology requires streaming ingestion, end-to-end backpressure, specialized storage, and staged aggregation. For similar distributed systems, the practical recommendation is to design explicitly for overload and partial slowdown rather than relying on unbounded buffering, dropped data, or stale batch snapshots.

backpressurekafkaebpfgraph-database+4