spotifyJul 20, 2026

Content Ingestion & Podcast Video Incident Report | Spotify Engineering (opens in new tab)

Spotify’s June 24 publishing incident delayed video podcast episodes for several hours after transcoding capacity was overwhelmed. The disruption resulted from insufficient capacity headroom, a concurrent batch job, increased processing costs, and a scheduling bug that reduced throughput by about 10%. Spotify cleared the backlog by the following morning and has since expanded capacity, improved monitoring, and launched a broader publishing reliability initiative. ## What Caused the Delay - A spike in new video podcast submissions pushed transcoding infrastructure close to its maximum capacity. - A scheduled batch job was reprocessing existing episodes, consuming capacity needed for new publications. - Recent video-quality improvements increased the processing time and compute required per episode. - A resource-scheduling bug following an infrastructure migration underused available hardware, reducing throughput by approximately 10%. - Creators re-uploaded missing episodes, unintentionally adding more load because Spotify did not clearly confirm that uploads had been received and queued. ## Incident Timeline and Response - Early monitoring alerts fired at 13:30 UTC, but they were not initially recognized as a broader capacity problem. - At 15:00, a delivery spike brought transcoding close to capacity. - The batch job was stopped at 16:35 to free resources. - Formal incident response began at 17:34 after queue thresholds were exceeded. - A scheduling fix was deployed at 20:49, and an additional processing cluster came online at 00:14 on June 25. - All queues were cleared by 01:02, with normal pipeline operation confirmed at 07:30. - Approximately four hours passed between the first alerts and formal incident response, highlighting a major monitoring and escalation gap. ## Remediation Measures - Spotify increased transcoding capacity by approximately 67% to provide more room for traffic spikes and batch processing. - The resource-scheduling bug was fixed, restoring use of previously underutilized compute capacity. - Monitoring was improved to warn earlier when capacity is approaching operational limits. ## Broader Reliability Program - Capacity planning is being expanded to account for burst traffic, background workloads, and recovery needs—not only typical steady-state demand. - Publishing systems will prioritize real-time creator uploads over background operations. - Spotify is extending rate limiting and backpressure mechanisms across the pipeline so unexpected load can be handled more gracefully. - A dedicated cross-team effort is coordinating these improvements across the publishing infrastructure. Spotify’s main lesson is that reliable publishing requires both additional capacity and better operational safeguards. Earlier detection, clearer upload acknowledgments, stronger workload prioritization, and deliberate burst-capacity planning should reduce the likelihood and impact of similar incidents.

incident-responsemonitoringrate-limitingvideo-transcoding+4

githubJun 29, 2026

Inside the Advisory Database and what happens when vulnerability volume breaks records (opens in new tab)

Madison Ficorilli is a vulnerability transparency advocate and senior security manager at GitHub. She leads the advisory database curation team and contributes to vulnerability reporting, response, and disclosure through several industry organizations. Her perspective combines current leadership experience with prior roles in incident response and vulnerability coordination. ## Leadership at GitHub - Leads GitHub’s advisory database curation team. - Focuses on improving vulnerability transparency and the quality of security advisory information. ## Industry and Open Source Security Work - Co-chairs a relevant Open Source Security Foundation (OpenSSF) working group. - Serves on the CVE Program Board. - Advocates for effective vulnerability reporting, response, and disclosure practices. ## Professional Background - Previously worked as a product incident response analyst at GitHub. - Served as a vulnerability coordinator at the CERT Coordination Center at Carnegie Mellon University’s Software Engineering Institute. Her career reflects deep expertise across vulnerability coordination, incident response, database curation, and security disclosure policy.

incident-responsegithubvulnerability-managementcve+2

lineMay 29, 2026

ODW #8: A Hands-On Internal Workshop on Accelerating Incident Response and FAQ Generation with Slack MCP (opens in new tab)

LY Corporation’s Orchestration Development Workshop demonstrated how Slack MCP can turn scattered Slack conversations into structured operational knowledge. Using AI, employees can summarize incidents, generate reports, create FAQs, and publish results to Confluence with minimal setup. The workshop’s central conclusion was that hands-on practice, reusable skills, and human review are essential for turning new AI capabilities into practical workplace tools. ## The Information and Adoption Challenge - Slack contains valuable real-time information from incident response, customer inquiries, and project discussions. - Much of this information remains unstructured because employees lack time to document it. - Documentation quality varies by author, causing useful knowledge to become difficult to find. - Although Slack MCP became available internally in March 2026, adoption barriers remained: - Limited time to explore new tools - Concerns about complex configuration - Slow internal distribution of technical knowledge ## Introducing Slack MCP Through Hands-On Practice - Slack MCP is an internally developed MCP server connected to company authentication. - Employees can access internal Slack data without issuing personal tokens or configuring OAuth. - The workshop began with a simple exercise: - Launch a coding tool such as Claude Code - Ask the AI to post “Hello” in a designated Slack channel - Confirm that the message was actually posted - This immediate success helped participants understand MCP’s practical capabilities. ## Combining Slack MCP with Other MCP Servers Slack MCP supports several core operations: - Reading messages and threads - Posting messages and performing actions - Looking up channels and members - Searching Slack content Combined with other MCP servers, it can support broader workflows: - Slack plus Confluence MCP: Generate and publish project reports or FAQs - Slack plus Jira MCP: Create work tickets from discussions - Slack conversations can be transformed into structured documents rather than remaining isolated in chat history. ## Automatically Creating FAQs from Slack Inquiries The first major exercise converted repeated support discussions into reusable knowledge. - Slack inquiry threads were collected and converted into FAQ-formatted Markdown. - Existing Confluence content was checked to identify duplicates. - New FAQs were published as child pages under an existing Confluence knowledge base. - The output was formatted as a table containing: - Symptoms - Causes - Solutions The workflow was packaged into reusable skills such as: - `slack-to-faq`: Searches recent inquiry threads and generates new FAQ files - `faq-to-confluence`: Converts and publishes the FAQs to Confluence This demonstrated how MCP can automate the entire path from conversation search to knowledge-base publication. ## Supporting Incident Response The second exercise focused on reducing the time needed to understand and document incidents. ### Rapid Situation Summaries Participants could ask the AI to summarize an outage in natural language. MCP searched relevant Slack threads and organized the information into: - Current resolution status - Customer impact - Actions being taken by team members - A chronological timeline This helps managers or newly joining responders understand the situation quickly without reading every thread. ### Automated Incident Reports After resolution, the AI generated reports in a specified format, including: - Incident and detection times - Duration - Root cause - Affected users and features - Whether data was lost - Remediation steps The `slack-incident-status` and `slack-incident-report` skills separated real-time status checking from post-incident documentation. ## Practical Guidelines and Safeguards - Clean and constrain source data before processing it: - Compare results with existing Confluence FAQs - Filter messages using reactions or other markers - Limit searches to relevant channels and threads - Do not publish AI-generated documents without review. - Check for personal information and confirm that the output accurately reflects the source conversations. - Include links or references to the original Slack threads. - Specify the desired output structure, such as a three-column table for symptoms, causes, and solutions. - Convert successful prompts into reusable skills so teams can avoid rewriting complex instructions and maintain consistent output quality. ## Lessons from the Workshop - **Timing matters:** Holding the workshop soon after Slack MCP became available captured user interest and accelerated experimentation. - **Practice is more effective than explanation:** Starting with a simple Slack post and progressing to FAQs and incident reports made the benefits immediately tangible. - **Real work makes training relevant:** Inquiry handling and incident response were chosen because they are common, time-consuming tasks. - **Reusable skills improve adoption:** Prompt patterns were tested manually, refined, saved as skills, and shared with participants for continued workplace use. The recommended approach is to introduce new AI tools through timely, task-focused workshops, then refine successful workflows into shared skills. MCP can greatly reduce the effort of operational documentation, but human validation remains necessary before generated knowledge is published.

incident-responsegen-aimodel-context-protocolslack+4

discordJul 20, 2025

ROOST Announces “Coop” and “Osprey”: Free, Open-Source Trust and Safety Infrastructure for the AI Era (opens in new tab)

ROOST, a non-profit dedicated to digital safety, has launched two open-source tools, Coop and Osprey, to provide enterprise-grade content moderation and threat investigation capabilities to organizations of all sizes. By open-sourcing technology previously developed by industry leaders like Discord and Cove, ROOST aims to democratize access to the infrastructure required to detect, triage, and respond to online harms. This initiative shifts Trust and Safety from a proprietary competitive advantage to a shared public resource, enabling platforms to prioritize user protection without the burden of expensive enterprise software. ### Content Review and Compliance with Coop Built on technology acquired from Cove and utilized by platforms like Notion, Coop focuses on the human-in-the-loop aspect of content moderation. * The platform provides robust tools for content review, allowing teams to route specific cases to subject-matter experts for deeper analysis. * It includes built-in integration with the National Center for Missing & Exploited Children’s (NCMEC) API, automating the mandatory reporting process for child sexual abuse material (CSAM). * The interface is designed to surface relevant context and metadata, ensuring moderators can make informed decisions and take immediate action against policy violations. ### Incident Response and Investigation with Osprey Osprey is a lightweight investigation tool originally developed by Discord to manage large-scale safety incidents and platform-wide threats. * It serves as a foundation for incident response, helping safety teams understand platform trends and investigate coordinated threats like phishing or harassment campaigns. * The tool is designed to be user-friendly and accessible for grassroots communities while remaining powerful enough for established platforms. * Early adopters, including the decentralized social network Bluesky, are implementing Osprey to demonstrate that effective safety infrastructure can be scalable and resource-efficient. ### A Collaborative Model for Safety Infrastructure The launch of these tools represents a strategic shift toward a collaborative "public-interest" model for digital defense. * ROOST acquired the intellectual property of Cove and received the donation of Osprey from Discord to ensure these tools remain available as a public good. * The initiative is backed by philanthropic funding and legal support from Perkins Coie, removing the financial barriers that often prevent smaller platforms from implementing high-level safety measures. * Major industry players like Notion and Bluesky are championing the move, signaling an industry-wide push to share safety innovations rather than silo them. Platforms and developers should prepare to integrate these tools into their safety stacks as they become publicly available in the coming months. By adopting open-source infrastructure for routine tasks like NCMEC reporting and incident triage, organizations can focus their internal resources on platform-specific innovations while maintaining a high standard of digital safety.

incident-responseopen-sourceapi-integrationtrust-and-safety+3

datadogJun 3, 2025

Detecting faulty deployments: Our journey from unlabeled data to supervised learning | Datadog (opens in new tab)

The supplied content does not include the blog post itself; it contains Datadog’s navigation menu and a link titled “Detecting Faulty Deployments.” As a result, there is not enough information to accurately summarize the article’s arguments, implementation details, or conclusions. ## Available context - The linked article appears to concern identifying deployments that introduce faults or regressions. - Datadog’s platform covers related capabilities such as: - Application Performance Monitoring - Metrics and infrastructure monitoring - Logs and error tracking - CI Visibility and software delivery monitoring - Service-level objectives and incident response - The page also promotes Datadog’s recognition as a Leader in the Gartner Magic Quadrant for Observability Platforms. ## Missing information - The article’s detection methodology - Metrics, queries, or deployment signals used - Alerting, rollback, or remediation procedures - Technical examples and conclusions Please provide the article text or a page extract containing the post body for an accurate summary.

incident-responsek8sdatadogobservability+4

datadogJan 15, 2025

Unraveling a Postgres segfault that uncovered an Arm64 JIT compiler bug (opens in new tab)

Postgres was crashing with segmentation faults when executing certain expensive queries on an Arm64 Kubernetes cluster. Investigators reduced the failure to a simple table scan and discovered that disabling JIT compilation prevented the crash. Assembly-level debugging ultimately traced the problem to a bug in LLVM’s Arm64 JIT support. ## Isolating the Crash - The failures occurred across multiple EC2 nodes, ruling out faulty hardware. - Query logs showed that the crashes consistently followed a small number of query patterns. - The simplest reproducer was: ```sql SELECT repo_id FROM repository; ``` - Core dumps had badly corrupted stacks, but surviving frames pointed to `ExecRunCompiledExpr`, suggesting a failure during JIT execution. - The unusually short backtraces reinforced the suspicion that the stack itself had been corrupted. ## How PostgreSQL JIT Works - PostgreSQL normally evaluates SQL expressions through a general-purpose interpreter. - JIT compilation converts expressions such as `1+1` into native machine code, reducing interpreter overhead for large workloads. - JIT can also optimize tuple deforming by converting disk tuples into in-memory values more efficiently. - PostgreSQL uses LLVM to generate the compiled code. - Because compilation adds overhead and compiled functions are not reused between queries, PostgreSQL enables JIT primarily for expensive queries based on cost thresholds. ## The Query of Death - The affected query scanned a partitioned `repository` table with: - 64 partitions - More than 1.6 million rows - 128 JIT-generated functions - Its query plan enabled expression compilation and tuple deforming: ```text JIT: Functions: 128 Expressions: true Deforming: true Inlining: false Optimization: false ``` - Running the query with: ```sql SET jit = off; ``` completed successfully. - Disabling JIT cluster-wide immediately stopped the crashes without noticeable query-latency effects. ## Root Cause Direction - The release build of PostgreSQL offered limited debugging flexibility, so the team planned to reproduce the failure in a dedicated test environment. - Further investigation eventually isolated the defect to JIT compilation on Arm64 systems. - The underlying issue was identified as an LLVM bug rather than a PostgreSQL query or hardware problem. - The investigation continued down to generated assembly and resulted in an upstream fix. The immediate mitigation was to disable PostgreSQL JIT, while the durable solution was to adopt the LLVM fix addressing the Arm64 code-generation bug.

incident-responsedatabase-designk8spostgresql+3

datadogJun 1, 2023

2023-03-08 incident: A deep dive into our incident response (opens in new tab)

Datadog’s March 8, 2023 global outage tested an incident-response process designed for large-scale failures. The company’s monitoring, on-call structure, training, and blameless culture enabled a coordinated response, but the incident also exposed challenges in diagnosing and managing a rapidly evolving, global outage. Datadog’s central lesson is that effective response depends less on rigid runbooks than on preparation, clear ownership, autonomous decision-making, and continuous learning. ## Datadog’s Incident Response Model - Datadog follows a “you build it, you own it” operating model. - Teams instrument their services extensively and configure monitors to detect problems around the clock. - Independent, out-of-band monitoring checks Datadog’s APIs from outside its infrastructure, ensuring that monitoring still works if Datadog itself becomes unavailable. - Slack channels are automatically created for incidents to provide shared situational awareness and enable additional engineers to contribute. ## Handling High-Severity Incidents - Senior engineers rotate on call for incidents involving substantial customer impact or multiple teams. - The first senior responder becomes the incident commander and retains overall responsibility. - A communications lead may manage internal updates and coordination. - For the most serious incidents, an engineering executive and customer-support manager join to provide leadership, business context, and customer-facing communication. - The incident commander remains accountable for coordinating the overall response. ## Preparation, Training, and Postmortems - Datadog uses a relatively low threshold for declaring incidents, giving engineers frequent practice with its response process. - Engineers complete incident-response training before joining an on-call rotation and repeat refresher training every six months. - Training covers on-call responsibilities, response roles, and blameless investigation practices. - Every high-severity incident receives a detailed postmortem focused on preventing recurrence. - Automation prompts responders to begin postmortems while the incident is still fresh. ## Autonomy and a Blameless Culture - Because large systems change constantly, detailed recovery procedures can quickly become outdated. - Datadog therefore gives engineers authority to choose the best response based on their knowledge of the affected services. - The company treats failures as weaknesses in systems rather than evidence of individual fault. - Blamelessness is intended to encourage creativity, honesty, and effective decision-making under pressure. ## The March 8 Outage - A systemd upgrade began around 06:00 UTC and ultimately triggered the outage. - Monitoring detected the problem within three minutes, and engineering teams were paged shortly afterward. - A high-severity incident was declared at 06:18, with an incident commander joining five minutes later. - The first public status update was posted at 06:31, and the outage was officially diagnosed as global at 06:32. - By 07:20, responders identified a Kubernetes failure and unhealthy intake systems as central problems. - Engineers confirmed by 08:00 that the Kubernetes failure was not spreading to additional or newly provisioned nodes. - A working mitigation for the EU1 region was found by 08:30. - Most US1 compute capacity recovered automatically by 11:00, while teams began organizing a longer recovery effort. - At 11:36, unattended upgrades were identified as the triggering event. - Compute capacity in EU1—the first step toward recovery—was restored by 12:05. ## Practical Lessons Datadog’s experience demonstrates the value of independent monitoring, practiced incident roles, rapid communication, and empowered responders. Organizations operating complex systems should regularly rehearse incident management, invest in resilient observability outside the primary platform, and use blameless postmortems to turn major outages into improvements.

incident-responsek8sdatadogautomation+4

datadogJun 1, 2023

2023-03-08 incident: A deep dive into our incident response | Datadog (opens in new tab)

Datadog’s first global outage on March 8, 2023, served as a rigorous stress test for their established incident response framework and "you build it, you own it" philosophy. While the outage was triggered by a systemic failure during a routine systemd upgrade, the company's commitment to blameless culture and decentralized engineering autonomy allowed hundreds of responders to coordinate a complex recovery across multiple regions. Ultimately, the event validated their investment in out-of-band monitoring and rigorous, bi-annual incident training as essential components for managing high-scale system disasters. ## Incident Response Structure and Philosophy * Datadog employs a decentralized "you build it, you own it" model where individual engineering teams are responsible for the 24/7 health and monitoring of the services they build. * For high-severity incidents, a specialized rotation is paged, consisting of an Incident Commander to lead the response, a communications lead, and a customer liaison to manage external messaging. * The organization prioritizes "people over process," empowering engineers to use their judgment to find creative solutions rather than following rigid, pre-written playbooks that may not apply to unprecedented failures. * A blameless culture is strictly maintained across all levels of the company, ensuring that post-incident investigations focus on systemic improvements rather than assigning fault to individuals. ## Multi-Layered Monitoring Strategy * Standard telemetry provides internal visibility, but Datadog also maintains "out-of-band" monitoring that operates completely outside its own infrastructure. * This out-of-band system interacts with Datadog APIs exactly like a customer would, ensuring that engineers are alerted even if the internal monitoring platform itself becomes unavailable. * Communication is streamlined through a dedicated Slack incident app that automatically generates coordination channels, providing situational awareness to any engineer who joins the effort. ## Anatomy of the March 8 Outage * The outage began at 06:00 UTC, triggered by a systemd upgrade that caused widespread Kubernetes failures and prevented pods from restarting correctly. * The global nature of the outage was diagnosed within 32 minutes of the initial monitoring alerts, leading to the activation of executive on-calls and the customer support management team. * Responders identified "unattended upgrades" as the incident trigger approximately five and a half hours after the initial failure. * Recovery was executed in stages: compute capacity was restored first in the EU1 region, followed by the US1 region, with full infrastructure restoration completed by 19:00 UTC. Organizations should treat incident response as a perishable skill that requires constant practice through a low threshold for declaring incidents and regular training. By combining out-of-band monitoring with a culture that empowers individual engineers to act autonomously during a crisis, teams can more effectively navigate the "not if, but when" reality of large-scale system failures.

incident-responsek8sdatadogslack+4

figmaDec 14, 2022

How Linear made the most of a DDoS | Figma Blog (opens in new tab)

Linear’s website went down during a DDoS attack shortly after a major redesign launch. Rather than leave visitors at a login page, the team turned the crisis into an opportunity by publishing the redesign directly as a Figma file. The unexpected workaround generated significant attention and demonstrated the value of fast, creative incident response. ## The Redesign and Its Complexity - Linear’s team had spent months exploring ideas for the new website. - The Figma file contained thousands of iterations, frames, large images, and design inspirations. - The file became so large that Figma warned it was approaching the browser’s memory limits. - Although the redesign took months to develop, the final work came together during the last 24 hours before launch. ## The DDoS Attack - On October 13, 2022, Linear’s homepage became unavailable. - The team initially suspected that the new redesign had introduced a technical problem. - Investigation revealed that the outage was caused by a distributed denial-of-service attack overwhelming the site with traffic. - Their immediate priority was restoring access to the application, so visitors were redirected from site pages directly to the login page. ## Turning the Outage into an Alternative Homepage - The direct login redirect restored app access but made the public launch feel anticlimactic. - As social media discussions about the redesign continued, Jori Lallo suggested publishing the Figma design file itself. - Paco Coursey and Edgar Ambartsoumian were initially hesitant, but Jori encouraged them to proceed. - The Figma file became an unconventional replacement for the unavailable homepage and attracted widespread attention online. The incident shows how a team can respond constructively under pressure: stabilize critical functionality first, then use creativity to preserve the user-facing experience when the normal solution is unavailable.

incident-responsefigmaddosweb-application-security+1

figmaJun 14, 2022

Postmortem: Service disruptions on June 6 & 7 2022 | Figma Blog (opens in new tab)

Figma experienced four service disruptions between June 6–7, 2022, lasting from seven minutes to 1 hour 20 minutes. Existing files remained usable, but users could not open new files or collaborate; no data was lost. The root cause was a rare AWS ElastiCache/Redis bug that saturated the Engine CPU of a Cluster-Mode Enabled node under heavy Publish/Subscribe load. ## Impact and Timeline - Incidents occurred between 11:34 PM PDT on June 6 and 10:43 AM PDT on June 7. - The web application became fully functional at approximately 10:41 AM; some API features were restored later. - Users could continue working in already-open files. - Opening files and real-time collaboration were unavailable during disruptions. - Local changes were preserved and synchronized when service connectivity returned. ## Redis and ElastiCache Architecture - Figma uses AWS ElastiCache, a managed Redis service, for: - Caching frequently accessed data - Routing messages between services - Its setup included: - A Cluster-Mode Disabled instance - A Cluster-Mode Enabled instance, which supports horizontal scaling by adding Redis nodes - Several weeks before the outage, Figma moved its Redis Publish/Subscribe traffic from the CMD instance to the CME instance. - The workload had operated normally for weeks, with no obvious traffic or usage changes immediately before the incidents. ## Root Cause: Engine CPU Saturation - Monitoring alerted Figma within seconds that one CME ElastiCache node had reached 100% Engine CPU. - AWS later identified a rare Redis bug triggered by high Publish/Subscribe traffic on CME clusters. - Figma could not obtain CPU profiles from the underlying ElastiCache machines, making it difficult to identify the exact operation consuming CPU. - Increasing cluster capacity did not solve the problem and ultimately made the behavior worse. ## Mitigation and Investigation - Figma initially suspected insufficient capacity or faulty hardware. - Engineers: - Initiated a failover of the affected node - Created a larger ElastiCache cluster with more nodes - Redirected traffic to the replacement clusters when failover took too long - The first traffic redirection restored service, but the new cluster later experienced the same CPU saturation. - A subsequent failover completed successfully and restored service more quickly. - Engineers investigated and ruled out: - Routine backups or unexpected snapshots - Sudden increases in Redis command volume - Slow commands, large keys, or other obvious misuse - Scheduled background jobs - Blocking requests that appeared to be waiting longest for Redis commands did not resolve the issue. Figma’s outage was ultimately caused by an AWS Redis/ElastiCache defect rather than a data-loss event or an observable change in application traffic. The incident underscores the need to validate Redis Publish/Subscribe workloads on Cluster-Mode Enabled deployments and maintain mitigation strategies that do not rely solely on adding capacity.

incident-responseredisaws-elasticachepublish-subscribe+1

datadogApr 13, 2022

It's always DNS . . . except when it's not: A deep dive through gRPC, Kubernetes, and AWS networking | Datadog (opens in new tab)

The supplied content does not include the blog post itself; it mainly contains Datadog navigation links and a promotional banner stating that Datadog was named a Leader in Gartner’s Magic Quadrant for Observability Platforms. The URL suggests the missing article concerns a gRPC, DNS, and load-balancing incident, but no incident details are provided. ## Available Content - Datadog promotes its recognition as a Gartner Magic Quadrant Leader. - The navigation lists products across: - Infrastructure and application monitoring - Logs, databases, and data observability - Security - Digital experience monitoring - Software delivery and service management - AI and platform capabilities - The linked page path references an engineering post titled “gRPC, DNS, and Load Balancing Incident.” ## Missing Technical Details - No description of the incident or its impact - No explanation of the DNS or load-balancing failure - No timeline, root-cause analysis, or remediation steps - No lessons learned or recommendations Please provide the full article text for a substantive technical summary.

incident-responseobservabilitygrpcdns+1