Postmortem: Service disruptions on June 6 & 7 2022 | Figma Blog (opens in new tab)
Figma experienced four service disruptions between June 6–7, 2022, lasting from seven minutes to 1 hour 20 minutes. Existing files remained usable, but users could not open new files or collaborate; no data was lost. The root cause was a rare AWS ElastiCache/Redis bug that saturated the Engine CPU of a Cluster-Mode Enabled node under heavy Publish/Subscribe load. ## Impact and Timeline - Incidents occurred between 11:34 PM PDT on June 6 and 10:43 AM PDT on June 7. - The web application became fully functional at approximately 10:41 AM; some API features were restored later. - Users could continue working in already-open files. - Opening files and real-time collaboration were unavailable during disruptions. - Local changes were preserved and synchronized when service connectivity returned. ## Redis and ElastiCache Architecture - Figma uses AWS ElastiCache, a managed Redis service, for: - Caching frequently accessed data - Routing messages between services - Its setup included: - A Cluster-Mode Disabled instance - A Cluster-Mode Enabled instance, which supports horizontal scaling by adding Redis nodes - Several weeks before the outage, Figma moved its Redis Publish/Subscribe traffic from the CMD instance to the CME instance. - The workload had operated normally for weeks, with no obvious traffic or usage changes immediately before the incidents. ## Root Cause: Engine CPU Saturation - Monitoring alerted Figma within seconds that one CME ElastiCache node had reached 100% Engine CPU. - AWS later identified a rare Redis bug triggered by high Publish/Subscribe traffic on CME clusters. - Figma could not obtain CPU profiles from the underlying ElastiCache machines, making it difficult to identify the exact operation consuming CPU. - Increasing cluster capacity did not solve the problem and ultimately made the behavior worse. ## Mitigation and Investigation - Figma initially suspected insufficient capacity or faulty hardware. - Engineers: - Initiated a failover of the affected node - Created a larger ElastiCache cluster with more nodes - Redirected traffic to the replacement clusters when failover took too long - The first traffic redirection restored service, but the new cluster later experienced the same CPU saturation. - A subsequent failover completed successfully and restored service more quickly. - Engineers investigated and ruled out: - Routine backups or unexpected snapshots - Sudden increases in Redis command volume - Slow commands, large keys, or other obvious misuse - Scheduled background jobs - Blocking requests that appeared to be waiting longest for Redis commands did not resolve the issue. Figma’s outage was ultimately caused by an AWS Redis/ElastiCache defect rather than a data-loss event or an observable change in application traffic. The incident underscores the need to validate Redis Publish/Subscribe workloads on Cluster-Mode Enabled deployments and maintain mitigation strategies that do not rely solely on adding capacity.