Lights Out, Systems On: Validating Instant Power Loss Readiness (opens in new tab)

Meta’s Instantaneous PowerLoss Storm is a disaster-readiness testing program designed to prepare data centers for sudden, zero-notice power loss. It extends existing fault-tolerance mechanisms across facilities, servers, storage, compute, and the Twine orchestrator, while addressing region-wide failures and autonomous recovery. Through incremental testing and carefully defined tradeoffs, Meta aims to make losing an entire region as manageable as losing a smaller fault domain.

Defense-in-Depth for Instant Failures

• Power-loss tolerance was built into the full data-center stack, including mechanical and electrical systems, server racks, storage, compute, and Twine.
• Batteries and Power Loss Siren (PLS) preserve in-memory data when racks lose power.
• Twine services use region-wide asynchronous unavailability events (UEs) to coordinate shutdown and recovery.
• Existing mechanisms had been tested against smaller fault domains, but region-wide failures introduced new challenges involving scale, replica placement, and autonomous startup.

Solving Region Bootstrap Problems

• Restarting a region may require millions of services to start simultaneously and discover their dependencies.
• Circular dependencies among Twine control-plane services—such as Scheduler, Allocator, Broker, and Zelos—could prevent the orchestrator from starting itself.
• Belljar CI/CD tests continuously identify critical startup dependencies before deployment.
• A Twine recovery kit, supported by Belljar and Twrko, provides a manual “jumpstart” mechanism for breaking unexpected dependency cycles.
• Meta also encountered a “boomerang” problem in which UEs shut down the control-plane services responsible for generating and distributing those signals.
• The simpler solution was to let control-plane services ignore power-related shutdown UEs, preventing orphaned services that could not be reaped or recovered.

Balancing Reliability and Engineering Velocity

• Absolute tolerance to instant power loss could require costly or overly complex infrastructure and might create false positives during normal operations.
• Meta defined unacceptable impacts as:
  • Storage or database data loss
  • Permanent damage to data-center facilities
  • Sustained disruption beyond one region
• The company accepted bounded risks such as transient service errors, limited rack failures, and temporary staleness in routing or region-availability information.
• Issues were considered tolerable when they could be remediated after the incident within a reasonable mean time to respond (MTTR).

Incremental Validation Through PowerLoss Storms

• Because testing a full region carried significant risk, Meta validated readiness progressively:
  • Dependency tests in new and pre-production regions
  • Exercises in shadow regions that mirror production
  • Tests in small production regions
  • Full tests in large regions supporting storage, AI, and data-warehouse workloads
• During a Storm, Meta injects a power-supply fault to immediately de-energize an entire region.
• After a short, realistic MTTR, remedial drain actions isolate the region from global controllers and schedulers.
• The tests avoid preemptive preparation so they accurately represent an unexpected power failure.
• Repeated exercises train both systems and engineers to handle regional loss with the resilience normally expected from smaller fault domains.

Meta’s approach is to expand disaster readiness gradually: define unacceptable consequences, build layered recovery mechanisms, test at increasing scale, and use each exercise to improve both architecture and operational practice.