Chaos Engineering | Techlist.io

netflixJun 19, 20263 min readCurated summary

The Data Canary: How Netflix Validates Catalog Metadata

Netflix built an automated “data canary” system to validate catalog metadata changes with real production traffic. The system compares a new catalog version against a known-good baseline, detects customer-impacting regressions in under 10 minutes, and blocks corrupted data before it reaches most members. The effort treats data deployments with the same rigor traditionally applied to code deployments. ## Why Catalog Data Needs Canarying - Catalog metadata defines available titles, artwork, playback eligibility, and regional availability. - A previous incident corrupted a feed without any code or configuration change. - The resulting empty data for some titles prevented manifest generation and caused playback failures. - Existing code canaries detected nothing because the failure occurred in transformed data, not application code. - Validating individual upstream feeds was insufficient because corruption could emerge during final transformation. ## Challenges of Fast, Production-Level Validation - Data cycles occur frequently, leaving only one cycle to detect problems and block publication. - Traditional canary analysis requires 30–60 minutes to reach statistical confidence. - Shadow traffic could replay catalog requests but could not reproduce the full playback lifecycle across services. - Real production traffic was necessary to expose actual customer impact. - The system also needed to contain regressions so that validation itself did not create a large outage. ## The Data Canary Orchestrator - Netflix created a dedicated canary environment with: - An orchestrator instance coordinating validation. - A permanent baseline cluster serving the latest production catalog. - A canary cluster receiving the new catalog version. - Before testing, the orchestrator verifies that both clusters are healthy and version-synchronized. - It then triggers a chaos experiment that compares customer behavior across the two versions. - Results are returned to the transformer through a generic REST endpoint, allowing other data sources to adopt the pattern without transformer-specific changes. ## Extending the Chaos Platform - Experiment thresholds were customized to meet the 10-minute detection requirement. - Separate tests were run for major client types because they have different traffic patterns and dependencies. - Playback traffic was especially effective at revealing failures. - Sticky canaries used session affinity to keep each user on either the baseline or canary cluster, enabling a clean comparison. - Starts Per Second (SPS) became the primary metric because it measures successful playback attempts more directly than latency or catalog-service error rates. - Metrics are streamed in real time, and experiments abort immediately when a regression appears. - This prioritizes rapid protection over maximum statistical confidence, which is appropriate given the strong customer-impact signal. ## Production-Hardened Reliability - The orchestrator resumes polling experiments after restarts instead of abandoning active validation cycles. - Leader election prevents multiple orchestrator instances from triggering duplicate experiments during deployment. - Version tracking ensures baseline and canary clusters are aligned across tenants with different data-consumption schedules. ## Controlled Failure Injection - Netflix validated the validator by deliberately corrupting catalog data. - Tests included denylisting prominent titles and simulating realistic data-corruption scenarios. - These experiments demonstrated whether the canary could identify meaningful playback regressions before corrupted metadata was broadly released. Netflix’s approach shows that high-velocity data pipelines require deployment safeguards distinct from code canaries. Teams managing critical data should validate final transformed outputs with representative production traffic, use direct business-impact metrics, and automatically stop publication when regressions appear.

Read original

Chaos Engineering Rest Api Data Pipelines Data Canary+3

metaJun 3, 20263 min readCurated summary

Lights Out, Systems On: Validating Instant Power Loss Readiness

Meta’s Instantaneous PowerLoss Storm is a disaster-readiness testing program designed to prepare data centers for sudden, zero-notice power loss. It extends existing fault-tolerance mechanisms across facilities, servers, storage, compute, and the Twine orchestrator, while addressing region-wide failures and autonomous recovery. Through incremental testing and carefully defined tradeoffs, Meta aims to make losing an entire region as manageable as losing a smaller fault domain. ## Defense-in-Depth for Instant Failures - Power-loss tolerance was built into the full data-center stack, including mechanical and electrical systems, server racks, storage, compute, and Twine. - Batteries and Power Loss Siren (PLS) preserve in-memory data when racks lose power. - Twine services use region-wide asynchronous unavailability events (UEs) to coordinate shutdown and recovery. - Existing mechanisms had been tested against smaller fault domains, but region-wide failures introduced new challenges involving scale, replica placement, and autonomous startup. ## Solving Region Bootstrap Problems - Restarting a region may require millions of services to start simultaneously and discover their dependencies. - Circular dependencies among Twine control-plane services—such as Scheduler, Allocator, Broker, and Zelos—could prevent the orchestrator from starting itself. - Belljar CI/CD tests continuously identify critical startup dependencies before deployment. - A Twine recovery kit, supported by Belljar and Twrko, provides a manual “jumpstart” mechanism for breaking unexpected dependency cycles. - Meta also encountered a “boomerang” problem in which UEs shut down the control-plane services responsible for generating and distributing those signals. - The simpler solution was to let control-plane services ignore power-related shutdown UEs, preventing orphaned services that could not be reaped or recovered. ## Balancing Reliability and Engineering Velocity - Absolute tolerance to instant power loss could require costly or overly complex infrastructure and might create false positives during normal operations. - Meta defined unacceptable impacts as: - Storage or database data loss - Permanent damage to data-center facilities - Sustained disruption beyond one region - The company accepted bounded risks such as transient service errors, limited rack failures, and temporary staleness in routing or region-availability information. - Issues were considered tolerable when they could be remediated after the incident within a reasonable mean time to respond (MTTR). ## Incremental Validation Through PowerLoss Storms - Because testing a full region carried significant risk, Meta validated readiness progressively: - Dependency tests in new and pre-production regions - Exercises in shadow regions that mirror production - Tests in small production regions - Full tests in large regions supporting storage, AI, and data-warehouse workloads - During a Storm, Meta injects a power-supply fault to immediately de-energize an entire region. - After a short, realistic MTTR, remedial drain actions isolate the region from global controllers and schedulers. - The tests avoid preemptive preparation so they accurately represent an unexpected power failure. - Repeated exercises train both systems and engineers to handle regional loss with the resilience normally expected from smaller fault domains. Meta’s approach is to expand disaster readiness gradually: define unacceptable consequences, build layered recovery mechanisms, test at increasing scale, and use each exercise to improve both architecture and operational practice.

Read original

Chaos Engineering CI/CD Distributed Systems Fault Tolerance+3

datadogJun 6, 20223 min readCurated summary

Engineering Spotlight: Tay Nishimura

Tay Nishimura’s career shows that succeeding in tech is often less about fitting a standard engineering mold and more about finding work that matches one’s strengths. Although she initially struggled with the speed and coding demands of software development, her rigor, visual thinking, and careful approach became valuable in site reliability engineering. Her transition was enabled by self-directed learning, community education, and ToyNet, an open source networking simulator that demonstrated her practical abilities. ## Entering Tech from Mathematics - Tay began as a mathematics major focused on real analysis, then added computer science after advice from a professor. - Internships at Amazon and Google introduced her to the technology industry. - She found a sharp contrast between academia and industry: - School rewarded theoretical rigor. - Industry emphasized practical, fast, and agile solutions. - Tay also felt like an outsider because she had little exposure to computers growing up. ## Struggling with Traditional Software Engineering - Coding did not come naturally to Tay’s visual way of thinking. - She translated code into drawings to understand and modify it, then converted those ideas back into code. - This process produced high-quality, careful work but made her slower than colleagues expected. - A manager suggested product management and site reliability engineering as possible alternatives. - Tay discovered that her deliberate pace was useful for SRE work, particularly when evaluating failure modes and making critical changes. - Because her company offered no path into those roles, she eventually left rather than continue facing increasing stress. ## Discovering Networking and Technical Program Work - Tay’s next role had a software engineer title but involved work closer to product or technical program management. - She learned that job titles and responsibilities vary significantly between companies. - With better work-life balance, she began studying computer networking in her free time. - She created visual diagrams and learning modules to explain switches, routers, and packet flows. - These efforts became Project Reclass, a nonprofit teaching technical skills to incarcerated people and military veterans. - The program used improvised equipment, such as fake routers and switches, to teach concepts in environments where real networking hardware was unavailable. ## Building ToyNet During the Pandemic - After her company laid off its entire office during COVID-19, Tay decided to pursue SRE directly. - When prisons suspended in-person education, Project Reclass adapted by creating a digital networking simulator. - Tay architected ToyNet, an open source platform built with: - React - A Flask backend - Containerized Mininet instances for network emulation - Users can connect simulated routers, switches, and hosts, configure IP addresses, and run commands such as `ping` and `arp`. - ToyNet was designed to work for incarcerated learners with restricted internet access. - Deploying it in the cloud also gave Tay practical experience that helped compensate for limited professional cloud experience. - Companies interested in the project were more likely to advance her through the interview process, eventually leading to Datadog. ## Finding the Right Environment at Datadog - At Datadog, Tay learned Kubernetes, chaos engineering, network traffic control, and Go. - She found that her rigor and visual thinking were assets rather than liabilities. - While learning Datadog’s Chaos Controller codebase, she mapped files and dependencies by drawing boxes and arrows. - Her experience suggests that engineers do not need to learn or reason in a single conventional way; the right environment can turn an apparent weakness into a strength. Tay’s path recommends experimenting broadly, studying independently, and building concrete projects that reveal how you think and solve problems. The most suitable tech role may emerge only after moving between companies and disciplines rather than forcing yourself to succeed in an ill-fitting position.

Read original

Chaos Engineering Kubernetes Open Source React+4