How Discord Automates ScyllaDB Clusters at Scale (opens in new tab)
Discord’s Persistence Infrastructure team replaced fragile, manually sequenced scripts with the Scylla Control Plane (SCP), a framework for safely automating large-scale database operations. The effort was driven by the difficulty of creating shadow clusters and managing hundreds of ScyllaDB nodes with a seven-person team. SCP emphasizes resumability, safety checks, configurable parallelism, and incremental development. ## The Scale of Discord’s Database Operations - Discord operates Elasticsearch, Postgres, and ScyllaDB infrastructure across dozens of clusters and hundreds of nodes. - ScyllaDB stores critical data, including messages, channels, servers, and much of Discord’s user data. - Routine work includes: - Rolling restarts after configuration changes - Cluster expansion as traffic grows - Operating-system upgrades without downtime - Creating test clusters for validating ScyllaDB releases - These operations require careful sequencing and continuous validation rather than simple, fire-and-forget automation. ## From Scripts to the Scylla Control Plane - Discord initially accumulated Python, Bash, and other scripts incrementally. - The scripts were useful but fragile and dependent on institutional knowledge. - As operational demands grew, Discord created the Scylla Control Plane, or SCP, to provide a more structured automation system. ## Shadow Clusters for Safer Upgrades - Shadow clusters are temporary, full replicas of production that receive the same reads and writes as live traffic. - They allow Discord to detect upgrade problems under realistic load before changing production. - Building one manually requires: - Provisioning and configuring nodes - Joining nodes to the cluster - Validating replication - Establishing dual-write pipelines - Eventually tearing the environment down - Repeating this process across every ScyllaDB cluster made automation essential, especially for testing operating-system, hardware, and ScyllaDB version changes. ## Lessons from the Previous Automation Discord identified three major weaknesses in its old scripts: - **Unsafe:** Scripts could be run against the wrong nodes or in the wrong order, often without precondition checks. - **Unrecoverable:** A failure late in a multi-step process required restarting from the beginning. - **Difficult to extend:** New operations often required copying and modifying existing scripts instead of composing reusable components. SCP was designed around four goals: - Provide an extensible task framework that hides orchestration complexity. - Support configurable parallelism, including constraints such as avoiding simultaneous work in different availability zones. - Make safety the default through preconditions, retries, and persisted state. - Deliver functionality incrementally and refine it through real-world use. ## SCP’s Task-Based Architecture - SCP is organized around **tasks, workflows, and jobs**. - A task represents one unit of work, such as draining a node, checking repair status, or running cleanup. - **Node tasks** operate on individual nodes. - **Cluster tasks** coordinate operations across an entire cluster and may run node tasks across many nodes. - SCP also uses **conditions**, which pause execution until a required state is reached. - Conditions poll ScyllaDB APIs or Prometheus metrics. - They either succeed when the criterion is met or fail after a timeout. - For example, after restarting a node, SCP can wait for compactions to settle before continuing. - This avoids unreliable fixed-duration sleeps and reduces the risk of creating cascading pressure during rolling operations. ## Practical Recommendation For large-scale database operations, automation should be built as a reusable, stateful orchestration framework rather than a collection of scripts. Explicit preconditions, observable conditions, retries, controlled parallelism, and resumable state make complex infrastructure changes safer and more repeatable.