How Discord Automates ScyllaDB Clusters at Scale (opens in new tab)

Discord’s Persistence Infrastructure team replaced fragile, manually sequenced scripts with the Scylla Control Plane (SCP), a framework for safely automating large-scale database operations. The effort was driven by the difficulty of creating shadow clusters and managing hundreds of ScyllaDB nodes with a seven-person team. SCP emphasizes resumability, safety checks, configurable parallelism, and incremental development.

The Scale of Discord’s Database Operations

• Discord operates Elasticsearch, Postgres, and ScyllaDB infrastructure across dozens of clusters and hundreds of nodes.
• ScyllaDB stores critical data, including messages, channels, servers, and much of Discord’s user data.
• Routine work includes:
  • Rolling restarts after configuration changes
  • Cluster expansion as traffic grows
  • Operating-system upgrades without downtime
  • Creating test clusters for validating ScyllaDB releases
• These operations require careful sequencing and continuous validation rather than simple, fire-and-forget automation.

From Scripts to the Scylla Control Plane

• Discord initially accumulated Python, Bash, and other scripts incrementally.
• The scripts were useful but fragile and dependent on institutional knowledge.
• As operational demands grew, Discord created the Scylla Control Plane, or SCP, to provide a more structured automation system.

Shadow Clusters for Safer Upgrades

• Shadow clusters are temporary, full replicas of production that receive the same reads and writes as live traffic.
• They allow Discord to detect upgrade problems under realistic load before changing production.
• Building one manually requires:
  • Provisioning and configuring nodes
  • Joining nodes to the cluster
  • Validating replication
  • Establishing dual-write pipelines
  • Eventually tearing the environment down
• Repeating this process across every ScyllaDB cluster made automation essential, especially for testing operating-system, hardware, and ScyllaDB version changes.

Lessons from the Previous Automation

Discord identified three major weaknesses in its old scripts:

• Unsafe: Scripts could be run against the wrong nodes or in the wrong order, often without precondition checks.
• Unrecoverable: A failure late in a multi-step process required restarting from the beginning.
• Difficult to extend: New operations often required copying and modifying existing scripts instead of composing reusable components.

SCP was designed around four goals:

• Provide an extensible task framework that hides orchestration complexity.
• Support configurable parallelism, including constraints such as avoiding simultaneous work in different availability zones.
• Make safety the default through preconditions, retries, and persisted state.
• Deliver functionality incrementally and refine it through real-world use.

SCP’s Task-Based Architecture

• SCP is organized around tasks, workflows, and jobs.
• A task represents one unit of work, such as draining a node, checking repair status, or running cleanup.
• Node tasks operate on individual nodes.
• Cluster tasks coordinate operations across an entire cluster and may run node tasks across many nodes.
• SCP also uses conditions, which pause execution until a required state is reached.
  • Conditions poll ScyllaDB APIs or Prometheus metrics.
  • They either succeed when the criterion is met or fail after a timeout.
• For example, after restarting a node, SCP can wait for compactions to settle before continuing.
• This avoids unreliable fixed-duration sleeps and reduces the risk of creating cascading pressure during rolling operations.

Practical Recommendation

For large-scale database operations, automation should be built as a reusable, stateful orchestration framework rather than a collection of scripts. Explicit preconditions, observable conditions, retries, controlled parallelism, and resumable state make complex infrastructure changes safer and more repeatable.