From Single-Node to Multi-GPU Clusters: How Discord Made Distributed Compute Easy for ML Engineers (opens in new tab)
Discord argues that distributed machine learning becomes practical when developer experience is treated as a first-class engineering problem. Ray provided the distributed-computing foundation, while Discord built a platform around it with a CLI, Dagster and KubeRay orchestration, and the X-Ray observability interface. This transformed GPU-intensive ML from manual experimentation into reproducible production pipelines, enabling Ads Ranking to move to multi-GPU neural networks and produce major business gains. ## Scaling Beyond Single-Node ML - Discord’s ML systems grew from simple classifiers to complex models serving hundreds of millions of users. - Teams needed: - Multiple GPUs for training - Datasets larger than a single machine - More compute than existing infrastructure could provide - Ray addressed the distributed-computing challenge, but Discord still needed a standardized internal platform to make it easy to use. ## Problems with Ad-Hoc Ray Clusters - Early ML engineers manually created Ray clusters using open-source documentation. - This led to: - Inconsistent cluster configurations - Uneven resource management - No centralized scheduling - Limited monitoring - Multiple teams independently rebuilding infrastructure solutions - Discord concluded that Ray needed an internal platform layer rather than direct, manual use. ## A Parameterized CLI for Cluster Creation - Discord replaced numerous GPU-specific YAML templates with one parameterized template. - Engineers specify requirements such as: - GPU type - Worker count - Memory - The CLI generates Kubernetes configuration, security settings, and hardware-specific resource requests. - It manages the full cluster lifecycle, including creation and deletion. - This made multi-GPU environments available through a single command and standardized deployments across teams. ## Automated Orchestration with Dagster, KubeRay, and Ray - Discord combined three systems: - **Dagster** defines workflows, dependencies, schedules, and validated configuration. - **KubeRay** dynamically provisions Ray clusters on Kubernetes with the appropriate namespace, service account, and GPU node pool. - **Ray** executes distributed training, evaluation, and batch inference. - The workflow is: 1. An engineer launches or schedules a Dagster pipeline. 2. Dagster submits the job specification. 3. KubeRay creates the required Ray cluster. 4. Ray distributes the workload across GPUs. 5. Logs and metrics flow back to Dagster and monitoring systems. - The approach provides predictable, reproducible jobs with centralized visibility. - Discord’s ad relevance model now trains daily without engineers manually editing cluster configurations. ## Centralized Observability with X-Ray - Discord built X-Ray as a web UI for monitoring Ray infrastructure. - It displays: - Active clusters - Cluster ownership - Machine types - Current status - Engineers can inspect dashboards and launch interactive notebooks for experimentation from one place. ## Ads Ranking as a Production Test - Ads Ranking determines which Quest advertisements are most relevant to individual users. - Before Ray, the system relied on XGBoost and lacked: - Model sharding - Multi-GPU support - Scalable, frequent retraining - Ray enabled sharded neural networks trained on multi-GPU clusters. - Reported results included: - Twice as many players joining Quests - Ad coverage increasing from roughly 40% to nearly 100% - A production pipeline that retrains daily and continuously delivers new model versions Discord’s experience suggests that distributed ML succeeds when powerful infrastructure is paired with simple interfaces, automated orchestration, and strong observability. Organizations adopting Ray should build comparable platform tooling around it rather than expecting ML engineers to manage clusters, scheduling, and monitoring themselves.