metaFeb 24, 2026

RCCLX: Innovating GPU communications on AMD platforms (opens in new tab)

RCCLX is Meta’s open-source enhancement of RCCL for AMD GPUs, integrated with Torchcomms to support portable distributed AI workloads. It introduces Direct Data Access (DDA) and low-precision collectives, targeting communication bottlenecks in inference and training. On AMD MI300X systems, these optimizations deliver lower latency and higher throughput while maintaining acceptable accuracy. ## RCCLX and Torchcomms Integration - RCCLX is based on RCCL and tested on Meta’s internal workloads. - It integrates CTran transport technology for AMD platforms. - CTran enables features such as `AllToAllvDynamic`, a GPU-resident collective; additional CTran capabilities are planned for future releases. - Through Torchcomms, applications can use a common communication API across AMD, NVIDIA, and other backends without major code changes. - RCCLX is intended to achieve feature parity with Meta’s NCCLX backend for NVIDIA systems. ## Direct Data Access for Intra-Node Collectives - LLM inference has two distinct phases: - **Prefill** is compute-bound and generates the model’s key-value cache. - **Decoding** is memory-bound and generates tokens incrementally. - Tensor parallelism can make AllReduce responsible for up to 30% of end-to-end latency. - RCCLX introduces two DDA algorithms: - **DDA flat** lets each rank directly read other ranks’ memory and perform local reductions. It reduces latency from O(N) to O(1) for small messages by increasing data exchange from O(n) to O(n²). - **DDA tree** divides AllReduce into reduce-scatter and all-gather phases, retaining ring-like data movement while reducing latency for somewhat larger messages. - On AMD MI300X GPUs, DDA improves over RCCL by: - 10–50% for decode workloads. - 10–30% for prefill workloads. - Approximately 10% lower time-to-incremental-token. ## Low-Precision Collectives - RCCLX provides optimized low-precision versions of AllReduce, AllGather, AlltoAll, and ReduceScatter. - These target AMD Instinct MI300 and MI350 GPUs and support FP32 and BF16 inputs. - FP8 quantization provides up to 4:1 compression, reducing communication overhead for messages of at least 16 MB. - Parallel peer-to-peer mesh communication uses AMD Infinity Fabric for bandwidth and low latency. - Computation remains in FP32 to improve numerical stability. - Users can enable the feature with: ```bash RCCL_LOW_PRECISION_ENABLE=1 ``` - Internal evaluations showed: - About a 0.3% change on GSM8K accuracy evaluations. - 9–10% lower latency. - Approximately 7% higher throughput. - The current implementation is tuned for single-node deployments. ## Getting Started - Install Torchcomms with the RCCLX backend. - Create an RCCLX communicator through Torchcomms using the `"rcclx"` backend and a HIP device. - Existing Torchcomms operations such as `allreduce` can then run without backend-specific API changes. - Distributed initialization uses standard `torchrun` environment variables such as `MASTER_ADDR`, `MASTER_PORT`, `RANK`, and `WORLD_SIZE`. RCCLX is positioned as a practical way to improve AMD-based AI training and inference without requiring applications to adopt a new communication API. Teams can use DDA for lower inference latency and selectively enable low-precision collectives for higher throughput, while evaluating numerical accuracy for their own workloads.

distributed-computinggpu-accelerationllm-inferencercclx+4

pinterestFeb 17, 2026

Drastically Reducing Out-of-Memory Errors in Apache Spark at Pinterest (opens in new tab)

Pinterest developed **Auto Memory Retries** to reduce Spark out-of-memory failures without permanently assigning oversized executors to every task. The system detects OOM failures and retries affected tasks with progressively larger resource profiles, reducing both on-call incidents and wasted compute. Instead of tuning every job for its peak memory demand, Pinterest can size jobs around typical usage while handling exceptional tasks elastically. ## Pinterest’s Spark Environment - Pinterest processes more than **90,000 Spark jobs daily** across tens of thousands of nodes. - Its infrastructure includes: - Kubernetes clusters - Spark 3.2, with Spark 3.5 adoption underway - Apache Celeborn for shuffle - Apache YuniKorn for scheduling - Apache Gluten and Meta’s Velox for acceleration - Archer, Pinterest’s internal submission service - More than **4.6% of job failures** were caused by OOM errors. ## Why Manual Memory Tuning Was Insufficient - Pinterest’s clusters are memory-bound, so simply increasing executor sizes is expensive and difficult. - Automatic tuning generally reduces executor memory to match historical usage and improve resource efficiency. - Manual tuning can work, but requires substantial expertise because: - Different stages perform different operations. - Individual tasks may have very different memory needs because of data skew. - Configurations that work for most tasks may fail for a small number of high-memory tasks. - Auto Memory Retries allow jobs to target approximately their **P90 memory usage**, while automatically giving unusually demanding tasks more capacity. ## How Spark Executor Memory Works - An executor’s memory and CPU capacity determine how many tasks can run concurrently. - By default, each CPU core provides a task slot. - For example, with `spark.task.cpus=2`, an executor with two usable task slots and 8 GB of memory provides roughly 4 GB per task on average. - Memory is shared, so one task may temporarily use more than its average allocation if another uses less. - An OOM occurs when the combined memory usage of concurrent tasks exceeds the executor’s available memory. ## Auto Memory Retries Design Pinterest modified Spark’s scheduling loop so individual tasks can use resource profiles different from their parent `TaskSet`. - Each task can store an optional `taskRpId` identifying its retry resource profile. - Pinterest creates immutable retry profiles at **2x, 3x, and 4x** the base profile. - If off-heap memory is enabled, it is scaled as well. - Retries use a hybrid strategy: - **First retry:** Double `cpus per task`, allowing the task to run on an existing executor with fewer concurrent tasks. - **Later retry:** Launch a physically larger executor if the task still fails or already requires the entire executor. - The approach prioritizes reusing existing executors before provisioning larger ones. ## Changes to Spark Internals Pinterest extended core Spark components through Pinterest-specific subclasses rather than using a listener-only implementation. - **Task** - Stores the optional task resource profile ID. - **TaskSetManager** - Tracks tasks with non-default profiles. - Assigns the next larger retry profile after an OOM. - **TaskSchedulerImpl** - Allows tasks with increased CPU requirements to run on standard executors. - **ExecutorAllocationManager** - Tracks pending tasks by retry profile. - Requests larger executors when physical memory is required. - The feature-specific classes are loaded only when Auto Memory Retries is enabled. - The Spark UI was updated to display each task’s resource profile ID. ## Handling Tasks After an OOM - When a task fails on an executor with more than one core, its first retry doubles `spark.task.cpus`. - Other tasks in the same stage or future stages are unaffected. - Spark cannot reliably determine which concurrent task caused the executor-level OOM. - As a result, Pinterest treats **all tasks running on the terminated executor** as having failed due to OOM and routes them to retries that do not share the executor with other tasks. ## Practical Conclusion Pinterest’s approach makes executor sizing elastic at the task level: configure jobs for normal memory usage, then progressively increase resources only for tasks that need them. This can reduce OOM-related failures and operational load while avoiding the cost of running every task on oversized executors.

distributed-computingk8sapache-sparkmemory-management+4