GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model
Meta’s Generative Ads Recommendation Model (GEM), which powers ad recommendations across Instagram and Facebook, now trains at LLM scale across several thousand GPUs. By co-designing kernels, numerical precision, parallelism, networking, and memory management, Meta doubled end-to-end training efficiency to 20–25% Model FLOPs Utilization (MFU) while increasing training compute fourfold in 12 months. The work shows that recommendation models require infrastructure specifically adapted to their hybrid architecture and data patterns rather than a direct reuse of LLM techniques. ## GEM’s Architecture and Training Challenges - GEM combines: - Trillions of sparse embedding parameters. - Billions of dense parameters. - Sequence features, such as user activity history. - Non-sequence features, such as user location and ad representations. - Different feature groups use customized attention mechanisms while still supporting cross-feature learning. - Recommendation workloads differ substantially from typical LLMs: - User histories have highly variable lengths, making padding inefficient and potentially wasting up to 50% of computation. - Attention patterns are asymmetric, including long sequences with short windows and long queries with short key/value sets. - Small embedding dimensions and normalization layers create memory-bound operations. - CTR and CVR optimization are numerically sensitive, so aggressive low-precision training can harm model quality. ## Scaling Across Thousands of GPUs - GEM’s distributed training latency is determined by the slowest rank and the larger of its local computation or communication time. - Efficient scaling requires: - Computation to dominate communication. - Communication to overlap with computation without resource contention. - Minimal activation recomputation. - Balanced workloads across GPU ranks. - GEM makes these requirements difficult because: - Trillion-scale sparse parameters generate substantial communication. - Different layer types provide uneven opportunities for communication overlap. - Long sequences and large activations pressure GPU memory. - Jagged inputs create changing load imbalance and stragglers. ## Separating Compute and Scaling Efficiency - Meta measures end-to-end efficiency with: - **E2E MFU = Local MFU × Scaling Ratio** - **Local MFU** measures how effectively one GPU uses its compute hardware, including Tensor Cores and memory hierarchies. - **Scaling Ratio** measures how much single-GPU performance is retained across thousands of GPUs. - This framework separates: - Kernel design and numerical precision issues affecting individual GPUs. - Parallelism, networking, memory, and load-balancing issues affecting distributed training. ## Compute-Efficiency Optimizations - Meta developed recommendation-specific GPU kernels, including: - Jagged Flash Attention (JFA) for variable-length sequences. - Generalized Dot-Product Attention (GDPA). - BlockAttention. - These kernels are designed around GEM’s irregular shapes and asymmetric attention patterns rather than conventional LLM assumptions. - Mixed ultra-low-precision training, including MXFP8 for attention and MLP layers, improves throughput while accounting for recommendation models’ numerical sensitivity. - The kernels and precision recipes are customized to exploit the architecture of the latest-generation GPUs. ## Scaling-Efficiency Optimizations - Meta uses topology-aware five-dimensional parallelism to distribute GEM efficiently. - Dense parameters use: - Two-dimensional Fully Sharded Data Parallelism (FSDP). - Expert Parallelism. - Sparse parameters use fully sharded two-dimensional model parallelism. - These strategies are co-designed with Meta’s multi-tier network hierarchy to reduce communication overhead. - Streaming Multiprocessor (SM)-free collectives help communication run with less interference from GPU computation. - The overall design targets communication overlap, memory constraints, load balance, and the differing behavior of dense and sparse parameters. ## Results - GEM’s end-to-end training efficiency increased to 20–25% MFU. - Efficiency doubled over a 12-month period. - Total training FLOPs increased fourfold. - The results demonstrate that recommendation foundation models can reach LLM-scale training, but only through coordinated hardware and software optimization across kernels, precision, parallelism, networking, and memory. For large recommendation models, LLM infrastructure provides a starting point but is not sufficient. The practical recommendation is to optimize compute and distributed scaling as separate but connected problems, using workload-specific kernels, carefully validated low precision, topology-aware parallelism, and communication strategies tailored to sparse and dense model components.
Read original(opens in new tab)