ATLAS: Practical scaling laws for multilingual models
ATLAS introduces practical scaling laws for training multilingual language models, addressing the lack of public guidance for non-English systems. Based on 774 runs covering 400+ languages and models from 10M to 8B parameters, it predicts how to combine languages, data, and model capacity efficiently. The study finds strong cross-lingual transfer, a manageable multilingual capacity tax, and clear trade-offs between fine-tuning and pretraining from scratch. ## Adaptive Scaling for Multilingual Mixtures - ATLAS extends traditional scaling laws with: - A cross-lingual transfer matrix identifying helpful language combinations. - Rules for scaling model size and data as supported languages increase. - Guidance on whether to pretrain from scratch or fine-tune a multilingual checkpoint. - It separates training data into: - The target language. - Similar “transfer languages,” such as Spanish, Portuguese, and Italian for Catalan. - All other languages. - This allows ATLAS to estimate which languages help or hinder a target language. ## Evaluation Across Languages and Model Sizes - Experiments used MADLAD-400, spanning more than 750 monolingual, bilingual, and multilingual runs. - ATLAS outperformed earlier scaling laws when predicting performance for new: - Model sizes. - Data volumes. - Language mixtures. - Optimal scaling patterns were broadly similar across English, French, Russian, Chinese, Hindi, and Swahili. - Multilingual vocabularies and data impose a compute-efficiency tax, particularly for English. - Low-resource languages eventually encounter data repetition, causing their scaling curves to bend upward. ## Cross-Lingual Transfer - The transfer matrix measures how training on one language affects another. - Examples of strong transfer include: - Norwegian benefiting from Swedish and German. - Malay benefiting from Indonesian. - Arabic benefiting from Hebrew. - English, French, and Spanish are broadly useful training languages, partly because of their large, diverse, and high-quality web corpora. - Shared writing systems and language families are the strongest predictors of positive transfer, with statistical significance of p < .001. - Transfer is asymmetric: language A may help language B more than B helps A. - The results replace informal language-selection assumptions with empirical data. ## Scaling the Number of Supported Languages - ATLAS formalizes the “curse of multilinguality,” in which adding languages can reduce performance because model capacity is limited. - Adding languages creates a modest capacity cost but also substantial positive transfer. - To support twice as many languages, the study recommends approximately: - 1.18× larger model size. - 1.66× more total training data. - Although each language receives less data individually, cross-lingual synergies offset much of the degradation. ## Pretraining Versus Fine-Tuning - Fine-tuning a strong multilingual “Unimax” checkpoint generally delivers the best early performance for the least additional compute. - Pretraining from scratch can eventually produce better results when substantially more tokens are affordable. - For 2B-parameter models, the crossover typically occurs between roughly 144B and 283B tokens, depending on the language. - The supplied article ends while discussing how ATLAS further models this crossover point. ## Practical Recommendation Use ATLAS to select language mixtures based on measured transfer rather than intuition. Fine-tune an existing multilingual checkpoint under tight compute budgets, but consider training from scratch when enough data and compute are available to pass the language-specific crossover point.
Read original(opens in new tab)