ATLAS: Practical scaling laws for multilingual models

ATLAS introduces practical scaling laws for training multilingual language models, addressing the lack of public guidance for non-English systems. Based on 774 runs covering 400+ languages and models from 10M to 8B parameters, it predicts how to combine languages, data, and model capacity efficiently. The study finds strong cross-lingual transfer, a manageable multilingual capacity tax, and clear trade-offs between fine-tuning and pretraining from scratch.

Adaptive Scaling for Multilingual Mixtures

• ATLAS extends traditional scaling laws with:
  • A cross-lingual transfer matrix identifying helpful language combinations.
  • Rules for scaling model size and data as supported languages increase.
  • Guidance on whether to pretrain from scratch or fine-tune a multilingual checkpoint.
• It separates training data into:
  • The target language.
  • Similar “transfer languages,” such as Spanish, Portuguese, and Italian for Catalan.
  • All other languages.
• This allows ATLAS to estimate which languages help or hinder a target language.

Evaluation Across Languages and Model Sizes

• Experiments used MADLAD-400, spanning more than 750 monolingual, bilingual, and multilingual runs.
• ATLAS outperformed earlier scaling laws when predicting performance for new:
  • Model sizes.
  • Data volumes.
  • Language mixtures.
• Optimal scaling patterns were broadly similar across English, French, Russian, Chinese, Hindi, and Swahili.
• Multilingual vocabularies and data impose a compute-efficiency tax, particularly for English.
• Low-resource languages eventually encounter data repetition, causing their scaling curves to bend upward.

Cross-Lingual Transfer

• The transfer matrix measures how training on one language affects another.
• Examples of strong transfer include:
  • Norwegian benefiting from Swedish and German.
  • Malay benefiting from Indonesian.
  • Arabic benefiting from Hebrew.
• English, French, and Spanish are broadly useful training languages, partly because of their large, diverse, and high-quality web corpora.
• Shared writing systems and language families are the strongest predictors of positive transfer, with statistical significance of p < .001.
• Transfer is asymmetric: language A may help language B more than B helps A.
• The results replace informal language-selection assumptions with empirical data.

Scaling the Number of Supported Languages

• ATLAS formalizes the “curse of multilinguality,” in which adding languages can reduce performance because model capacity is limited.
• Adding languages creates a modest capacity cost but also substantial positive transfer.
• To support twice as many languages, the study recommends approximately:
  • 1.18× larger model size.
  • 1.66× more total training data.
• Although each language receives less data individually, cross-lingual synergies offset much of the degradation.

Pretraining Versus Fine-Tuning

• Fine-tuning a strong multilingual “Unimax” checkpoint generally delivers the best early performance for the least additional compute.
• Pretraining from scratch can eventually produce better results when substantially more tokens are affordable.
• For 2B-parameter models, the crossover typically occurs between roughly 144B and 283B tokens, depending on the language.
• The supplied article ends while discussing how ATLAS further models this crossover point.

Practical Recommendation

Use ATLAS to select language mixtures based on measured transfer rather than intuition. Fine-tune an existing multilingual checkpoint under tight compute budgets, but consider training from scratch when enough data and compute are available to pass the language-specific crossover point.