Scaling Laws | Techlist.io

metaAug 5, 20263 min readCurated summary

From User Sequences to Scaling Laws: A Multi-Stage Architecture for Meta’s Ads Ranking

Meta’s new sequence-learning platform improves ads recommendations by separating deep offline user modeling from fast online ranking. Combined with dense tokenization and target-aware attention, it enables richer behavioral representations, predictable compute-to-performance scaling, and major gains: 6% more Instagram conversions, 3% more Facebook conversions, and 3.5% more Facebook ad clicks. The system is also a core part of Meta’s Generative Ads Recommendation Model (GEM). ## Challenges of Earlier Sequence Models - Ads systems must rank thousands of candidates within milliseconds and process millions of candidates per second. - Hybrid architectures typically use: - One model for user event sequences. - Another for sparse feature interactions. - This design can cause: - Lossy knowledge transfer between components. - Continued dependence on manually engineered features. - Scaling limits caused by interference between sequence modeling and ranking. - Increasing sequence lengths and transformer capacity can therefore raise serving costs without delivering proportional improvements. ## Multi-Stage Sequence Modeling Meta separates sequence learning into two complementary stages: - **Offline user modeling** - Processes long user histories asynchronously. - Uses deep transformer models with thousands of events and multiple layers. - Produces cached, user-level embeddings that represent long-term behavioral patterns. - Keeps user features separate from ad and context features so embeddings remain independent of individual candidates. - **Online ranking** - Combines cached user embeddings with fresh user signals, ad features, and context. - Performs final ranking under strict latency requirements. - Uses a lightweight architecture optimized for real-time serving. This separation allows the offline model to grow in depth, width, and sequence length without proportionally increasing online serving costs. ## Dense Tokenization and Target-Aware Attention - **Dense tokenization** - Converts sparse features and sequential behavioral data into a shared dense vocabulary. - Allows the model to learn feature interactions directly instead of relying on manually engineered cross-features. - **Target-aware multi-head attention** - Combines user behavior sequences with the specific ad candidate being scored. - Lets each attention layer determine which past behaviors matter for that candidate. - Stacked attention blocks capture increasingly complex interactions and compress long histories into compact representations. - The approach is designed to be memory-efficient while preserving candidate-specific information. ## Predictable Scaling Laws - On real-world ads traffic, the architecture shows an LLM-like log-linear relationship between compute and recommendation performance. - Improvements were measured using normalized entropy across: - Model depth. - Model width. - Sequence length. - Content and semantic enrichment. - The scaling behavior suggests the architecture is well suited to continued investment in sequence learning, despite recommendation systems combining sparse IDs with temporal data rather than dense text. ## Scaling Strategies - **Balanced model shape** - Depth, width, and sequence length should grow together. - Scaling only one dimension can create bottlenecks and diminishing returns. - Meta calls this the “scaling synergy principle.” - **Multi-stage tunability** - Online models offer strong improvements per unit of compute but are constrained by request latency. - Offline models improve more gradually but can scale aggressively because inference is asynchronous. - **Sequence composition** - Longer sequences generally improve performance. - Diversity of actions is more valuable than simply adding more homogeneous events. ## Practical Conclusion Meta’s approach makes sequence learning more scalable and operationally practical by moving expensive user-history processing offline while retaining fast, target-specific ranking online. Dense tokenization and target-aware attention reduce manual feature engineering, while the observed scaling laws provide a framework for deciding where additional model capacity and compute will produce the greatest gains.

Read original

Scaling Laws Transformer Recommendation Systems Attention Mechanisms+3

googleJan 28, 20263 min readCurated summary

Towards a science of scaling agent systems: When and why agent systems work

AI agent systems do not improve simply by adding more agents. Google Research’s evaluation of 180 configurations found that coordination helps substantially on parallelizable tasks but can severely hurt sequential workflows and tool-heavy tasks. The study proposes measurable design principles and a predictive model that selected the best architecture for 87% of unseen tasks. ## Defining Agentic Tasks The study distinguishes agentic tasks from static benchmarks by requiring: - Sustained, multi-step interaction with an external environment. - Iterative information gathering under partial observability. - Adaptive strategy changes based on environmental feedback. Researchers tested five architectures across Finance-Agent, BrowseComp-Plus, PlanCraft, and Workbench: - **Single-agent:** One agent handles reasoning and actions sequentially. - **Independent:** Agents work in parallel without communication and combine results at the end. - **Centralized:** An orchestrator delegates work and synthesizes outputs. - **Decentralized:** Agents communicate directly in a peer-to-peer network. - **Hybrid:** Hierarchical oversight is combined with peer coordination. ## Coordination Must Match the Task - Multi-agent systems produced very different results across GPT, Gemini, and Claude models. - On parallelizable financial reasoning tasks, centralized coordination improved performance by **80.9%** over a single agent. - Parallel agents can independently analyze areas such as revenue, costs, and market comparisons before combining their findings. - On sequential planning tasks, every multi-agent architecture performed worse, with declines of **39–70%**. - Communication and synchronization overhead can fragment reasoning and consume the available cognitive budget. ## The Tool-Coordination Trade-off - As tasks require more tools, coordinating multiple agents becomes increasingly expensive. - Tool-heavy systems, such as coding agents with access to 16 or more tools, face a disproportionate coordination “tax.” - Adding agents is therefore especially risky when actions must be tightly ordered or frequently synchronized. ## Architecture and Reliability - Architecture affects not only performance but also how errors spread. - Independent agents amplified errors by up to **17.2×**, because no mechanism checked their intermediate results. - Centralized systems limited error amplification to **4.4×**. - An orchestrator acts as a validation bottleneck, detecting and containing mistakes before they propagate. ## Predicting the Best Architecture - The researchers built a predictive model using properties such as task decomposability and tool count. - The model achieved an **R² of 0.513**. - It correctly predicted the optimal coordination strategy for **87% of unseen task configurations**. - These results point toward systematic, task-driven agent design rather than relying on the assumption that more agents are always better. For practical deployments, choose architecture based on the task: use coordinated parallel agents for decomposable work, simpler sequential systems for tightly ordered reasoning, and centralized oversight when reliability and error containment are priorities.

Read original

Scaling Laws Large Language Models AI Agents Multi Agent Systems+3

googleJan 27, 20263 min readCurated summary

ATLAS: Practical scaling laws for multilingual models

ATLAS introduces practical scaling laws for training multilingual language models, addressing the lack of public guidance for non-English systems. Based on 774 runs covering 400+ languages and models from 10M to 8B parameters, it predicts how to combine languages, data, and model capacity efficiently. The study finds strong cross-lingual transfer, a manageable multilingual capacity tax, and clear trade-offs between fine-tuning and pretraining from scratch. ## Adaptive Scaling for Multilingual Mixtures - ATLAS extends traditional scaling laws with: - A cross-lingual transfer matrix identifying helpful language combinations. - Rules for scaling model size and data as supported languages increase. - Guidance on whether to pretrain from scratch or fine-tune a multilingual checkpoint. - It separates training data into: - The target language. - Similar “transfer languages,” such as Spanish, Portuguese, and Italian for Catalan. - All other languages. - This allows ATLAS to estimate which languages help or hinder a target language. ## Evaluation Across Languages and Model Sizes - Experiments used MADLAD-400, spanning more than 750 monolingual, bilingual, and multilingual runs. - ATLAS outperformed earlier scaling laws when predicting performance for new: - Model sizes. - Data volumes. - Language mixtures. - Optimal scaling patterns were broadly similar across English, French, Russian, Chinese, Hindi, and Swahili. - Multilingual vocabularies and data impose a compute-efficiency tax, particularly for English. - Low-resource languages eventually encounter data repetition, causing their scaling curves to bend upward. ## Cross-Lingual Transfer - The transfer matrix measures how training on one language affects another. - Examples of strong transfer include: - Norwegian benefiting from Swedish and German. - Malay benefiting from Indonesian. - Arabic benefiting from Hebrew. - English, French, and Spanish are broadly useful training languages, partly because of their large, diverse, and high-quality web corpora. - Shared writing systems and language families are the strongest predictors of positive transfer, with statistical significance of p < .001. - Transfer is asymmetric: language A may help language B more than B helps A. - The results replace informal language-selection assumptions with empirical data. ## Scaling the Number of Supported Languages - ATLAS formalizes the “curse of multilinguality,” in which adding languages can reduce performance because model capacity is limited. - Adding languages creates a modest capacity cost but also substantial positive transfer. - To support twice as many languages, the study recommends approximately: - 1.18× larger model size. - 1.66× more total training data. - Although each language receives less data individually, cross-lingual synergies offset much of the degradation. ## Pretraining Versus Fine-Tuning - Fine-tuning a strong multilingual “Unimax” checkpoint generally delivers the best early performance for the least additional compute. - Pretraining from scratch can eventually produce better results when substantially more tokens are affordable. - For 2B-parameter models, the crossover typically occurs between roughly 144B and 283B tokens, depending on the language. - The supplied article ends while discussing how ATLAS further models this crossover point. ## Practical Recommendation Use ATLAS to select language mixtures based on measured transfer rather than intuition. Fine-tune an existing multilingual checkpoint under tight compute budgets, but consider training from scratch when enough data and compute are available to pass the language-specific crossover point.

Read original

Scaling Laws Large Language Models Fine Tuning Pre Training+4

googleSep 11, 2025Original article

VaultGemma: The world's most capable differentially private LLM (opens in new tab)

VaultGemma represents a significant milestone in privacy-preserving AI as the most capable large language model trained from scratch using differential privacy (DP). By establishing new scaling laws specifically for DP training, researchers have optimized the complex trade-offs between compute, privacy budgets, and model utility. The resulting 1-billion-parameter model demonstrates that high-performance generative AI can be achieved while maintaining rigorous mathematical guarantees against data memorization. ## Scaling Laws for Differentially Private Training * Performance in DP-trained models is primarily governed by the "noise-batch ratio," which measures the amount of random privacy noise relative to the size of the training data groups. * Research suggests that for any given compute and privacy budget, there exists an optimal training configuration that balances model size, iterations, and batch size to achieve the lowest possible training loss. * A critical finding indicates that DP training requires a departure from standard scaling practices, favoring significantly larger batch sizes and smaller model architectures than traditional non-DP training. ## Synergies in Privacy, Compute, and Data * Increasing the privacy budget (epsilon) in isolation leads to diminishing returns unless it is paired with a proportional increase in compute (FLOPs) or data (tokens). * Visualizations of the scaling laws show that different model sizes can provide similar utility if the number of training iterations and batch sizes are correctly adjusted. * The optimal configuration shifts between investing in larger models versus more iterations depending on the specific constraints of the data and privacy budgets. ## Training at Scale with Algorithmic Advancements * VaultGemma is built on the Gemma 2 architecture and utilizes a 1B parameter setup optimized for the unique constraints of DP. * To overcome hardware limitations when processing the massive batch sizes required for DP training, the team developed a "Virtual Batch" technique in JAX to aggregate gradients across multiple steps. * Training from scratch allows the model to outperform traditional DP-finetuned models, which often struggle to balance utility with the noise introduced during the fine-tuning process. ## Performance and Evaluation * VaultGemma achieves competitive results against standard 1B parameter models while providing formal privacy protections. * The model demonstrates superior privacy-utility trade-offs, proving that carefully scaled DP models can retain high levels of reasoning and language capability. * The release includes the model weights and a comprehensive technical report to assist the community in developing the next generation of private-by-design AI. VaultGemma provides a practical blueprint for developers who need to balance the power of large language models with strict data confidentiality requirements. By leveraging the provided scaling insights, organizations can now train models that are mathematically resistant to data leakage without sacrificing significant performance.

Scaling Laws AI Large Language Models Differential Privacy+3