Meta’s HSM-based Backup Key Vault supports end-to-end encrypted backups for WhatsApp and Messenger by storing recovery codes in tamper-resistant hardware that Meta and third parties cannot access. The geographically distributed vault uses majority-consensus replication for resilience. Meta is strengthening the system with over-the-air fleet-key distribution for Messenger and public evidence of secure HSM fleet deployments.
## Over-the-Air Fleet Key Distribution
- Clients verify HSM fleet authenticity using fleet public keys before establishing sessions.
- WhatsApp embeds these keys directly in the application.
- Messenger can receive keys over the air, allowing Meta to deploy new HSM fleets without requiring an app update.
- Keys are delivered in validation bundles:
- Signed by Cloudflare
- Counter-signed by Meta
- Recorded in a Cloudflare audit log
- The complete validation process is documented in Meta’s *Security of End-To-End Encrypted Backups* whitepaper.
## Transparent HSM Fleet Deployment
- Meta plans to publish evidence of the secure deployment of every new HSM fleet.
- Users will be able to verify deployment evidence using the audit procedures in the whitepaper.
- Deployments are expected to occur infrequently, generally no more than once every few years.
- The transparency initiative is intended to demonstrate that Meta cannot access users’ encrypted backups.
The system combines tamper-resistant HSMs, geographic replication, independently verifiable key distribution, and public deployment evidence. Readers seeking implementation details should consult the full whitepaper.
Facebook has re-architected Groups Search to make community knowledge easier to discover, understand, and validate. Its new hybrid retrieval system combines keyword matching with semantic search, while automated model-based evaluation measures relevance at scale. The result is improved search engagement and relevance without increasing error rates.
## Friction in Community Search
- **Discovery:** Traditional lexical search depends on exact words, so a query for “small individual cakes with frosting” might miss posts discussing “cupcakes.” Semantic matching helps connect different phrasing with the same intent.
- **Consumption:** Users often must read dozens of comments to identify consensus or useful advice, creating an “effort tax.”
- **Validation:** Relevant expertise is frequently scattered across group discussions, making it difficult to evaluate purchases or decisions using community knowledge.
## Hybrid Retrieval Architecture
- Queries are tokenized, normalized, and rewritten before retrieval.
- The **lexical path**, powered by Facebook’s Unicorn inverted index, retrieves exact or closely matching terms and preserves precision for proper nouns and quotations.
- In parallel, the **semantic path** uses a 12-layer, 200-million-parameter Search Semantic Retriever to encode queries into dense vectors.
- Approximate nearest-neighbor search over a Faiss index retrieves conceptually similar posts, even when they use different words.
## Multi-Task Ranking
- Results from lexical and semantic retrieval are merged for ranking.
- The ranking model combines traditional signals such as TF-IDF and BM25 with semantic cosine-similarity scores.
- A multi-task, multi-label model jointly optimizes for clicks, shares, and comments.
- This approach balances theoretical relevance with the likelihood of meaningful community engagement.
## Automated Relevance Evaluation
- Semantic similarity scores can be difficult to interpret, so evaluation was integrated into build verification testing.
- Llama 3 with multimodal capabilities acts as an automated judge of search results.
- Evaluation recognizes nuanced outcomes, including “somewhat relevant” results that share a broader domain or theme.
- This enables scalable measurement of conceptual matching and result diversity without relying entirely on human labeling.
## Results and Future Work
- The hybrid system outperformed the keyword-only baseline in offline quality and search-engagement metrics.
- Facebook reports improved relevance without higher error rates.
- Future plans include using LLMs directly during ranking and dynamically adapting retrieval parameters to query complexity.
The approach demonstrates that combining lexical precision with neural semantic understanding can make community search more effective. Further LLM integration may help the system interpret post content and tailor retrieval more intelligently.
Meta’s Capacity Efficiency Program uses AI agents to automate both the discovery and resolution of infrastructure performance issues. By combining standardized tools with encoded expertise from senior efficiency engineers, the platform turns investigations that once took hours into minutes and has recovered hundreds of megawatts of power. The approach aims to let Meta scale efficiency improvements across more product areas without proportionally increasing engineering headcount.
## Capacity Efficiency at Hyperscale
- At Meta’s scale, even a 0.1% performance regression can significantly increase power consumption across systems serving more than 3 billion people.
- The program has two complementary functions:
- **Offense:** Proactively identify and implement optimizations.
- **Defense:** Detect production regressions, identify their causes, and deploy mitigations.
- Human investigation is often the bottleneck, requiring engineers to analyze profiling data, review documentation and prior fixes, inspect deployments, and search internal discussions.
- AI automation can reduce roughly 10 hours of manual diagnosis to about 30 minutes.
## A Unified Platform for AI Efficiency Agents
- Meta built one platform for both offensive and defensive workflows because they share the same basic structure:
- Gather relevant technical context.
- Apply domain-specific reasoning.
- Produce a code change for review.
- **MCP tools** provide standardized interfaces for querying profiling data, retrieving experiment results, examining configuration history, searching code, and accessing documentation.
- **Skills** encode expert reasoning, including which tools to use and how to interpret their results.
- The same tools support both use cases, while specialized skills handle different optimization and regression scenarios.
## Defense: Automated Regression Resolution
- FBDetect monitors noisy production time series and can identify regressions as small as 0.005%.
- Traditional root-cause analysis correlates the regression with recent pull requests or configuration changes.
- Previously, teams often rolled back problematic changes—reducing engineering velocity—or left them unresolved, allowing resource waste to accumulate.
- The AI Regression Solver:
- Identifies affected functions and regression symptoms.
- Locates the responsible pull request, files, and changed lines.
- Applies mitigation expertise appropriate to the codebase, language, or regression type.
- Generates a corrective pull request and sends it to the original author for review.
- Faster resolution prevents small regressions from compounding across Meta’s infrastructure.
## Offense: Converting Opportunities into Code
- Efficiency opportunities describe potential improvements to existing code, but implementing them traditionally required substantial investigation and engineering time.
- Meta’s AI workflow gathers:
- Opportunity metadata.
- Optimization documentation.
- Examples of similar fixes.
- Relevant files and functions.
- Validation criteria.
- Skills then apply specialized knowledge, such as memoizing a function to reduce CPU usage.
- The agent generates a guarded candidate fix, checks syntax and style, validates that it addresses the intended issue, and presents the change in an engineer’s editor for review or one-click application.
- This expands the number of optimization opportunities engineers can pursue manually.
## Scaling Efficiency with AI
- The platform has already recovered hundreds of megawatts of power—enough to supply hundreds of thousands of U.S. homes for a year.
- Automated regression handling reduces ongoing waste, while automated opportunity resolution increases the volume of proactive improvements.
- The long-term goal is a self-sustaining efficiency engine in which AI handles the long tail of investigations and fixes, allowing engineers to focus on new products and higher-value work.
Meta’s approach recommends treating performance expertise as reusable, composable software: standardize access to engineering data, encode proven reasoning into skills, and let agents carry issues from detection through ready-to-review code changes.
Meta argues that organizations should begin migrating to post-quantum cryptography (PQC) before quantum computers become practical. The “store now, decrypt later” threat means attackers may already be collecting encrypted data for future decryption, making long-lived sensitive information vulnerable today. Meta’s experience suggests a phased strategy based on risk prioritization, cryptographic inventories, technical readiness, deployment, and operational guardrails.
## Why PQC Migration Is Urgent
- Quantum computers are expected to eventually break conventional public-key cryptography, potentially within 10–15 years.
- Attackers can use “store now, decrypt later” (SNDL) attacks by collecting encrypted data today and decrypting it once quantum capabilities mature.
- NIST and the UK NCSC have issued migration guidance, including target timeframes such as 2030 for protecting critical systems.
- NIST has standardized algorithms including:
- **ML-KEM (Kyber)** for key encapsulation
- **ML-DSA (Dilithium)** for digital signatures
- **HQC**, which includes contributions from Meta cryptographers
## Meta’s Migration Goals
Meta’s multi-year migration is guided by four objectives:
- **Effectiveness:** Protect systems against quantum-enabled adversaries.
- **Timeliness:** Deploy protections as standards and technologies evolve.
- **Performance:** Minimize latency, resource use, and user impact.
- **Cost efficiency:** Balance investment against the risk and sensitivity of each use case.
## PQC Migration Levels
Meta proposes a maturity ladder that measures how quickly an organization can respond to a relevant quantum event, such as a major technical breakthrough, new standards, or changing industry practices.
- **PQ-Unaware:** The organization has not recognized the quantum threat.
- **PQ-Aware:** The threat and eventual requirements have been assessed, but design work has not begun.
- **PQ-Ready:** A suitable PQC solution has been identified or prepared, but deployment is deferred because of cost, prioritization, or other constraints.
- **PQ-Hardened:** All currently available protections have been implemented, but complete mitigation is impossible because required primitives—such as efficient post-quantum OPRFs—do not yet exist.
- **PQ-Enabled:** A post-quantum-secure solution is deployed for the use case. This is the desired end state for every application.
Even reaching PQ-Ready can reduce future reaction time and create useful technical and organizational foundations, although it does not itself protect systems from quantum attacks.
## Meta’s PQC Migration Strategy
Meta describes migration as several potentially overlapping workstreams:
- **Define prioritization:** Classify applications by high, moderate, or low risk so the most exposed use cases move first.
- **Build a cryptographic inventory:** Identify where cryptography is used and which applications rely on quantum-vulnerable algorithms.
- **Address external dependencies:** Track standards, PQC-capable hardware security modules, and the maturity of available implementations.
- **Implement PQC components:** Build reusable post-quantum cryptographic capabilities for later integration.
- **Deploy guardrails:** Update cryptographic standards, prevent creation of new vulnerable keys, and restrict affected APIs.
- **Integrate protections:** Apply PQC components to prioritized use cases and internal traffic.
## Prioritizing Applications
The first prioritization category focuses on applications vulnerable to attacks that can begin now and be completed later using quantum algorithms such as Shor’s algorithm.
- Applications using quantum-vulnerable public-key encryption or key-exchange mechanisms are considered high priority.
- Systems handling sensitive data with long confidentiality requirements are especially exposed to SNDL attacks.
- Risk-based prioritization helps organizations avoid attempting a costly, simultaneous migration of every application.
Organizations should begin by identifying high-value and long-lived data, inventorying vulnerable cryptography, and moving each use case progressively toward PQ-Enabled status.
Meta escaped the “forking trap” by replacing its divergent WebRTC fork with a modular architecture based on the latest upstream release. The system builds legacy and current WebRTC versions side by side, enabling runtime A/B testing across more than 50 use cases before rollout. This improved performance, binary size, and security while establishing a repeatable process for continuous upstream upgrades.
## Why the WebRTC Fork Became a Problem
- Meta’s RTC stack supports Messenger, Instagram video calls, Cloud Gaming, and Meta Quest casting.
- Internal optimizations and bug fixes gradually caused its WebRTC fork to diverge from upstream.
- As the fork accumulated custom changes, merging community improvements became increasingly expensive and risky.
- A one-time upgrade was impractical because WebRTC serves billions of users across diverse devices and environments.
## Requirements for a Sustainable Upgrade Strategy
- Meta needed to:
- Run legacy and upstream-based WebRTC implementations simultaneously.
- Dynamically assign users to either version for safe A/B testing.
- Statically link both versions into the same application.
- Maintain custom patches in a monorepo without repeatedly rebuilding the migration process.
- Standard patch-file workflows were considered difficult to scale for Meta’s large codebase.
## Shim Layer and Dual-Stack Architecture
- A shim library was placed between application code and WebRTC.
- Applications call a unified, version-neutral API rather than calling either WebRTC implementation directly.
- A runtime “flavor” configuration routes each call to either the legacy or latest implementation.
- Shimming at the lowest practical layer avoided duplicating the higher-level call orchestration library:
- Full duplication would have added about 38 MB uncompressed.
- The shim-based design added roughly 5 MB, an 87% reduction.
## Resolving C++ Symbol Collisions
- Linking two WebRTC copies normally violates the C++ One Definition Rule and creates thousands of duplicate symbols.
- Meta automated namespace rewriting:
- `webrtc::` in the current version became `webrtc_latest::`.
- The legacy version became `webrtc_legacy::`.
- Global functions, variables, and classes outside namespaces were moved into namespaces where possible or assigned flavor-specific names.
- Macro conflicts, including `RTC_CHECK` and `RTC_LOG`, were addressed by:
- Removing unnecessary includes.
- Renaming infrequently used macros.
- Sharing modules such as `rtc_base` between versions to reduce duplication and shimming work.
## Preserving Backward Compatibility
- Renaming symbols could have broken existing call sites, especially code built for only one WebRTC flavor.
- An initial solution forward-declared every required symbol, but this created a large and fragile maintenance burden.
- The improved approach used C++ `using` declarations to bulk-import a flavor namespace into the familiar `webrtc::` namespace.
- This preserved existing source-level APIs without adding binary overhead, while allowing Meta to migrate selected call sites incrementally.
## Runtime Flavor Dispatch
- Shim adapters and converters must instantiate objects from either the legacy or current namespace.
- A template-based helper library keeps shared adapter logic in one place.
- Template specializations handle version-specific behavior.
- A global flavor enum, initialized during application startup, determines which WebRTC implementation is used.
- The design also supports single-flavor builds during the transition.
Meta’s approach demonstrates that large internal modifications do not have to require a permanent fork. A low-level shim, automated renamespacing, compatibility imports, and template-based dispatch provide a practical foundation for continuously rebasing custom functionality onto upstream WebRTC while safely validating each release through A/B testing.
As AI accelerates software development, stronger safeguards are needed to prevent faster mistakes from becoming larger incidents. Meta’s Configurations team uses canarying, progressive rollouts, health checks, and monitoring to detect regressions early. Data and AI also help reduce alert noise and speed up identifying the changes responsible for failures.
## Safe Configuration Rollouts
- Meta deploys configuration changes gradually rather than releasing them everywhere at once.
- Canarying exposes changes to a small subset of systems or users first.
- Progressive rollouts expand the deployment only when monitoring indicates that the change is healthy.
- These practices limit the impact of faulty configurations and provide opportunities to stop or reverse a rollout.
## Monitoring and Health Checks
- Automated health checks and operational signals help identify regressions soon after deployment.
- Monitoring provides evidence for deciding whether a rollout should continue, pause, or be rolled back.
- Early detection is especially important at Meta’s scale, where a small configuration error can affect many systems.
## Learning from Incidents
- Incident reviews focus on improving tools, processes, and safeguards rather than assigning blame to individuals.
- The goal is to make future failures less likely and reduce their potential impact.
- These reviews turn operational problems into improvements across the configuration management system.
## AI-Assisted Operations
- Data-driven techniques reduce alert noise so engineers can focus on meaningful signals.
- AI and machine learning help speed up bisection, narrowing down which change introduced a problem.
- Faster diagnosis can shorten recovery times and make progressive deployment practices more effective.
The episode recommends combining gradual releases, strong observability, blameless incident reviews, and AI-assisted analysis to keep increasingly rapid development safe at scale.
AI coding assistants struggle when they lack a map of a large, proprietary codebase. To address this, the team built a pre-compute system using 50+ specialized agents that analyzed over 4,100 files across four repositories and three languages, producing 59 concise context files. The approach gave agents complete module coverage, captured previously undocumented tribal knowledge, reduced tool calls by about 40%, and made complex development tasks much faster.
## The Problem: Powerful Tools Without Codebase Context
- The pipeline combines Python configuration, C++ services, and Hack automation across multiple repositories.
- A seemingly simple change, such as adding a data field, can affect:
- Configuration registries
- Routing logic
- DAG composition
- Validation rules
- C++ code generation
- Automation scripts
- AI agents often explored repeatedly, guessed at conventions, and produced code that compiled but was subtly incorrect.
- Important examples of missing context included:
- Different field names for the same operation in separate configuration modes
- “Deprecated” enum values that must remain for serialization compatibility
- Hidden intermediate field names used between pipeline stages
## The Pre-Compute Approach
The team used a large-context model and orchestrated specialized agents in several phases:
- Two agents explored and mapped the codebase.
- Eleven analysts read every file and answered five questions:
- What does the module configure?
- How is it commonly modified?
- What non-obvious patterns can cause failures?
- What are its cross-module dependencies?
- What tribal knowledge is hidden in comments?
- Writers generated context files.
- More than ten critic passes reviewed quality across three rounds.
- Fixers, upgraders, gap-fillers, prompt testers, and final critics corrected and validated the results.
- In total, more than 50 specialized tasks were coordinated in one session.
This process uncovered over 50 non-obvious design patterns, including naming conventions and append-only identifier rules that were not documented elsewhere.
## Context Files: “A Compass, Not an Encyclopedia”
Each of the 59 context files is intentionally short—about 25–35 lines or roughly 1,000 tokens—and contains:
- Quick Commands for common operations
- Key Files limited to the most relevant three to five files
- Non-Obvious Patterns
- See Also references to related modules
Together, the files use less than 0.1% of a modern model’s context window. They are designed for targeted, opt-in use rather than being loaded into every task.
## Routing and Dependency Navigation
- An orchestration layer routes natural-language requests to the appropriate tool.
- Operational questions can trigger dashboard scans and matching against more than 85 historical incident patterns.
- Development requests can launch configuration generation and multi-phase validation.
- A cross-repository dependency index and data-flow maps show how changes propagate.
- Dependency questions that previously required about 6,000 tokens of exploration can be answered through a graph lookup using roughly 200 tokens.
## Results and Quality Controls
- Preliminary tests across six tasks showed approximately 40% fewer tool calls and tokens.
- Work that previously required around two days of research and engineer consultation took about 30 minutes.
- Critic reviews raised quality scores from 3.65 to 4.20 out of 5.
- Every referenced file path was verified, with no hallucinated paths.
- Coverage expanded from navigation guidance for roughly 5% of modules to all 4,100+ files across three repositories.
## Why This Differs from Generic Context Files
Research has found that AI-generated context files can reduce agent performance on familiar open-source projects. The team argues that this result does not directly apply to proprietary systems whose conventions and tribal knowledge are absent from model training data.
Their approach addresses common problems by making context:
- Concise rather than encyclopedic
- Opt-in rather than always loaded
- Quality-gated through independent critics
- Continuously refreshed to prevent stale information
Without this context, agents typically spend 15–25 tool calls exploring and remain vulnerable to subtle domain-specific errors.
## Keeping the Knowledge Fresh
Automated jobs refresh the system every few weeks by:
- Validating file paths
- Detecting coverage gaps
- Re-running critic reviews
- Finding and repairing stale references
- Updating routing and dependency information
The system treats AI not merely as a consumer of documentation, but as the engine that creates and maintains it.
## Applying the Method Elsewhere
Teams can adapt the approach by:
- Identifying where agents most often fail due to undocumented conventions or dependencies
- Applying the five-question analysis framework to each module
- Keeping context files short and action-oriented
- Using independent quality critics before publishing generated guidance
- Automating freshness checks and self-repair
The practical recommendation is to build a small, targeted, continuously maintained knowledge layer for proprietary codebases. Concise navigation and dependency context can reduce exploration costs while preventing the subtle errors that arise when agents lack domain-specific understanding.
KernelEvolve is Meta’s agentic system for automating the creation and optimization of low-level AI kernels across diverse hardware. It treats kernel tuning as a search problem rather than one-shot code generation, evaluating hundreds of alternatives with profiling and diagnostics. The system reduces optimization work from weeks to hours and has delivered over 60% higher inference throughput for an Ads model on NVIDIA GPUs and over 25% higher training throughput on Meta’s MTIA chips.
## Kernel Optimization at Meta
- AI models rely on optimized kernels that translate high-level operations into hardware-specific instructions.
- Meta runs models across NVIDIA GPUs, AMD GPUs, custom MTIA accelerators, and CPUs.
- Production workloads require many custom operators beyond standard GEMMs and convolutions available in vendor libraries.
- Kernels must be developed and tuned for each combination of:
- Hardware type and generation
- Model architecture
- Operator type
## The Challenge of Hardware Heterogeneity
- NVIDIA, AMD, MTIA, and CPU platforms differ in:
- Memory architectures and hierarchies
- Instruction sets
- Execution models
- Supported numeric data types
- A kernel optimized for one platform may perform poorly or fail on another.
- Hardware generations also require new optimization strategies. Meta’s MTIA roadmap includes four generations, from MTIA 300 through MTIA 500, in two years.
- Manual tuning by kernel specialists cannot keep pace with these changes.
## Increasing Model and Operator Complexity
- Meta’s recommendation systems have evolved from embedding-based models to sequence models with attention, GEM, and LLM-scale models such as Meta Adaptive Ranking Model.
- Each new model generation introduces operators that earlier systems did not require.
- Multiple model families may be involved in a single ads-serving request.
- As model architectures and operator inventories grow, the number of kernel configurations expands rapidly into the thousands.
## How KernelEvolve Works
- KernelEvolve generates candidate implementations in languages and DSLs including:
- Triton, Cute DSL, and FlyDSL
- CUDA, HIP, and MTIA C++
- A dedicated job harness compiles, runs, profiles, and evaluates each candidate.
- Performance results, correctness checks, and diagnostic information are fed back to the LLM.
- The system continuously searches through hundreds of alternatives instead of stopping at the first plausible implementation.
- Its automated workflow includes profiling, optimization, testing, and cross-hardware debugging.
## Results and Broader Impact
- KernelEvolve improved Andromeda Ads inference throughput by more than 60% on NVIDIA GPUs.
- It improved training throughput for an ads model by more than 25% on Meta’s MTIA silicon.
- The system operates across both public and proprietary hardware.
- In production, it optimizes code supporting trillions of daily inference requests.
- By automating kernel development, Meta can enable new hardware and adapt to changing model architectures with substantially less engineering effort.
KernelEvolve turns kernel development from a manual, expert-driven bottleneck into a continuous automated process. Its search-based approach is particularly valuable as Meta’s hardware portfolio and model architectures continue to diversify.
Meta’s Adaptive Ranking Model scales ad recommendation models toward LLM-level complexity without sacrificing sub-second latency or cost efficiency. It replaces uniform inference with intelligent request routing, selecting the most appropriate model for each user context. Since launching on Instagram in Q4 2025, it reportedly increased ad conversions by 3% and click-through rates by 5% among targeted users.
## The Inference Trilemma
- More complex models require substantially more compute and memory.
- Ads must still be selected and delivered within a sub-second latency budget.
- Simply adding hardware is too expensive for a service operating at global scale.
- Adaptive Ranking Model addresses these competing demands by matching model complexity to each request’s context and intent.
## Three Core Innovations
- **Inference-efficient model scaling**
- Reaches complexity comparable to roughly 10 GFLOPs per token in advanced LLMs.
- Maintains approximately 100 ms bounded latency—an order of magnitude faster than standard LLM inference.
- **Model/system co-design**
- Aligns model architectures with hardware and silicon capabilities.
- Achieves approximately 35% model FLOPs utilization across different hardware types.
- **Reimagined serving infrastructure**
- Uses multi-card GPU systems to overcome the memory limits of individual devices.
- Supports models with roughly one trillion parameters.
## Request-Oriented Computation
- Traditional ranking processes each user-ad pair independently, duplicating expensive user-side computations.
- Request-Oriented Optimization computes dense user signals once per request and reuses them across all candidate ads.
- Request-Oriented Computation Sharing and In-Kernel Broadcast distribute shared embeddings directly within GPU kernels.
- These techniques change scaling behavior from linear toward sub-linear while reducing memory-bandwidth pressure.
- Long user behavior sequences can also be processed once per request and reused across candidates.
- A centralized key-value store avoids duplicating user logs and joins them with training data when needed, reducing storage and serving costs.
## Wukong Turbo Architecture
- Wukong Turbo builds on Meta’s Wukong architecture, which combines:
- Stackable factorization machines
- Sequence learning
- Cross-layer attention
- A **No-Bias** design removes unstable terms, improving throughput without increasing parameter counts or FLOPs.
- Small parameter delegation moves selected parameters from Fully Sharded Data Parallel (FSDP) to Distributed Data Parallel (DDP), reducing network and memory overhead.
- Sparsity-based simplification removes redundant linear-layer components.
- Together, these changes improve numerical stability and throughput while preserving the sub-second inference target.
## Holistic Latency Optimization
- The system also targets feature preprocessing, which can create client-memory pressure and leave GPUs waiting for data.
- The article indicates that Adaptive Ranking Model addresses this bottleneck through end-to-end latency optimization and GPU-based preprocessing, though the provided text ends before detailing the implementation.
Adaptive Ranking Model demonstrates that LLM-scale recommendation intelligence can be practical in real-time advertising when model architecture, request execution, hardware, and serving infrastructure are designed together. Its central recommendation is to avoid one-size-fits-all inference and instead allocate computation dynamically according to each request’s value and complexity.
Meta is expanding its use of AI to help concrete producers create stronger, more sustainable, and more domestically sourced mixes. Its new open-source model, Bayesian Optimization for Concrete (BOxCrete), uses existing performance data and lab results to identify promising formulations faster than traditional trial-and-error methods. Early projects show that AI can improve curing speed and reduce cracking while supporting greater use of U.S.-made cement and materials.
## The Case for AI-Designed Concrete
- The U.S. produces about 400 million cubic yards of concrete annually.
- Although ready-mix concrete is generally produced domestically, roughly 20–25% of cement consumption is supplied by imports.
- Concrete mix designers must balance:
- Structural strength
- Curing speed
- Workability and slump
- Cost
- Sustainability
- Traditional design depends on laboratory experimentation, engineer judgment, and historical knowledge, making it slow and expensive to adapt.
- Different cements have different chemistries, so a formulation that works with one cement may fail with another.
## Supporting Domestic Cement Production
- Greater use of U.S.-made cement could strengthen domestic manufacturing, jobs, and investment.
- Reshoring and foreign direct investment have returned more than 1.1 million jobs to the U.S. since 2020.
- The cement and concrete sector contributes over $130 billion annually and supports approximately 600,000 jobs.
- AI can help producers reformulate mixes around locally available materials without compromising performance.
## BOxCrete and Open Data
- Meta is releasing BOxCrete on GitHub as an open-source model for concrete mix design.
- Compared with earlier models, BOxCrete is more robust to noisy data and can predict concrete slump, an important measure of workability.
- Meta is also publishing the foundational dataset used to develop the concrete mix for its Rosemount, Minnesota, data center.
- The associated research paper describes the model, data, and methodology.
## Results in Minnesota
- Meta, Amrize, Mortenson, and the University of Illinois used BOxCrete to design a mix for a data center foundation.
- The mix used domestically sourced materials.
- It reached full structural strength 43% faster than the original formulation.
- It reduced cracking risk by nearly 10%.
- After meeting structural requirements, the mix was approved for use in additional parts of the data center.
## Industry Partnerships in Illinois and Pennsylvania
- Meta is working with Amrize and the University of Illinois to apply AI to industrial-scale concrete production.
- Amrize operates 18 cement plants, 141 cement terminals, and 269 ready-mix sites across North America.
- Amrize has introduced a “Made in America” cement label and announced nearly $1 billion in planned 2026 investments, partly aimed at increasing domestic cement production.
- Pennsylvania-based Quadrel integrated Meta’s open-source framework into its ready-mix software.
- Quadrel uses the technology for data preprocessing, batch and test normalization, feature engineering, customer-specific model training, and quality-control workflows.
- Its models improve continuously as new field-test results are incorporated.
## Adaptive Experimentation
- Meta’s Adaptive Experimentation platform uses Bayesian optimization to search the large space of possible concrete formulations.
- The system:
- Learns from historical mix designs, laboratory results, and performance metrics.
- Proposes candidate mixes likely to satisfy target specifications.
- Compares the performance of domestic and imported materials.
- Applies technical and ingredient constraints before testing.
- Updates its predictions after each new experiment.
Meta’s work suggests that open-source AI can make concrete development faster, more data-driven, and better suited to domestic materials. Producers can use BOxCrete and adaptive experimentation to reduce laboratory costs, improve performance, and support more sustainable and resilient U.S. cement and concrete supply chains.
Friend bubbles in Facebook Reels surface videos that friends have liked or interacted with, combining content discovery with opportunities for conversation. The system uses machine-learning models to estimate viewer-friend closeness, retrieve relevant friend-interacted videos, and rank them alongside conventional recommendation signals. Its goal is not to show the most bubbles possible, but to identify meaningful connections and content that can drive both engagement and social interaction.
## System Architecture
- The recommendation system combines:
- **Viewer-friend closeness**, determining whose interactions matter most.
- **Video relevance**, determining which friend-interacted videos best fit the viewer.
- Multiple friends interacting with the same video can indicate stronger shared interest.
- Social discovery and engagement reinforce one another: relevant friend content encourages interaction, which improves the system’s understanding of the social graph.
## Modeling Viewer-Friend Closeness
- Facebook uses two complementary models:
- A survey-based model estimating real-world relationship strength.
- An activity-based model estimating closeness from on-platform behavior.
- The survey model considers:
- Mutual friends and interaction patterns.
- User-provided attributes such as location.
- Number of friends and posts shared.
- Communication frequency and other survey proxies for offline closeness.
- Users are periodically asked whether they feel close to a randomly selected connection.
- The model is refreshed regularly and performs weekly inference across trillions of friend relationships.
- The activity-based model learns from likes, comments, reshares, and interactions occurring after bubbles are shown.
- Facebook prioritizes connection quality over quantity: larger friend networks may create more opportunities, but the system aims to surface only relationships likely to make recommendations meaningful.
## Retrieving and Ranking Friend Content
### Expanding Candidate Retrieval
- The retrieval stage explicitly sources videos interacted with by close friends.
- This expands the recommendation funnel so high-quality friend content can reach downstream ranking systems.
- Without dedicated retrieval, relevant friend videos might never become candidates.
### Adding Social Context to Ranking Models
- Friend-interacted videos could rank poorly when models lacked viewer-friend closeness information.
- The system added bubble interaction signals and relationship-strength features to early- and late-stage multi-task, multi-label ranking models.
- These features help models distinguish social relevance from ordinary content-interest signals.
- Feedback from bubble impressions and resulting interactions continuously flows back into model training.
- Ranking objectives consider:
- Watch time.
- Likes and comments.
- The probability of engagement after a bubble impression: `P(video engagement | bubble impression)`.
- Tunable weights balance entertainment and video quality against social goals such as discovering friends’ interests and encouraging conversation.
## Client Infrastructure and Reels Performance
- Friend-bubble metadata had to be integrated without harming Reels’ core experience.
- The implementation targeted:
- Smooth scrolling.
- No additional loading latency.
- Low CPU usage during metadata retrieval and processing.
- Facebook aligned bubble metadata retrieval with the existing video prefetch window, which already loads metadata, thumbnails, and buffered content before playback.
- This allows the system to reuse cached results and avoid adding unnecessary work during scrolling.
Friend bubbles work best when social relevance and content quality are optimized together. By combining relationship models, friend-aware retrieval and ranking, feedback-driven learning, and performance-conscious client infrastructure, Facebook turns shared video interests into lightweight opportunities for discovery and conversation.
Meta’s Ranking Engineer Agent (REA) autonomously manages much of the ads-ranking ML experimentation lifecycle, from generating hypotheses and launching training jobs to debugging failures and analyzing results. Unlike session-based AI assistants, REA maintains context across workflows lasting days or weeks, while engineers retain oversight at strategic checkpoints. In its first production rollout, REA doubled average model accuracy across six models and helped three engineers produce launch proposals for eight models—about five times the historical engineering output.
## The Bottleneck in Traditional ML Experimentation
- Meta’s advertising systems rely on large, complex ML models serving billions of users across Facebook, Instagram, Messenger, and WhatsApp.
- Improving these models traditionally requires engineers to:
- Form hypotheses
- Design experiments
- Launch training jobs
- Debug failures
- Analyze results
- Iterate on promising approaches
- Each cycle can take days or weeks, and mature models make meaningful improvements increasingly difficult to find.
- The sequential, hands-on process became a bottleneck to experimentation and innovation.
## REA as an Autonomous ML Agent
- Existing ML AI tools generally assist with isolated tasks such as drafting hypotheses, writing configurations, or interpreting logs.
- REA instead coordinates the full experimentation process and advances it without continuous prompting.
- Its design addresses three central challenges:
- **Long-running workflows:** Persistent state and memory allow REA to manage multiday or multiweek experiments.
- **Hypothesis quality:** It combines historical experiment data with current ML research.
- **Operational resilience:** It handles failures and compute limits within engineer-approved safeguards.
## Hibernate-and-Wake Workflow Management
- Training jobs may run for hours or days, so REA delegates waiting to a background system.
- It hibernates to conserve resources and automatically wakes when jobs finish.
- This lets it preserve context and continue experiments without constant human supervision.
- REA is built on Meta’s Confucius agent framework, which provides:
- Code-generation capabilities
- Integration with job schedulers
- Experiment tracking
- Codebase navigation tools
## Dual-Source Hypothesis Generation
- REA draws ideas from two systems:
- **Historical Insights Database:** A repository of previous experiments, successes, and failures used for pattern recognition and in-context learning.
- **ML Research Agent:** A research component that examines baseline configurations and proposes new optimization strategies.
- Combining these sources produces configurations that may not emerge from either source alone.
- Some of REA’s strongest improvements resulted from combining model architecture changes with training-efficiency techniques.
## Three-Phase Experiment Planning
- Before running experiments, REA proposes an exploration plan, estimates GPU costs, and obtains engineer confirmation.
- Its typical strategy includes:
- **Validation:** Test individual hypotheses in parallel to establish baselines.
- **Combination:** Combine promising ideas to identify synergistic effects.
- **Exploitation:** Intensively optimize the strongest candidates within the approved compute budget.
## Autonomous Failure Handling and Safeguards
- REA adapts to infrastructure problems, unexpected errors, poor results, and compute constraints without escalating every issue to an engineer.
- It uses runbooks and diagnostic reasoning to:
- Exclude jobs with clear out-of-memory failures
- Detect training instability, such as exploding losses
- Debug preliminary infrastructure failures
- Reprioritize experiments when results are weak
- Its autonomy is constrained by:
- Access limited to Meta’s ads-ranking codebase
- Explicit engineer approval through preflight reviews
- Confirmed GPU budgets
- Automatic pausing or stopping when thresholds are reached
## Production Results
- Across six models, REA-driven iterations achieved approximately **2× the average model accuracy** compared with baseline.
- Three engineers produced proposals to launch improvements for eight models.
- Historically, that volume of work would have required roughly two engineers per model, resulting in about **5× greater engineering output** with REA.
REA demonstrates that autonomous agents can improve ML experimentation by handling long-running execution, generating broader hypotheses, and recovering from routine failures. The most effective deployment model combines substantial agent autonomy with explicit compute limits, codebase restrictions, and human review at major strategic decisions.
Updating security-sensitive APIs across a massive mobile codebase is difficult because vulnerable patterns may appear across hundreds of call sites and millions of lines of code. Meta’s Product Security team addresses this through secure-by-default Android frameworks and generative AI that automates migrations to those frameworks. The approach enables security patches to be proposed, validated, and submitted with minimal effort from code owners.
## Secure-by-Default Mobile Frameworks
- Meta wraps potentially unsafe Android OS APIs in frameworks designed to make secure implementations the easiest option.
- Developers are guided toward safer behavior by default rather than being expected to recognize and avoid every security risk manually.
- This strategy helps prevent a single vulnerability class from recurring across Meta’s many mobile applications.
## AI-Assisted Code Migration
- Generative AI is used to migrate existing code from unsafe APIs to the new secure frameworks.
- The system operates across millions of lines of code and numerous call sites.
- It can propose security changes, validate them, and submit patches for review.
- This reduces the manual work required from the engineers responsible for each application or codebase.
## Security at Massive Scale
- Meta’s scale—thousands of engineers, multiple apps, and billions of users—makes conventional security updates difficult to coordinate.
- The initiative combines framework design, automation, and engineering ownership to reduce friction while maintaining validation.
- The accompanying Meta Tech Podcast episode features Product Security engineers Alex and Tanu discussing the challenges and lessons from this effort.
Meta’s approach demonstrates that large-scale mobile security improvements are most practical when safer APIs and automated migration tools work together, allowing secure changes to spread broadly without requiring every engineer to perform the migration manually.
Advanced Browsing Protection (ABP) extends Messenger’s Safe Browsing beyond on-device detection by checking links against a frequently updated database of millions of potentially malicious websites. Its central challenge is balancing effective URL matching with privacy: Messenger must identify unsafe links without revealing users’ exact queries or distributing the entire blocklist. ABP combines private information retrieval, cryptographic techniques, sharding, and client-side preprocessing to achieve this balance.
## Safe Browsing Within End-to-End Encryption
- Messenger’s end-to-end encryption protects messages and calls, but it does not by itself protect users from malicious links.
- Safe Browsing warns users when a link may lead to phishing, credential theft, or other harmful activity.
- The standard feature uses on-device models.
- Advanced Browsing Protection adds access to a continually updated watchlist containing millions of potentially malicious websites.
## Private Information Retrieval as the Foundation
- Private information retrieval (PIR) allows a client to ask whether an item exists in a server-held database while revealing as little as possible about the query.
- Sending the full database to each device is impractical because:
- The database is large and frequently updated.
- Exposing the complete list could help attackers evade detection.
- Existing PIR approaches use oblivious pseudorandom functions (OPRFs) and divide the database into buckets or shards.
- ABP had to address two limitations:
- OPRFs are designed for exact matches, whereas URLs require prefix matching.
- The client generally must identify which bucket to query, creating a privacy-versus-efficiency tradeoff.
- More advanced lattice-based constructions may reduce the need for sharding, but they were not yet practical at ABP’s scale.
## Privacy-Preserving Prefix Matching for URLs
- A database entry such as `example.com` should match a longer URL such as `example.com/a/b/index.html`.
- Querying every prefix separately would work functionally:
- `example.com`
- `example.com/a`
- `example.com/a/b`
- `example.com/a/b/index.html`
- However, each query can leak information about the original URL. If one query leaks `B` bits and there are `P` prefixes, the total leakage may reach `P × B` bits.
- ABP instead groups URLs by domain so the client makes one bucket request and checks path prefixes within that bucket.
- This reduces query leakage but creates uneven bucket sizes.
- Domains such as link-shortening services may contain huge numbers of URLs, producing oversized buckets and potentially large padded responses.
## Preprocessing Rulesets to Balance Buckets
- The server addresses bucket imbalance by generating a ruleset that tells clients how to process URLs before selecting a bucket.
- Each rule maps an 8-byte hash prefix to a number of path segments that should be appended to the current URL before hashing again.
- For example:
- The client hashes `example.com`.
- If the hash matches a ruleset entry, it appends specified path segments, such as `/a/b`.
- It hashes the resulting URL again and repeats the process.
- When no ruleset entry matches, the client uses the first two bytes of the final hash as the bucket identifier.
- The server builds the ruleset iteratively:
- It initially hashes URLs by domain.
- It identifies the largest bucket.
- It finds the most common domain in that bucket.
- It adds rules that incorporate additional URL path segments to split the oversized bucket.
- Clients receive the ruleset in advance and perform the same deterministic processing during lookups.
ABP’s design demonstrates how privacy-preserving lookup can support real-world URL semantics without exposing users’ links. The combination of PIR, controlled sharding, prefix-aware processing, and adaptive rulesets allows Messenger to warn about malicious sites while limiting what the server learns about each user’s browsing query.
FFmpeg is central to Meta’s media infrastructure, running tens of billions of times daily. Meta’s heavily modified internal fork became difficult to maintain as upstream FFmpeg evolved, so the company collaborated with the FFmpeg community to upstream key capabilities. Threaded multi-lane transcoding and real-time quality metrics now allow Meta to rely entirely on upstream FFmpeg for VOD and livestreaming.
## Moving Away from an Internal FFmpeg Fork
- Meta’s fork had diverged substantially from upstream while supporting specialized requirements.
- Maintaining both the fork and newer open-source FFmpeg versions created:
- Divergent feature sets
- Difficult rebases
- Increased regression risk
- Collaboration with FFmpeg developers, FFlabs, and VideoLAN enabled the needed functionality to be integrated upstream.
## More Efficient Multi-Lane Transcoding
- Meta creates multiple DASH encodings for each uploaded video, varying resolution, codec, framerate, and quality.
- Running separate FFmpeg processes wastes resources by repeatedly decoding the same source and starting multiple processes.
- A single FFmpeg command can decode once and send frames to multiple encoder instances.
- Earlier FFmpeg versions still processed those encoders serially for each frame.
- Improved parallel encoder threading, introduced from FFmpeg 6.0 and completed in FFmpeg 8.0, allows encoder instances to run concurrently.
- The change reduces compute overhead across Meta’s more than one billion daily video uploads.
## Real-Time Quality Metrics for Livestreaming
- Metrics such as PSNR, SSIM, and VMAF measure compression-related visual quality loss.
- Traditional FFmpeg workflows calculate these metrics after encoding, which is unsuitable for live video.
- Meta needed a decoder after each encoder to reconstruct compressed frames and compare them with the original frames during processing.
- FFmpeg’s “in-loop” decoding, available beginning in FFmpeg 7.0, enables real-time per-lane quality measurement within one command.
## Deciding What to Upstream
- Meta aims to upstream changes that provide broad value to FFmpeg users.
- Infrastructure-specific patches are kept internal when they cannot be reasonably tested or used by the wider community.
- Meta’s custom MSVP transcoding ASIC is integrated through FFmpeg’s standard hardware APIs, alongside NVIDIA, AMD, and Intel technologies.
- Because external developers cannot access MSVP hardware, Meta maintains and validates those internal patches itself.
## Continued Investment in FFmpeg
- Upstream improvements allowed Meta to retire its internal fork for all VOD and livestreaming pipelines.
- Standardized hardware interfaces make it easier to combine specialized ASICs with software-based encoding and decoding.
- Meta plans to continue contributing to FFmpeg to improve efficiency, reliability, codec support, and media experiences across the industry.
Meta’s experience demonstrates that upstreaming broadly useful infrastructure improvements can reduce long-term maintenance costs while strengthening the entire FFmpeg ecosystem.
