meta·Mar 2, 2026·2 min readCurated summary

Investing in Infrastructure: Meta’s Renewed Commitment to jemalloc

Meta is renewing its commitment to jemalloc, recognizing its long-term role in delivering reliable and efficient infrastructure alongside the Linux kernel and compilers. After acknowledging that short-term decisions created technical debt and slowed development, Meta has unarchived the original repository and begun rebuilding a long-term roadmap. The effort will focus on modernization, reduced maintenance, hardware adaptation, and closer collaboration with the open-source community. ## Why jemalloc Matters - jemalloc is a high-performance memory allocator used as a foundational component of Meta’s software stack. - It has adapted to changing hardware and workloads over time. - Its impact is comparable to other core infrastructure components such as the Linux kernel and compilers. ## Reflecting on Technical Debt - Meta says recent development gradually moved away from the rigorous engineering principles needed for foundational software. - Some changes provided short-term benefits but introduced technical debt. - That debt increased maintenance burdens and slowed future progress. - Community feedback, including discussions with jemalloc founder Jason Evans, prompted Meta to reassess its stewardship. ## Renewed Development Priorities - **Technical debt reduction:** Clean up, refactor, and improve the codebase to make jemalloc more efficient, reliable, and maintainable. - **Huge-page allocation:** Continue improving the hugepage allocator (HPA) and its use of transparent hugepages (THP) to improve CPU efficiency. - **Memory efficiency:** Optimize memory packing, caching, and purging mechanisms. - **AArch64 support:** Improve out-of-the-box performance on ARM64 systems. - **Hardware and workload adaptation:** Continue evolving jemalloc for current and emerging platforms. ## Open-Source Collaboration - The original jemalloc repository has been unarchived. - Meta intends to work with the open-source community on the project’s future. - The company acknowledges that renewed trust must come through measurable improvements and sustained development. - Community members are invited to provide feedback, contributions, and collaboration. Meta’s practical next step is to demonstrate its renewed commitment through code cleanup, performance improvements, and transparent collaboration. The project’s long-term health will depend on consistent execution rather than statements alone.

Read original(opens in new tab)

Open SourcePerformance OptimizationLinuxMemory Allocation+3

meta·Feb 24, 2026·3 min readCurated summary

RCCLX: Innovating GPU communications on AMD platforms

RCCLX is Meta’s open-source enhancement of RCCL for AMD GPUs, integrated with Torchcomms to support portable distributed AI workloads. It introduces Direct Data Access (DDA) and low-precision collectives, targeting communication bottlenecks in inference and training. On AMD MI300X systems, these optimizations deliver lower latency and higher throughput while maintaining acceptable accuracy. ## RCCLX and Torchcomms Integration - RCCLX is based on RCCL and tested on Meta’s internal workloads. - It integrates CTran transport technology for AMD platforms. - CTran enables features such as `AllToAllvDynamic`, a GPU-resident collective; additional CTran capabilities are planned for future releases. - Through Torchcomms, applications can use a common communication API across AMD, NVIDIA, and other backends without major code changes. - RCCLX is intended to achieve feature parity with Meta’s NCCLX backend for NVIDIA systems. ## Direct Data Access for Intra-Node Collectives - LLM inference has two distinct phases: - **Prefill** is compute-bound and generates the model’s key-value cache. - **Decoding** is memory-bound and generates tokens incrementally. - Tensor parallelism can make AllReduce responsible for up to 30% of end-to-end latency. - RCCLX introduces two DDA algorithms: - **DDA flat** lets each rank directly read other ranks’ memory and perform local reductions. It reduces latency from O(N) to O(1) for small messages by increasing data exchange from O(n) to O(n²). - **DDA tree** divides AllReduce into reduce-scatter and all-gather phases, retaining ring-like data movement while reducing latency for somewhat larger messages. - On AMD MI300X GPUs, DDA improves over RCCL by: - 10–50% for decode workloads. - 10–30% for prefill workloads. - Approximately 10% lower time-to-incremental-token. ## Low-Precision Collectives - RCCLX provides optimized low-precision versions of AllReduce, AllGather, AlltoAll, and ReduceScatter. - These target AMD Instinct MI300 and MI350 GPUs and support FP32 and BF16 inputs. - FP8 quantization provides up to 4:1 compression, reducing communication overhead for messages of at least 16 MB. - Parallel peer-to-peer mesh communication uses AMD Infinity Fabric for bandwidth and low latency. - Computation remains in FP32 to improve numerical stability. - Users can enable the feature with: ```bash RCCL_LOW_PRECISION_ENABLE=1 ``` - Internal evaluations showed: - About a 0.3% change on GSM8K accuracy evaluations. - 9–10% lower latency. - Approximately 7% higher throughput. - The current implementation is tuned for single-node deployments. ## Getting Started - Install Torchcomms with the RCCLX backend. - Create an RCCLX communicator through Torchcomms using the `"rcclx"` backend and a HIP device. - Existing Torchcomms operations such as `allreduce` can then run without backend-specific API changes. - Distributed initialization uses standard `torchrun` environment variables such as `MASTER_ADDR`, `MASTER_PORT`, `RANK`, and `WORLD_SIZE`. RCCLX is positioned as a practical way to improve AMD-based AI training and inference without requiring applications to adopt a new communication API. Teams can use DDA for lower inference latency and selectively enable low-precision collectives for higher throughput, while evaluating numerical accuracy for their own workloads.

Read original(opens in new tab)

Gpu AccelerationDistributed ComputingLlm InferenceRcclx+4

meta·Feb 11, 2026·2 min readCurated summary

The Death of Traditional Testing: Agentic Development Broke a 50-Year-Old Field, JiTTesting Can Revive It

Just-in-Time Tests (JiTTests) are an LLM-driven testing approach designed for fast, agentic software development. Instead of maintaining static test suites, the system generates tests for each code change, simulates likely faults, and reports only meaningful regressions. The goal is to reduce test maintenance and false positives while catching serious bugs before production. ## Limitations of Traditional Testing - Tests are manually written as code changes enter the system. - They must account for both current behavior and unknown future changes. - This often leads to: - Tests that fail to detect relevant bugs. - False positives when intended changes break outdated assumptions. - Ongoing maintenance and review costs. - Agentic development increases the volume and speed of changes, making these problems harder and more expensive to manage. ## How Catching JiTTests Work - A new code change or pull request is submitted. - An LLM infers the likely intent of the change. - The system creates mutants—versions of the code containing deliberately introduced faults. - It generates and runs tests designed to expose those faults. - Rule-based and LLM-based assessors evaluate failures and filter out likely false positives. - Engineers receive a focused report when the system identifies an unexpected behavior change. Because the tests are tailored to a specific change, they can reason about intended behavior and distinguish legitimate updates from regressions. ## Benefits for Agentic Development - Tests are generated on demand and do not remain in the codebase. - There is no ongoing test maintenance or test-code review. - Each test is specific to the change being evaluated. - Tests automatically adapt as the code evolves. - Human attention is required mainly when an actual bug is detected. - Testing shifts from measuring generic code quality to determining whether a specific change introduces a real fault. Catching JiTTests are presented as a way to make testing scale with AI-assisted development by moving routine test creation and maintenance from engineers to automated systems.

Read original(opens in new tab)

Large Language ModelsAutomated TestingRegression TestingJust In Time Testing+3

meta·Feb 9, 2026·3 min readCurated summary

Building Prometheus: How Backend Aggregation Enables Gigawatt-Scale AI Clusters

Meta uses Backend Aggregation (BAG) as a high-capacity Ethernet super-spine to connect tens of thousands of GPUs across data centers and regions. In the Prometheus AI cluster, BAG links regional networks and Meta’s backbone while bridging two L2 fabric technologies: Disaggregated Schedule Fabric (DSF) and Non-Scheduled Fabric (NSF). Its modular hardware, resilient topologies, and advanced routing are designed to deliver reliable, petabit-scale connectivity for a gigawatt-scale AI system. ## Backend Aggregation’s Role - BAG interconnects multiple spine fabrics across data centers and regions. - It aggregates regional networks and connects them to Meta’s backbone. - Inter-BAG capacity can reach 16–48 petabits per second per regional pair. - Prometheus will span multiple buildings and connect tens of thousands of GPUs. ## Regional BAG Connectivity - BAG layers are distributed regionally to serve groups of L2 fabrics while respecting distance, latency, and buffer constraints. - Two connection topologies are used: - **Planar topology:** One-to-one connections between BAG switches in different regions; simpler to manage but creates more concentrated failure domains. - **Spread topology:** Links are distributed across switches and planes, improving path diversity and resilience. - The choice depends on site size and available fiber. ## Connecting DSF and NSF Fabrics - Meta’s L2 networks use both: - **Disaggregated Schedule Fabric (DSF)** - **Non-Scheduled Fabric (NSF)** - DSF zones across multiple buildings connect to BAG through backend edge pods. - NSF connects to BAG planes through matching Spine Training Switches. - Oversubscription is carefully managed: - L2-to-BAG oversubscription is typically around 4.5:1. - One NSF example has an effective ratio of 4.98:1. - BAG-to-BAG ratios vary by region and link capacity. ## Hardware and Routing - BAG uses modular chassis with Jericho3 ASIC line cards. - Each line card supports up to 432 800G ports. - Larger central-hub chassis support many spoke connections and long-distance links. - eBGP with link-bandwidth attributes enables Unequal Cost Multipath (UCMP), improving load balancing and failure recovery. - BAG-to-BAG links use MACsec for network security. ## Resilience and Failure Management - The design includes detailed port striping, IP addressing, and failure-domain analysis. - Failures are evaluated at the BAG, data-hall, and power-distribution levels. - Mitigation techniques include: - Draining affected BAG planes - Conditional route aggregation - Reducing blackholing risks during failures ## Managing Long-Distance Links - Distributed BAG architecture keeps L2-to-edge distances short, which benefits shallow-buffer NSF switches. - Longer BAG-to-BAG connections require deep-buffer switches. - These buffers provide headroom for lossless congestion-control mechanisms such as Priority Flow Control (PFC). ## Broader Impact BAG provides the networking foundation for Prometheus and future AI clusters. By combining regional aggregation, high-density hardware, resilient connection topologies, and fabric interoperability, Meta can scale AI infrastructure across multiple data centers while maintaining bandwidth, reliability, and operational flexibility.

Read original(opens in new tab)

Ai InfrastructureBackend AggregationGpu NetworkingEthernet+4

meta·Feb 4, 2026·3 min readCurated summary

No Display? No Problem: Cross-Device Passkey Authentication for XR Devices

Passkeys provide phishing-resistant authentication, but standard cross-device flows depend on QR codes displayed on the device being authenticated. This creates a problem for XR headsets, smart-home hubs, sensors, and other devices with no accessible screen. The proposed solution uses an authenticated companion app to transport the FIDO hybrid-flow request, preserving proximity, trust, and standard WebAuthn security without requiring a QR code. ## The Challenge: Screenless Devices Cannot Display QR Codes - Traditional cross-device passkey authentication uses: - A QR code displayed by the desktop or other target device. - Bluetooth or NFC to verify proximity and establish communication. - Devices without usable displays cannot show a QR code for a phone to scan. - Proximity discovery alone is insufficient because users still need a clear, secure way to approve the correct authentication request. ## Companion Apps as Secure Message Transport - The target device generates the same hybrid-flow request normally encoded in a QR code. - Instead of displaying it, the device sends the request to an authenticated companion app associated with the same user account. - In-app notifications provide a user-consent surface and direct the user into the passkey flow. - Opening the app itself can initiate the flow because the user must deliberately open it, and mobile operating systems provide an additional verification step. ## Meta Quest and Meta Horizon Implementation - The implementation is broadly available on Meta Quest devices running Meta Horizon OS. - The Quest browser creates a FIDO URL containing: - A fresh ECDH public key. - A session-specific secret. - Routing information for the hybrid handshake. - The URL is packaged as structured data in a GraphQL-based push notification. - The Meta Horizon app validates that the request belongs to the correct signed-in user before processing it. ## Notification and Deep-Link Flow - iOS or Android displays a notification that a passkey login is pending. - Tapping the notification opens the Meta Horizon app, which launches the FIDO URL through the operating system. - The OS then invokes its passkey interface. - If notifications are disabled, opening the app queries the backend for pending requests. - Requests expire after five minutes, limiting the window for misuse. - The mobile device then performs the normal hybrid transport process: - Broadcasting a BLE advertisement. - Establishing an encrypted tunnel. - Generating the passkey assertion. ## WebAuthn Challenge and Response - The inaccessible device creates the normal WebAuthn challenge and waits for a response. - The mobile authenticator initiates the secure BLE or NFC connection. - The challenge travels through the encrypted channel. - After successful user verification, the phone creates an `AuthenticatorAssertionResponse` or `AuthenticatorAttestationResponse`. - The inaccessible device forwards that response to the relying-party server just as a display-equipped device would. ## Broader Impact - The approach removes the QR-code requirement while retaining FIDO hybrid transport and proximity protections. - It could extend passwordless authentication to: - XR headsets and wearables. - Screenless IoT devices. - Smart-home hubs. - Industrial sensors and hardware. - The work builds on FIDO Alliance standards and mobile operating-system support, helping broaden interoperability across device ecosystems. The companion-app model offers a practical way to bring secure cross-device passkeys to devices that cannot display QR codes. It is especially suitable where the device already has an authenticated mobile app and can rely on standard WebAuthn, BLE/NFC proximity, and platform user verification.

Read original(opens in new tab)

GraphQLPasskeysWebauthnIot Security+4

meta·Jan 27, 2026·3 min readCurated summary

Rust at Scale: An Added Layer of Security for WhatsApp

WhatsApp has deployed a Rust-based media security layer across billions of devices to defend against malware hidden in images, videos, PDFs, and other attachments. The system, called Kaleidoscope, validates file formats and identifies suspicious content before it reaches vulnerable downstream libraries. WhatsApp’s large-scale rollout demonstrates Rust’s production readiness and supports the company’s broader shift toward memory-safe languages. ## Media Handling as a Security Boundary - WhatsApp’s default end-to-end encryption protects messages, but shared media can still contain maliciously crafted files. - Attackers may exploit vulnerabilities in: - Operating system libraries - Media parsers - WhatsApp itself - Dangerous attachments can appear harmless, particularly when malware is concealed in images or videos. ## Lessons from the 2015 Stagefright Vulnerability - Android’s Stagefright vulnerability affected operating-system media-processing libraries. - Applications could not directly patch the vulnerable libraries, while users often took months to update their devices. - WhatsApp adapted its existing cross-platform C++ `wamedia` library to identify malformed MP4 files that could trigger vulnerable parsers. - This allowed WhatsApp to protect users faster than relying solely on operating-system updates. - Because the library automatically processes untrusted downloads, WhatsApp identified it as a strong candidate for memory-safe implementation. ## Replacing C++ with Rust - WhatsApp developed the Rust implementation alongside the original C++ version rather than performing a gradual rewrite. - Differential fuzzing, unit tests, and integration tests verified compatibility. - Key challenges included: - Increased binary size from the Rust standard library - Build-system support for WhatsApp’s many target platforms - The final implementation replaced approximately 160,000 lines of C++ with 90,000 lines of Rust, including tests. - Rust provided performance and runtime memory-use improvements. - The library was deployed across Android, iOS, Mac, Web, wearables, and other platforms. ## Kaleidoscope’s File Checks - Kaleidoscope expands beyond basic MP4 validation by checking for: - Non-conforming structures that could exploit parser differences - Embedded files and scripts in PDFs - Files that disguise their type through spoofed extensions or MIME types - Known dangerous formats such as executables and applications - These checks support safer handling in WhatsApp’s user interface and help defend against malicious attachments and unofficial clients. - The system cannot prevent every attack, but it adds an important defense-in-depth layer. ## WhatsApp’s Broader Security Strategy - WhatsApp distributes the libraries each month to billions of phones, computers, watches, and browsers across WhatsApp, Messenger, and Instagram. - The company describes this as the largest deployment of Rust code across diverse end-user platforms. - Its wider security program includes: - End-to-end encrypted messages, calls, and backups - Key transparency and additional calling protections - Fuzzing, static analysis, audits, and attack-surface monitoring - CVE reporting and an expanded bug bounty program - WhatsApp’s vulnerability strategy focuses on minimizing attack surface, strengthening remaining C and C++ code, and choosing memory-safe languages for new development. - Existing protections include control-flow integrity, hardened allocators, safer buffer APIs, specialized developer training, and automated analysis. WhatsApp plans to accelerate Rust adoption, particularly for security-sensitive, cross-platform components that process untrusted input. Its media library rollout provides evidence that Rust can deliver both memory safety and performance at global consumer scale.

Read original(opens in new tab)

RustApplication SecurityCross PlatformMemory Safety+3

meta·Jan 14, 2026Original article

Adapting the Facebook Reels RecSys AI Model Based on User Feedback (opens in new tab)

Meta has enhanced the Facebook Reels recommendation engine by shifting focus from traditional engagement signals, like watch time and likes, to direct user feedback. By implementing the User True Interest Survey (UTIS) model, the system now prioritizes content that aligns with genuine user preferences rather than just short-term interactions. This shift has resulted in significant improvements in recommendation relevance, high-quality content delivery, and long-term user retention. **Limitations of Engagement-Based Metrics** * Traditional signals like "likes" and "watch time" are often noisy and may not reflect a user’s actual long-term interests. * Models optimized solely for engagement tend to favor short-term value over the long-term utility of the product. * Internal research found that previous heuristic-based interest models only achieved 48.3% precision in identifying what users truly care about. * Effective interest matching requires understanding nuanced factors such as production style, mood, audio, and motivation, which implicit signals often miss. **The User True Interest Survey (UTIS) Model** * Meta collects direct feedback via randomized, single-question surveys asking users to rate video interest on a 1–5 scale. * The raw survey data is binarized to denoise responses and weighted to correct for sampling and nonresponse bias. * The UTIS model functions as a lightweight "alignment model layer" built on top of the main multi-task ranking system. * The architecture uses existing model predictions as input features, supplemented by engineered features that capture content attributes and user behavior. **Integration into the Ranking Funnel** * **Late Stage Ranking (LSR):** The UTIS score is used as an additional input feature in the final value formula, allowing the system to boost high-interest videos and demote low-interest ones. * **Early Stage Ranking (Retrieval):** The model aggregates survey data to reconstruct user interest profiles, helping the system source more relevant candidates during the initial retrieval phase. * **Knowledge Distillation:** Large sequence-based retrieval models are aligned using UTIS predictions as labels through distillation objectives. **Performance and Impact** * The deployment of UTIS has led to a measurable increase in the delivery of niche, high-quality content. * Generic, popularity-based recommendations that often lack depth have been reduced. * Meta observed robust improvements across core metrics, including higher follow rates, more shares, and increased user retention. * The system now offers better interpretability, allowing engineers to understand which specific factors contribute to a user’s sense of "interest match." To continue improving the Reels ecosystem, Meta is focusing on doubling down on personalization by tackling challenges related to sparse data and sampling bias while exploring more advanced AI architectures to further diversify recommendations.

AIMachine LearningRecommendation SystemsInformation Retrieval+5

meta·Jan 12, 2026Original article

CSS at Scale With StyleX (opens in new tab)

Scaling CSS within massive codebases presents unique challenges that traditional styling methods often struggle to solve effectively. Meta’s StyleX addresses these issues by offering a system that combines the intuitive ergonomics of CSS-in-JS with the runtime performance of static CSS. By prioritizing atomic styling and definition deduplication, StyleX minimizes bundle sizes and has become the primary styling standard across Meta's entire suite of applications. ### Performance-Driven Styling Architecture * Combines a CSS-in-JS developer experience with a compiler that outputs static CSS to ensure high performance and zero runtime overhead. * Utilizes atomic styling to break down CSS into small, reusable classes, which prevents style sheets from growing linearly with the size of the codebase. * Automatically deduplicates style definitions during the build process, significantly reducing the final bundle size delivered to the client. * Exposes a simple, consistent API that allows developers to manage complex styles and themes while maintaining type safety. ### Standardization and Industry Adoption * Serves as the foundational styling system for Meta’s most prominent platforms, including Facebook, Instagram, WhatsApp, Messenger, and Threads. * Gained significant industry traction beyond Meta, with large-scale organizations such as Figma and Snowflake adopting it for their own web applications. * Acts as an open-source force multiplier, allowing Meta engineers and the broader community to collaborate on solving CSS-at-scale problems. * Provides a mature ecosystem that bridges the gap between the flexibility of JavaScript-based styling and the efficiency of traditional CSS. For engineering teams managing large-scale web applications where bundle size and styling maintainability are critical, StyleX offers a battle-tested framework. Developers can leverage this tool to achieve the performance of static CSS without losing the expressive power of modern JavaScript tooling.

CSSAtomic CssStylexCss In Js+2

meta·Dec 22, 2025Original article

Python Typing Survey 2025: Code Quality and Flexibility As Top Reasons for Typing Adoption (opens in new tab)

The 2025 Typed Python Survey highlights that type hinting has transitioned from an optional feature to a core development standard, with 86% of respondents reporting frequent usage. While mid-career developers show the highest enthusiasm for typing, the ecosystem faces ongoing friction from tooling fragmentation and the complexity of advanced type logic. Overall, the community is pushing for a more robust system that mirrors the expressive power of TypeScript while maintaining Python’s hallmark flexibility. ## Respondent Demographics and Adoption Trends * The survey analyzed responses from 1,241 developers, the majority of whom are highly experienced, with nearly half reporting over a decade of Python expertise. * Adoption is highest among developers with 5–10 years of experience (93%), whereas junior developers (83%) and those with over 10 years of experience (80%) show slightly lower usage rates. * The lower adoption among seniors is attributed to the management of legacy codebases and long-standing habits formed before type hints were introduced to the language. ## Primary Drivers for Typing Adoption * **Incremental Integration:** Developers value the "gradual typing" approach, which allows them to add types to existing projects at their own pace without breaking the codebase. * **Improved Tooling and IDE Support:** Typing significantly enhances developer experience by enabling more accurate autocomplete, jump-to-definition, and inline documentation in IDEs. * **Bug Prevention and Readability:** Type hints act as living documentation that helps catch subtle bugs during refactoring and makes complex codebases easier for teams to reason about. * **Library Compatibility:** Features like Protocols and Generics are highly appreciated, particularly for their synergy with modern libraries like Pydantic and FastAPI that utilize type annotations at runtime. ## Technical Pain Points and Ecosystem Friction * **Third-Party Integration:** A major hurdle is the inconsistent quality or total absence of type stubs in massive libraries like NumPy, Pandas, and Django. * **Tooling Fragmentation:** Developers expressed frustration over inconsistencies between major type checkers like Mypy and Pyright, as well as the slow performance of Mypy in large projects. * **Conceptual Complexity:** Advanced features such as variance (co/contravariance), decorators, and complex nested Generics remain difficult for many developers to implement correctly. * **Runtime Limitations:** Because Python does not enforce types at the interpreter level, some developers find it difficult to justify the verbosity of typing when it offers no native runtime guarantees. ## Most Requested Type System Enhancements * **TypeScript Parity:** There is a strong demand for features found in TypeScript, specifically Intersection types (using the `&` operator), Mapped types, and Conditional types. * **Utility Types:** Developers are looking for built-in utilities like `Pick`, `Omit`, and `keyof` to handle dictionary shapes more effectively. * **Improved Structural Typing:** While `TypedDict` exists, respondents want more flexible, anonymous structural typing to handle complex data structures without excessive boilerplate. * **Performance and Enforcement:** There is a recurring request for an official, high-performance built-in type checker and optional runtime enforcement to bridge the gap between static analysis and execution. As the Python type system continues to mature, developers should prioritize incremental adoption in shared libraries and internal APIs to maximize the benefits of static analysis. While waiting for more advanced features like intersection types, focusing on tooling consistency—such as aligning team standards around a specific type checker—can mitigate much of the friction identified in the 2025 survey.

PythonDeveloper ToolsStatic TypingType Hints+4

meta·Dec 19, 2025Original article

DrP: Meta's Root Cause Analysis Platform at Scale (opens in new tab)

DrP is Meta’s programmatic root cause analysis (RCA) platform designed to automate incident investigations and reduce the burden of manual on-call tasks. By codifying investigation playbooks into executable "analyzers," the platform significantly lowers the mean time to resolve (MTTR) by 20% to 80% for over 300 teams. This systematic approach replaces outdated manual scripts with a scalable backend that executes 50,000 automated analyses daily, providing immediate context when alerts fire. ## Architecture and Core Components * **Expressive SDK:** Provides a framework for engineers to codify investigation workflows into "analyzers," utilizing a rich library of helper functions and machine learning algorithms. * **Built-in Analysis Tools:** The platform includes native support for anomaly detection, event isolation, time-series correlation, and dimension analysis to identify specific problem areas. * **Scalable Backend:** A multi-tenant execution environment manages a worker pool that handles thousands of requests securely and asynchronously. * **Workflow Integration:** DrP is integrated directly into Meta’s internal alerting and incident management systems, allowing for automatic triggering without human intervention. ## Authoring and Verification Workflow * **Template Bootstrapping:** Engineers use the SDK to generate boilerplate code that captures required input parameters and context in a type-safe manner. * **Analyzer Chaining:** The system allows for seamless dependency analysis by passing context between different analyzers, enabling investigations to span multiple interconnected services. * **Automated Backtesting:** Before deployment, analyzers undergo automated backtesting integrated into the code review process to ensure accuracy and performance. * **Decision Tree Logic:** Investigation steps are modeled as decision trees within the code, allowing the analyzer to follow different paths based on the data it retrieves. ## Execution and Post-Processing * **Trigger-based Analysis:** When an alert is activated, the backend automatically queues the relevant analyzer, ensuring findings are available as soon as an engineer begins triaging. * **Automated Mitigation:** A post-processing system can take direct action based on investigation results, such as creating tasks or submitting pull requests to resolve identified issues. * **DrP Insights:** This system periodically reviews historical analysis outputs to identify and rank the top causes of alerts, helping teams prioritize long-term reliability fixes. * **Alert Annotation:** Results are presented in both human-readable text and machine-readable formats, directly annotating the incident logs for the on-call responder. ## Practical Conclusion Organizations managing large-scale distributed systems should transition from static markdown playbooks to executable investigation code. By implementing a programmatic RCA framework like DrP, teams can scale their troubleshooting expertise and significantly reduce "on-call fatigue" by automating the repetitive triage steps that typically consume the first hour of an incident.

AIMachine LearningObservabilityAutomation+4

meta·Dec 17, 2025Original article

How We Built Meta Ray-Ban Display: From Zero to Polish (opens in new tab)

Meta's development of the Ray-Ban Display AI glasses focuses on bridging the gap between sophisticated hardware engineering and intuitive user interfaces. By pairing the glasses with a neural wristband, the team addresses the fundamental challenge of creating a high-performance wearable that remains comfortable and socially acceptable for daily use. The project underscores the necessity of iterative refinement and cross-disciplinary expertise to transition from a technical prototype to a polished consumer product. ### Hardware Engineering and Physics * The design process draws parallels between hardware architecture and particle physics, emphasizing the high-precision requirements of miniaturizing components. * Engineers must manage the strict physical constraints of the Ray-Ban form factor while integrating advanced AI processing and thermal management. * The development culture prioritizes the celebration of incremental technical wins to maintain momentum during the long cycle from "zero to polish." ### Display Technology and UI Evolution * The glasses utilize a unique display system designed to provide visual overlays without obstructing the wearer’s natural field of vision. * The team is developing emerging UI patterns specifically for head-mounted displays, moving away from traditional touch-screen paradigms toward more contextual interactions. * Refining the user experience involves balancing the information density of the display with the need for a non-intrusive, "heads-up" interface. ### The Role of Neural Interfaces * The Ray-Ban Display is packaged with the Meta Neural Band, an electromyography (EMG) wristband that translates motor nerve signals into digital commands. * This wrist-based input mechanism provides a discrete and low-friction way to control the glasses' interface without the need for voice commands or physical buttons. * Integrating EMG technology represents a shift toward human-computer interfaces that are intended to feel like an extension of the user's own body. To successfully build the next generation of wearables, engineering teams should look toward multi-modal input systems—combining visual displays with neural interfaces—to solve the ergonomic and social challenges of hands-free computing.

Human Computer InteractionAugmented RealityWearable TechnologyHardware Design+3

meta·Dec 15, 2025Original article

How AI Is Transforming the Adoption of Secure-by-Default Mobile Frameworks (opens in new tab)

Meta utilizes secure-by-default frameworks to wrap potentially unsafe operating system and third-party functions, ensuring security is integrated into the development process without sacrificing developer velocity. By leveraging generative AI and automation, the company scales the adoption of these frameworks across its massive codebase, effectively mitigating risks such as Android intent hijacking. This approach balances high-level security enforcement with the practical need for friction-free developer experiences. ## Design Principles for Secure-by-Default Frameworks To ensure high adoption and long-term viability, Meta follows specific architectural guidelines when building security wrappers: * **API Mirroring:** Secure framework APIs are designed to closely resemble the existing native APIs they replace (e.g., mirroring the Android Context API). This reduces the cognitive burden on developers and simplifies the use of automated tools for code conversion. * **Reliance on Public Interfaces:** Frameworks are built exclusively on public and stable APIs. Avoiding private or undocumented OS interfaces prevents maintenance "fire drills" and ensures the frameworks remain functional across various OS updates. * **Modularity and Reach:** Rather than creating a single monolithic tool, Meta develops small, modular libraries that target specific security issues while remaining usable across all apps and platform versions. * **Friction Reduction:** Frameworks must avoid introducing excessive complexity or noticeable performance overhead in terms of CPU and RAM, as high friction often leads developers to bypass security measures entirely. ## SecureLinkLauncher: Preventing Android Intent Hijacking SecureLinkLauncher (SLL) is a primary example of a secure-by-default framework designed to stop sensitive data from leaking via the Android intent system. * **Wrapped Execution:** SLL wraps native Android methods such as `startActivity()` and `startActivityForResult()`. Instead of calling `context.startActivity(intent)`, developers use `SecureLinkLauncher.launchInternalActivity(intent, context)`. * **Scope Verification:** The framework enforces scope verification before delegating to the native API. This ensures that intents are directed to intended "family" apps rather than being intercepted by malicious third-party applications. * **Mitigating Implicit Intents:** SLL addresses the risks of untargeted intents, which can be received by any app with a matching intent-filter. By enforcing a developer-specified scope, SLL ensures that data like `SECRET_INFO` is only accessible to authorized packages. ## Scaling Adoption through AI and Automation The transition from legacy, insecure patterns to secure frameworks is managed through a combination of automated tooling and artificial intelligence. * **Automated Migration:** Generative AI identifies insecure usage patterns across Meta’s vast codebase and suggests—or automatically applies—the appropriate secure framework replacements. * **Continuous Monitoring:** Automation tools continuously scan the codebase to ensure compliance with secure-by-default standards, preventing the reintroduction of vulnerable code. * **Scaling Consistency:** By reducing the manual effort required for refactoring, AI enables consistent security enforcement across different teams and applications without slowing down the shipping cycle. For organizations managing large-scale mobile codebases, the recommended approach is to build thin, developer-friendly wrappers around risky platform APIs and utilize automated refactoring tools to drive adoption. This ensures that security becomes an invisible, default component of the development lifecycle rather than a manual checklist.

AIGenerative AIAndroidAutomation+5

meta·Nov 21, 2025Original article

Zoomer: Powering AI Performance at Meta's Scale Through Intelligent Debugging and Optimization (opens in new tab)

Zoomer is Meta’s centralized, automated platform designed to solve performance bottlenecks and GPU underutilization across its massive AI training and inference infrastructure. By integrating deep analytics with scalable data collection, the tool has become the internal standard for optimizing workloads ranging from Llama 3 training to large-scale ads recommendation engines. Ultimately, Zoomer enables significant energy savings and hardware efficiency gains, allowing Meta to accelerate model iteration and increase throughput across its global fleet of GPUs. ### The Three-Layered Architecture * **Infrastructure and Platform Layer:** This foundation utilizes Meta’s Manifold blob storage for trace data and employs fault-tolerant processing pipelines to manage massive trace files across thousands of hosts. * **Analytics and Insights Engine:** This layer performs deep analysis using specialized tools such as Kineto for GPU traces, NVIDIA DCGM for hardware metrics, and StrobeLight for CPU profiling. It automatically detects performance anti-patterns and provides actionable optimization recommendations. * **Visualization and User Interface Layer:** The presentation layer transforms complex data into interactive timelines and heat maps. It integrates with Perfetto for kernel-level inspection and provides drill-down dashboards that highlight outliers across distributed GPU deployments. ### Automated Profiling and Data Capture * **Trigger Mechanisms:** To ensure data accuracy, Zoomer automatically triggers profiling for training workloads during stable states (typically around iteration 550) to avoid startup noise, while inference workloads use on-demand or benchmark-integrated triggers. * **Comprehensive Metrics:** The platform simultaneously collects GPU SM utilization, Tensor Core usage, memory bandwidth, and power consumption via DCGM. * **System-Level Telemetry:** Beyond the GPU, Zoomer captures host-level data including CPU utilization, storage access patterns, and network I/O through dyno telemetry. * **Distributed Communication:** For large-scale training, the tool analyzes NCCL collective operations and inter-node communication patterns to identify stragglers and network bottlenecks. ### Inference and Training Optimization * **Inference Performance:** Zoomer tracks request/response latency, GPU memory allocation patterns, and Thrift request-level profiling to identify bottlenecks in serving user requests at scale. * **Workflow Acceleration:** By correlating application-level annotations—such as forward/backward passes and optimizer steps—with hardware performance, developers can pinpoint exactly which part of a model's execution is inefficient. * **Operational Impact:** These insights have led to significant improvements in Queries Per Second (QPS) for recommendation models and reduced training times for generative AI features by eliminating resource waste. For organizations managing large-scale AI clusters, the Zoomer model suggests that the key to efficiency is moving away from manual, reactive debugging toward an "always-on" automated profiling system. Correlating high-level software phases with low-level hardware telemetry is essential for maximizing the return on investment for expensive GPU resources and maintaining rapid iteration cycles.

Ai InfrastructureDistributed TrainingGpu OptimizationPerformance Profiling+4

meta·Nov 20, 2025Original article

Key Transparency Comes to Messenger (opens in new tab)

Messenger has enhanced the security of its end-to-end encrypted chats by launching key transparency, a system that provides an automated, verifiable record of public encryption keys. By moving beyond manual key comparisons, this feature ensures that users can verify their contacts' identities without technical friction, even when those contacts use multiple devices. This implementation allows Messenger to provide a higher level of assurance that no third party, including Meta, has tampered with or swapped the keys used to secure a conversation. ## The Role of Key Transparency in Encrypted Messaging * Provides a verifiable and auditable record of public keys, ensuring that messages are always encrypted with the correct keys for the intended recipient. * Prevents "man-in-the-middle" attacks by a compromised server by making any unauthorized key changes visible to the system. * Simplifies the user experience by automating the verification process, which previously required users to manually compare long strings of characters across every device their contact owned. ## Architecture and Third-Party Auditing * Built upon the open-source Auditable Key Directory (AKD) library, which was previously used to implement similar security properties for WhatsApp. * Partners with Cloudflare to act as a third-party auditor, maintaining a public Key Transparency Dashboard that allows anyone to verify the integrity of the directory. * Leverages an "epoch" system where the directory is updated and published frequently to ensure that the global log of keys remains current and immutable. ## Scaling for Global Messenger Traffic * Manages a massive database that has already grown to billions of entries, reflecting the high volume of users and the fact that Messenger indexes keys for every individual device a user logs into. * Operates at a high frequency, publishing a new epoch approximately every two minutes, with each update containing hundreds of thousands of new key entries. * Optimized the algorithmic efficiency of the AKD library to ensure that cryptographic proof sizes remain small and manageable, even as the number of updates for a single key grows over time. ## Infrastructure Resilience and Recovery * Improved the system's ability to handle temporary outages and long delays in key sequencing, drawing on two years of operational data from the WhatsApp implementation. * Replaced older proof methods that grew linearly with the height of the transparency tree with more efficient operations to maintain high availability and real-time verification speeds. * Established a robust recovery process to ensure that the transparency log remains consistent even after infrastructure disruptions. By automating the verification of encryption keys through a transparent, audited directory, Messenger has made sophisticated cryptographic security accessible to billions of users. This rollout represents a significant shift in how trust is managed in digital communications, replacing manual user checks with a seamless, background-level guarantee of privacy.

CloudflareSecurityEnd To End EncryptionCryptography+2

meta·Nov 18, 2025Original article

Efficient Optimization With Ax, an Open Platform for Adaptive Experimentation (opens in new tab)

Meta has released Ax 1.0, an open-source platform designed to automate and optimize complex, resource-intensive experimentation through machine learning. By utilizing Bayesian optimization, the platform helps researchers navigate vast configuration spaces to improve AI models, infrastructure, and hardware design efficiently. The release aims to bridge the gap between sophisticated mathematical theory and the practical requirements of production-scale engineering. ## Real-World Experimentation and Utility * Ax is used extensively at Meta for diverse tasks, including tuning hyperparameter configurations, discovering optimal data mixtures for Generative AI, and optimizing compiler flags. * The platform is built to handle the logistical "overhead" of experimentation, such as managing experiment states, automating orchestration, and providing diagnostic tools. * It supports multi-objective optimization, allowing users to balance competing metrics and enforce "guardrail" constraints rather than just maximizing a single value. * Applications extend beyond software to physical engineering, such as optimizing design parameters for AR/VR hardware. ## System Insight and Analysis * Beyond finding optimal points, Ax serves as a diagnostic tool to help researchers understand the underlying behavior of their systems. * It includes built-in visualizations for Pareto frontiers, which illustrate the trade-offs between different metrics. * Sensitivity analysis tools identify which specific input parameters have the greatest impact on the final results. * The platform provides automated plots and tables to track optimization progress and visualize the effect of parameters across the entire input space. ## Technical Methodology and Architecture * Ax utilizes Bayesian optimization, an iterative approach that balances "exploration" (sampling new areas) with "exploitation" (refining known good areas). * The platform relies on **BoTorch** for its underlying Bayesian components and typically employs **Gaussian processes (GP)** as surrogate models. * GPs are preferred because they can make accurate predictions and quantify uncertainty even when provided with very few data points. * The system uses an **Expected Improvement (EI)** acquisition function to calculate the potential value of new configurations compared to the current best-known result. * This surrogate-based approach is designed to scale to high-dimensional settings involving hundreds of tunable parameters where traditional search methods are too costly. To begin implementing these methods, developers can install the platform via `pip install ax-platform`. Ax 1.0 provides a robust framework for moving cutting-edge optimization research directly into production environments.

AIMachine LearningBayesian OptimizationAdaptive Experimentation+5