meta·Jul 1, 2026·3 min readCurated summary

Meta’s AI Storage Blueprint at Scale

Meta argues that AI progress increasingly depends on storage that can deliver massive datasets with predictable, low latency. Traditional BLOB-storage designs optimized for durable, cost-efficient HDD storage create metadata and proxying bottlenecks that stall GPUs and slow research. Meta is therefore rebuilding its storage foundation around unified metadata, direct client-to-storage access, and regional deployments colocated with GPUs. ## Storage Architecture and AI’s Growing Demands - Meta operates hundreds of exabyte-scale storage clusters supporting products such as Facebook, Instagram, Meta AI, Ads, and internal databases. - Its storage APIs are built on Tectonic, a horizontally scalable block layer providing: - High durability and availability through erasure coding - HDD and flash tiering - Placement of hot, warm, and cold data - Multi-tenant regional storage - BLOB-storage layers built on Tectonic provide globally scalable object storage and configurable durability/availability policies. - Meta’s training systems historically used an NFS-like filesystem interface over Tectonic, but are increasingly moving to BLOB storage for unified access to massive data lakes and higher performance. ## Why Storage Latency Limits GPU Utilization - AI workloads require bursty and sustained high throughput with predictable worst-case latency. - Training runs use hundreds of thousands of GPUs processing data in batches and periodically synchronizing state. - A single slow GPU can delay synchronization and extend the completion time for every GPU. - Data loaders prefetch future batches while GPUs process current ones, but high-latency storage reads can still create GPU stalls. - These stalls directly increase training costs and extend time to market. ## Problems with the Legacy BLOB Architecture - The older service-oriented design accumulated multiple stateful layers, each with its own metadata store. - A single `getObject("/bucket/path")` request could require lookups across the namelayer, volumeslayer, and containerlayer. - Cross-region metadata requests could add hundreds of milliseconds, and one slow lookup could delay the entire operation. - The architecture’s original assumptions no longer matched AI requirements: - **Latency:** AI needs bounded pMax latency, not merely acceptable average performance. - **Reliability:** AI requires high availability, but does not always need global replication by default. - **Cost:** Flash is necessary for AI-level IOPS, making storage cost-per-byte less important. - **Power:** Power used by storage competes directly with power available for GPUs. ## Rebuilding the Storage Foundation Meta redesigned the system around three major changes: - **Unified metadata schema** - Metadata from separate layers was consolidated into a flat schema backed by ZippyDB. - Path resolution can now use O(1) lookups to map objects to `(blockId, offset, size)` locations. - **Direct data access** - The dataplane proxy was removed. - A “fat client” SDK streams data directly from Tectonic storage servers. - This reduces latency, increases throughput, and lowers storage power consumption. - **Regional deployment** - The BLOB stack can operate regionally or globally. - Regional instances are colocated with GPUs in AI regions, reducing cross-region access. With the new flow, the SDK requests a read plan from the API server, which performs the metadata lookup and returns storage locations. The SDK’s embedded Tectonic BlockClient then reads directly from the underlying blocks, adding essentially no extra dataplane overhead. The redesigned architecture is intended to improve GPU utilization, reduce latency, and preserve power for computation. The provided excerpt ends as Meta begins discussing how it handles workload spikes and hot spots during data and checkpoint loading.

Read original(opens in new tab)

Distributed StorageAi TrainingObject StorageErasure Coding+3

airbnb·Sep 24, 2025·3 min readCurated summary

Building a Next-Generation Key-Value Store at Airbnb

Airbnb rebuilt Mussel, its key-value store for derived data, from a complex EC2-based system into a cloud-native NewSQL platform. Mussel v2 combines bulk ingestion, streaming writes, low-latency reads, flexible consistency, and automated operations while supporting more than 100 existing use cases. A gradual, reversible blue/green migration moved production workloads without data loss or customer-visible downtime. ## Why Airbnb Rebuilt Mussel - New use cases—including real-time fraud detection, personalization, and dynamic pricing—required both streaming updates and large-scale bulk ingestion. - Mussel v1 had become difficult to operate and scale: - Node changes required multi-step Chef scripts on EC2. - Static hash partitioning created hotspots and latency spikes. - Consistency options were limited. - Resource consumption and costs were difficult to track. - Mussel v2 provides Kubernetes-based automation, dynamic range sharding, configurable consistency, namespace tenancy, quotas, and usage dashboards. ## Mussel v2 Architecture ### Stateless Dispatcher - A horizontally scalable Kubernetes service translates client requests into backend queries and mutations. - It supports: - Dual writes and shadow reads during migration - Retries, rate limiting, and dynamic throttling - Service-mesh security and discovery - Point lookups, range queries, prefix queries, and low-latency stale reads - Each dataname maps to a logical table, simplifying access patterns. ### Kafka-Based Write Pipeline - Writes are first persisted to Kafka for durability. - The Replayer and Write Dispatcher apply them to the backend in order. - Kafka absorbs traffic bursts and supports consistency, migrations, bootstrapping, and upgrades. - Airbnb plans to eventually rely more directly on the distributed database for ingestion and replication to reduce latency and operational complexity. ### Bulk Loading - Mussel retains support for both: - **Merge** jobs, which add data to existing tables - **Replace** jobs, which swap in a new dataset - Existing Airflow onboarding workflows transform warehouse data into a standard format and upload it to S3. - A stateless controller coordinates ingestion, while Kubernetes StatefulSet workers load data in parallel. - Deduplication, delta merges, and insert-on-duplicate-key-ignore improve throughput and reduce unnecessary writes. ## Scalable Data Expiration - Mussel v1 depended on storage-engine compaction for TTL expiration, which became inefficient at scale. - V2 uses a topology-aware expiration service: - Namespaces are divided into range-based subtasks. - Multiple workers scan and delete expired records concurrently. - Scheduling limits interference with live queries. - Max-version enforcement and targeted deletes help manage write-heavy tables. - The result is faster, more visible, and more scalable retention management. ## Blue/Green Migration - The migration had to handle massive datasets, thousands of tables, and mission-critical traffic with zero data loss and no availability impact. - Because v1 lacked table-level snapshots and CDC, Airbnb built a custom migration pipeline. - Tables were selected and migrated individually according to usage and risk. ### Migration Stages - **Blue:** All production traffic continued serving from v1. - **Shadowing:** Bootstrapped v2 tables processed parallel reads and writes, but v1 still served responses. - **Reverse:** V2 served live traffic while v1 remained available as a fallback. - **Cutover:** After validation, traffic was permanently moved to v2 one dataname at a time. - Automatic circuit breakers and fallback logic enabled rapid rollback if v2 showed errors or replication lag. - Kafka’s replication stream maintained eventual consistency between the two systems throughout the transition. ## Practical Takeaway Mussel v2 demonstrates that large datastore rearchitectures can be made safe through incremental migration, durable event logs, shadow traffic, and reversible per-table cutovers. The key recommendation is to combine a more scalable backend with strong operational automation and migration tooling, rather than attempting a single disruptive replacement.

Read original(opens in new tab)

Distributed StorageKubernetesApache KafkaDatabase Migration+4