metaJul 1, 2026

Meta’s AI Storage Blueprint at Scale (opens in new tab)

Meta argues that AI progress increasingly depends on storage that can deliver massive datasets with predictable, low latency. Traditional BLOB-storage designs optimized for durable, cost-efficient HDD storage create metadata and proxying bottlenecks that stall GPUs and slow research. Meta is therefore rebuilding its storage foundation around unified metadata, direct client-to-storage access, and regional deployments colocated with GPUs. ## Storage Architecture and AI’s Growing Demands - Meta operates hundreds of exabyte-scale storage clusters supporting products such as Facebook, Instagram, Meta AI, Ads, and internal databases. - Its storage APIs are built on Tectonic, a horizontally scalable block layer providing: - High durability and availability through erasure coding - HDD and flash tiering - Placement of hot, warm, and cold data - Multi-tenant regional storage - BLOB-storage layers built on Tectonic provide globally scalable object storage and configurable durability/availability policies. - Meta’s training systems historically used an NFS-like filesystem interface over Tectonic, but are increasingly moving to BLOB storage for unified access to massive data lakes and higher performance. ## Why Storage Latency Limits GPU Utilization - AI workloads require bursty and sustained high throughput with predictable worst-case latency. - Training runs use hundreds of thousands of GPUs processing data in batches and periodically synchronizing state. - A single slow GPU can delay synchronization and extend the completion time for every GPU. - Data loaders prefetch future batches while GPUs process current ones, but high-latency storage reads can still create GPU stalls. - These stalls directly increase training costs and extend time to market. ## Problems with the Legacy BLOB Architecture - The older service-oriented design accumulated multiple stateful layers, each with its own metadata store. - A single `getObject("/bucket/path")` request could require lookups across the namelayer, volumeslayer, and containerlayer. - Cross-region metadata requests could add hundreds of milliseconds, and one slow lookup could delay the entire operation. - The architecture’s original assumptions no longer matched AI requirements: - **Latency:** AI needs bounded pMax latency, not merely acceptable average performance. - **Reliability:** AI requires high availability, but does not always need global replication by default. - **Cost:** Flash is necessary for AI-level IOPS, making storage cost-per-byte less important. - **Power:** Power used by storage competes directly with power available for GPUs. ## Rebuilding the Storage Foundation Meta redesigned the system around three major changes: - **Unified metadata schema** - Metadata from separate layers was consolidated into a flat schema backed by ZippyDB. - Path resolution can now use O(1) lookups to map objects to `(blockId, offset, size)` locations. - **Direct data access** - The dataplane proxy was removed. - A “fat client” SDK streams data directly from Tectonic storage servers. - This reduces latency, increases throughput, and lowers storage power consumption. - **Regional deployment** - The BLOB stack can operate regionally or globally. - Regional instances are colocated with GPUs in AI regions, reducing cross-region access. With the new flow, the SDK requests a read plan from the API server, which performs the metadata lookup and returns storage locations. The SDK’s embedded Tectonic BlockClient then reads directly from the underlying blocks, adding essentially no extra dataplane overhead. The redesigned architecture is intended to improve GPU utilization, reduce latency, and preserve power for computation. The provided excerpt ends as Meta begins discussing how it handles workload spikes and hot spots during data and checkpoint loading.

erasure-codingai-trainingobject-storagedistributed-storage+3

dropboxApr 2, 2026

Improving storage efficiency in Magic Pocket, our immutable blob store (opens in new tab)

Magic Pocket’s immutable design protects data integrity but makes storage efficiency dependent on continuous reclamation. A new Live Coder service reduced write amplification while unintentionally creating severely under-filled volumes, driving fragmentation and storage overhead sharply upward. Dropbox responded by rethinking compaction, since its existing steady-state strategy was too slow to recover space from the resulting long tail of sparse volumes. ## The Cost of Immutability - Magic Pocket stores user files as immutable blobs distributed across its storage fleet. - Updates and deletions never modify data in place; obsolete blobs remain until compaction. - Garbage collection identifies unreferenced blobs, while compaction physically moves live blobs into new volumes and retires old ones. - Because closed volumes cannot be reopened, deleted data creates unused space unless it is actively consolidated. - Durability also increases storage requirements: - Replication stores multiple complete copies. - Erasure coding splits data into fragments and adds parity, providing fault tolerance with less overhead. - Fragmentation determines how efficiently that redundant capacity is used: - A volume with 50% live data effectively doubles required storage. - A volume with 10% live data uses roughly ten times the necessary space. ## The Live Coder Incident - A new on-the-fly erasure-coding service created severely under-filled volumes as it rolled out to new regions. - In the worst cases, less than 5% of a volume’s capacity contained live data. - Since volumes have fixed allocations, many mostly empty volumes consumed nearly as much raw capacity as full volumes. - Dropbox detected rising effective replication-factor signals, indicating more raw storage was being used per live byte. - The existing compaction system continued reclaiming space but was not designed for a long tail of extremely sparse volumes. - The incident demonstrated that compaction must adapt when the distribution of live data changes substantially. ## Steady-State L1 Compaction - Dropbox’s baseline strategy, L1, treats compaction as a packing problem. - It selects: - A highly filled host volume with available space. - Donor volumes whose live data fits into that space. - Live blobs from the donors are written into a new volume, eventually leaving the donors empty and removable. - L1 is simple, fast, and limits placement risk and metadata changes. - However, each run can read tens of GiB while typically producing only one densely packed volume. - Fewer than one complete volume is reclaimed on average because only donor volumes are fully drained. - This works well when volumes are already near full, but performs poorly when storage overhead is concentrated in many severely under-filled volumes.

erasure-codingdatabase-designgarbage-collectionmagic-pocket+4