Migrating Data Ingestion Systems at Meta Scale
Meta rebuilt its hyperscale MySQL data ingestion system to improve reliability, efficiency, and data-langing latency. The migration moved workloads from customer-owned pipelines to a simpler, self-managed warehouse service and ultimately transitioned 100% of jobs. Success depended on staged validation, continuous data comparison, and fast rollback mechanisms. ## Why Meta Migrated - The system incrementally moved several petabytes of social graph data from MySQL into Meta’s data warehouse each day. - This data supports analytics, reporting, machine learning, and product development. - The legacy architecture became increasingly unstable as data-landing requirements grew stricter. - Customer-owned pipelines worked at smaller scales but became difficult to manage reliably at hyperscale. ## Migration Success Criteria Each job had to meet defined requirements before advancing: - **Data correctness:** Old and new systems had matching row counts and checksums. - **Landing latency:** The new system performed at least as well as the legacy system. - **Resource usage:** Compute and storage consumption did not regress. - **Critical-table requirements:** Additional criteria were agreed upon with dependent teams. ## Three-Phase Migration Lifecycle ### Shadow Phase - New-system shadow jobs ran against the same production sources as existing jobs. - Their output was written to separate shadow tables. - Row counts and checksums were continuously compared with production data. - Compute and storage requirements were measured before production rollout. - Once validated in pre-production, shadow jobs were tested in production. ### Reverse Shadow Phase - The new system began writing to the production table. - The legacy system continued running, but wrote to a shadow table. - This preserved continuous comparison between both systems. - If discrepancies appeared, Meta could quickly restore the old system without rebuilding its configuration. ### Migration Cleanup - Both systems continued to be monitored for mismatches. - After validation, the legacy shadow job was removed. - The new system became the sole production pipeline. ## Data Quality and Debugging Tooling - Meta built tooling to compare corresponding table partitions from the two systems. - Comparisons included row counts, checksums, and example rows responsible for mismatches. - Mismatch records and debugging details were logged to Scuba for real-time analysis. - Hourly queries helped engineers identify root causes and determine whether issues were already known. - The same tooling remains part of post-migration release validation. ## Rollout and Rollback Controls - Both systems used change data capture (CDC), with internal full-dump and delta tables feeding customer-facing target tables. - Because CDC builds new data from previously landed data, an existing defect could propagate after migration. - Meta therefore emphasized: - Detecting problems before they reached data consumers. - Stopping further propagation quickly during rollback. - The reverse-shadow design provided early quality signals and preserved a ready-to-use legacy pipeline for rapid recovery. Meta’s migration demonstrates that large-scale infrastructure changes are safest when treated as controlled, observable lifecycle transitions rather than one-time cutovers. Parallel execution, automated data validation, explicit resource checks, and reversible rollouts enabled the company to migrate the entire workload while protecting downstream consumers.
Read original(opens in new tab)
