Data Migration | Techlist.io

metaMay 12, 20263 min readCurated summary

Migrating Data Ingestion Systems at Meta Scale

Meta rebuilt its hyperscale MySQL data ingestion system to improve reliability, efficiency, and data-langing latency. The migration moved workloads from customer-owned pipelines to a simpler, self-managed warehouse service and ultimately transitioned 100% of jobs. Success depended on staged validation, continuous data comparison, and fast rollback mechanisms. ## Why Meta Migrated - The system incrementally moved several petabytes of social graph data from MySQL into Meta’s data warehouse each day. - This data supports analytics, reporting, machine learning, and product development. - The legacy architecture became increasingly unstable as data-landing requirements grew stricter. - Customer-owned pipelines worked at smaller scales but became difficult to manage reliably at hyperscale. ## Migration Success Criteria Each job had to meet defined requirements before advancing: - **Data correctness:** Old and new systems had matching row counts and checksums. - **Landing latency:** The new system performed at least as well as the legacy system. - **Resource usage:** Compute and storage consumption did not regress. - **Critical-table requirements:** Additional criteria were agreed upon with dependent teams. ## Three-Phase Migration Lifecycle ### Shadow Phase - New-system shadow jobs ran against the same production sources as existing jobs. - Their output was written to separate shadow tables. - Row counts and checksums were continuously compared with production data. - Compute and storage requirements were measured before production rollout. - Once validated in pre-production, shadow jobs were tested in production. ### Reverse Shadow Phase - The new system began writing to the production table. - The legacy system continued running, but wrote to a shadow table. - This preserved continuous comparison between both systems. - If discrepancies appeared, Meta could quickly restore the old system without rebuilding its configuration. ### Migration Cleanup - Both systems continued to be monitored for mismatches. - After validation, the legacy shadow job was removed. - The new system became the sole production pipeline. ## Data Quality and Debugging Tooling - Meta built tooling to compare corresponding table partitions from the two systems. - Comparisons included row counts, checksums, and example rows responsible for mismatches. - Mismatch records and debugging details were logged to Scuba for real-time analysis. - Hourly queries helped engineers identify root causes and determine whether issues were already known. - The same tooling remains part of post-migration release validation. ## Rollout and Rollback Controls - Both systems used change data capture (CDC), with internal full-dump and delta tables feeding customer-facing target tables. - Because CDC builds new data from previously landed data, an existing defect could propagate after migration. - Meta therefore emphasized: - Detecting problems before they reached data consumers. - Stopping further propagation quickly during rollback. - The reverse-shadow design provided early quality signals and preserved a ready-to-use legacy pipeline for rapid recovery. Meta’s migration demonstrates that large-scale infrastructure changes are safest when treated as controlled, observable lifecycle transitions rather than one-time cutovers. Parallel execution, automated data validation, explicit resource checks, and reversible rollouts enabled the company to migrate the entire workload while protecting downstream consumers.

Read original

Data Migration Machine Learning Mysql Data Ingestion+3

lineMar 17, 20263 min readCurated summary

Integration of LINE App’s Multi-party Chat Features

LINE is consolidating its two multi-person chat types—temporary “Rooms” and long-term “Groups”—into a single Group Chat model. The change aims to simplify the user experience, make all chat features available everywhere, and reduce duplicated server and client resources. A gradual migration strategy is being used to avoid disruption. ## Two Original Chat Models - **Rooms** were designed for temporary conversations: - No room name was required. - Invited friends joined immediately without approval. - Features such as albums and notes were unavailable. - **Groups** were designed for long-term communities: - They had names and supported features such as group albums and notes. - Invitees had to accept or reject invitations before joining. - Users often created Rooms without realizing their limitations, then created a new Group later when they needed additional features. ## Reasons for Unification - Users found the distinction between Rooms and Groups difficult to understand. - Existing conversations could not be converted from Rooms into Groups, forcing users to abandon their conversation history. - Users frequently created multiple chats with the same members, causing: - Cluttered conversation lists. - Unnecessary data accumulation on servers. - Increased client and server resource usage. - The unified model standardizes behavior and features while retaining flexibility in how invitations work. ## Migrating Groups to Group Chats - LINE introduced new Group Chat APIs and used **dual reads** to maintain compatibility with existing Group APIs and storage. - The migration proceeded gradually: 1. The new API initially read Group data through a routing layer. 2. The number of Group Chats was progressively increased. 3. Eventually, only Group Chats were created. - Batch processing migrated all existing Group data. - After migration, LINE stopped dual reads and relied exclusively on the Group Chat model. ## Differences Between Rooms and Groups ### Invitation Mechanisms - Groups required invitees to explicitly accept or reject an invitation. - Rooms added people immediately when they were invited. - The unified creation flow lets users choose whether invitees should join immediately or confirm participation first. ### Feature Availability - Rooms lacked many Group features because they were intended to be temporary. - The new model is based on the Group architecture, so all newly created conversations support the full feature set, including future features. ## Improving Conversation Discovery - Users often created a new chat with the same participants instead of finding an older, inactive conversation in a long chat list. - The new creation workflow displays a hint when an equivalent existing conversation is found. - Users can then return to the existing conversation, reducing duplicate rooms and improving navigation. ## Migration Plans for Existing Rooms - Conversations created in current LINE versions are already Group Chats. - Groups created with older app versions are being converted server-side. - The remaining objective is to migrate existing Rooms so their participants can use the complete set of Group Chat features. The project is a long-term effort designed to minimize disruption while improving consistency and efficiency. Duplicate conversations with identical participants fell from 15% for Rooms to 0.78% for invitation-free Group Chats, demonstrating the practical impact of the consolidation.

Read original

Data Migration User Experience API Design Real Time Communication+2

tossNov 30, 2025Original article

From Legacy Payment Ledger to Scalable System (opens in new tab)

Toss Payments successfully modernized a 20-year-old legacy payment ledger by transitioning to a decoupled, MySQL-based architecture designed for high scalability and consistency. By implementing strategies like INSERT-only immutability and event-driven domain isolation, they overcame structural limitations such as the inability to handle split payments. Ultimately, the project demonstrates that robust system design must be paired with resilient operational recovery mechanisms to manage the complexities of large-scale financial migrations. ### Legacy Ledger Challenges * **Inconsistent Schemas:** Different payment methods used entirely different table structures; for instance, a table named `REFUND` unexpectedly contained only account transfer data rather than all refund types. * **Domain Coupling:** Multiple domains (settlement, accounting, and payments) shared the same tables and columns, meaning a single schema change required impact analysis across several teams. * **Structural Limits:** A rigid 1:1 relationship between a payment and its method prevented the implementation of modern features like split payments or "Dutch pay" models. ### New Ledger Architecture * **Data Immutability:** The system shifted from updating existing rows to an **INSERT-only** principle, ensuring a reliable audit trail and preventing database deadlocks. * **Event-Driven Decoupling:** Instead of direct database access, the system uses Kafka to publish payment events, allowing independent domains to consume data without tight coupling. * **Payment-Approval Separation:** By separating the "Payment" (the transaction intent) from the "Approval" (the specific financial method), the system now supports multiple payment methods per transaction. ### Safe Migration and Data Integrity * **Asynchronous Mirroring:** To maintain zero downtime, data was initially written to the legacy system and then asynchronously loaded into the new MySQL ledger. * **Resource Tuning:** Developers used dedicated migration servers within the same AWS Availability Zone to minimize latency and implemented **Bulk Inserts** to handle hundreds of millions of rows efficiently. * **Verification Batches:** A separate batch process ran every five minutes against a Read-Only (RO) database to identify and correct any data gaps caused by asynchronous processing failures. ### Operational Resilience and Incident Response * **Query Optimization:** During a load spike, the MySQL optimizer chose "Full Scans" over indexes; the team resolved this by implementing SQL hints and utilizing a 5-version Docker image history for rapid rollbacks. * **Network Cancellation:** To handle timeouts between Toss and external card issuers, the system uses specific logic to automatically send cancellation requests and synchronize states. * **Timeout Standardization:** Discrepancies between microservices were resolved by calculating the maximum processing time of approval servers and aligning all upstream timeout settings to prevent merchant response mismatches. * **Reliable Event Delivery:** While using the **Outbox pattern** for events, the team added log-based recovery (Elasticsearch and local disk) and idempotency keys in event headers to handle both missing and duplicate messages. For organizations tackling significant technical debt, this transition highlights that initial design is only half the battle. True system reliability comes from building "self-healing" structures—such as automated correction batches and standardized timeout chains—that can survive the unpredictable nature of live production environments.

Data Migration Database Design Apache Kafka Microservices+4

figmaAug 5, 20203 min readCurated summary

Inside Figma: reflections on a remote internship | Figma Blog

Jenning Chen reflects on her remote internship at Figma and the process of taking improvements to the style picker from concept to launch. Despite concerns about isolation and communication, Figma’s intentional remote culture helped her build relationships, learn the product, and receive support. Her project added search, a color-style list view, and visible text-style metrics while giving her experience across Figma’s technology stack. ## A Warm Welcome - Slack messages, virtual coffee chats, and a collaborative welcome card helped Chen feel connected from the start. - Although the internship was fully remote, Figma maintained the open and welcoming culture that had first attracted her years earlier. ## Building Relationships Remotely - Chen initially worried that remote work would make it harder to form relationships or ask for help. - Company-wide events created regular opportunities to meet colleagues: - Show-and-tell meetings for sharing work and learnings - Thursday tech talks about coworkers’ interests and projects - One-on-ones about Figma’s product and growth - Virtual cooking classes, escape rooms, and scavenger hunts - These activities helped replace informal office interactions and encouraged connections beyond her immediate team. ## Developing Product Fluency - Before beginning her project, Chen needed to understand Figma’s features and design terminology. - She relied on her mentor, teammates, and Slack channels for answers. - Colleagues were willing to schedule calls and work through obstacles with her, making remote collaboration more effective than she expected. ## Improving the Style Picker - The style picker lets users browse and apply paint, text, effect, and layout-grid styles. - As users accumulated more styles, the original grid-based interface became difficult to navigate: - Important style information was obscured. - Users had to scroll through long lists. - Finding a specific style was inefficient. - Working with product designer Shana Hu, Chen implemented: - **Search:** Lets users find styles with a few keystrokes. - **Color styles list view:** Displays style names clearly beside thumbnails. - **Text style metrics:** Shows font size and line height directly in the picker. ## Technical and Launch Challenges - The project exposed Chen to multiple parts of Figma’s stack: - TypeScript and C++ in the editor - Ruby in the backend - She presented the work at an internal show-and-tell and received supportive feedback from coworkers. - Launch preparation required migrating millions of existing text styles to add font-size and line-height metadata. - The migration took an entire day, alongside the work of incorporating critique feedback and fixing last-minute bugs. The internship demonstrated that a thoughtfully structured remote environment can support mentorship, relationship-building, and meaningful ownership. Chen’s experience also shows how an intern can contribute to a high-impact product feature while gaining broad technical and product knowledge.

Read original

Data Migration Figma Design Systems TypeScript+2