Database Migration

8 posts

line4 min readCurated summary

Flava DBaaS Deep Dive: From Architecture to Migration and Beyond

LY Corporation’s Flava DBaaS is designed to unify the former Verda and YNW cloud platforms on a Kubernetes-based architecture. Its operator pattern separates database business logic from IaaS management, while API servers, managers, and agents divide responsibilities within each DBMS service. The platform expands database support, improves scalability, security, and usability, and treats migration from legacy platforms as part of the DBaaS responsibility. ## Kubernetes Operator-Based Design - Flava DBaaS uses the Kubernetes operator pattern. - Users declare the desired database state through custom resources rather than issuing procedural commands. - Controllers continuously reconcile the actual state with the declared specification. - This approach: - Simplifies troubleshooting through resource status and controller logs. - Handles large database infrastructures efficiently through event-driven processing. - Reuses Kubernetes capabilities for CI/CD and access control. ## Infrastructure Operator Layer - DBaaS must manage IaaS resources such as: - Virtual machines - Storage - Domains and networking - Flava isolates this infrastructure logic in a separate infrastructure operator. - IaaS resources are exposed as Kubernetes custom resources, allowing DBaaS to create infrastructure declaratively without directly calling IaaS APIs. - The resulting layers are: - **DBaaS:** Database-specific business logic - **Infrastructure operator:** Abstraction of IaaS as Kubernetes resources - **IaaS:** Compute, network, and storage services - This separation allows multiple DBMS products to use infrastructure consistently while their developers focus on database operations. ## Custom Resources and DBaaS Components - Each database cluster is represented by a Kubernetes custom resource containing settings such as: - DBMS version - VM size - Storage type and capacity - Replication configuration - These resources are stored in Kubernetes etcd and managed through the Kubernetes API. - Each DBMS implementation consists of three components: - **API server:** Provides REST APIs for creating, modifying, and deleting database resources. Flava UI and IaC tools use these APIs. - **Manager:** Watches resource changes and reconciles the database cluster toward the declared state. - **Agent:** Runs on database VMs and executes local operating-system and database commands. - For example, creating a MySQL cluster causes the API server to create a MySQL custom resource, the manager to provision the required VMs through the infrastructure operator, and the agent to configure replication and database processes inside those VMs. ## Improvements in Flava DBaaS - Flava preserves core DBaaS capabilities such as provisioning, high availability, backup and recovery, scalability, and monitoring. - It combines the DBMS offerings of Verda and YNW, expanding the range of supported database systems. ### Flexible Storage and Scaling - Storage can be configured in 100 GiB increments. - Block-storage-based databases can use up to 5 TiB of storage. - Unlike the legacy platforms, storage is no longer tightly limited by a VM’s local disk capacity. - Custom instance types and separate block storage reduce the need to consider alternatives such as sharding for larger databases. - The 5 TiB limit was selected to cover most analyzed use cases while reducing infrastructure fragmentation. ### Consistent User Experience - All Flava DBaaS products share a common architecture and UI. - Skills learned while changing MySQL server specifications or configuring Cassandra alerts can be applied to other DBMS products. - Users do not need to learn separate operational workflows for each database system. ### Security and Convenience - TDE and TLS are provided as platform-level security features. - Additional features include: - **Custom DB Role:** Reusable database users with configurable permissions. - **Database Parameter Group:** Reusable groups of database configuration parameters. - **Restore backup:** Creation of a new cluster from a selected backup for disaster recovery or realistic performance testing. - Features not yet available for every DBaaS product are planned for broader support. - These improvements reportedly resulted in high internal user-satisfaction scores. ## Migration Responsibilities - A new DBaaS platform is expected to provide migration paths from existing platforms, not merely offer new database clusters. - For migrations between the same DBMS type, the article identifies three general approaches. ### Dump and Restore - Data is backed up from the source database and restored into the destination. - It is the simplest method. - To guarantee consistency, the application generally must be stopped during the migration. ### Replication-Based Migration - The source database is continuously replicated to the destination. - Once replication is caught up, the destination is promoted through failover. - The source database can then be removed. - Data consistency depends on the DBMS’s replication mechanism. - A short application interruption may still occur during primary-node failover. The overall recommendation is to use Flava’s layered, declarative architecture to standardize database operations while continuing to provide practical migration mechanisms from Verda and YNW.

Read original(opens in new tab)
datadog3 min readCurated summary

How we migrated a live routing system using AI-assisted refactoring

Stream Router evolved from a small configuration file into a critical control-plane service routing Datadog’s massive metrics workload. Its original FoundationDB key-value model eventually hit transaction-size and performance limits because relational relationships were reconstructed in application code. Datadog redesigned the system around PostgreSQL and DuckDB, using AI-assisted, test-driven refactoring to accelerate the migration without disrupting production traffic. ## Stream Router’s Role in Datadog’s Metrics Pipeline - Datadog processes more than a hundred trillion events per day. - Stream Router determines which Kafka cluster, topic, partitions, and sharding strategy should handle each datapoint. - It serves both producers and queriers but does not process Kafka messages itself. - Routing decisions change frequently as infrastructure evolves, making correctness and historical tracking essential. ## From Configuration File to Control Plane - In 2016, routing was managed through a small configuration file distributed to services. - As the platform grew, the file expanded to thousands of lines and required manual edits and rollouts. - Stream Router replaced this workflow with: - A centralized gRPC service - API-managed routes - Automated, gradual rollouts - The write path used FoundationDB, while the read path served static RocksDB snapshots restored into memory. - This eventually became a bottleneck as routing tables and operational changes grew larger. ## Why the Key-Value Model Stopped Scaling - Routes reference streams and sharding strategies, while rules reference routes. - These relationships are inherently relational and require cross-entity validation. - The KV implementation loaded tens of thousands of records into application processes and reconstructed database-like relationships in code. - Some operations exceeded FoundationDB transaction-size limits. - Moving to PostgreSQL without changing the access patterns would not solve the issue; certain operations were estimated to require 45 minutes because of thousands of sequential database round trips. - The fundamental problem was the data model and application logic, not simply the choice of database. ## Designing the New Storage Architecture - The team redesigned the schema manually before using AI tools. - The relational model introduced explicit foreign keys between: - Streams - Sharding strategies - Routes - Rules - PostgreSQL was selected for the write path because it provided the required relational semantics and transaction model. - DuckDB was selected for the read path because: - It is embeddable and suitable for snapshot-based serving - It supports array columns - Its SQL dialect is closely compatible with PostgreSQL - Shared query logic could therefore work across both storage engines. ## AI-Assisted Refactoring - Claude and Cursor were used to accelerate a systematic, test-driven migration. - For each method, developers supplied: - The old implementation - The new schema - A failing test - AI generated an initial implementation, while tests determined whether it was correct. - The models assisted with method-level refactoring rather than autonomously designing the architecture. - Human expertise remained central to schema design, migration strategy, and evaluating system-level risks. ## Foundations for a Safe Migration - The migration benefited from infrastructure already present at Datadog. - Stream Router’s storage layer was isolated behind an internal `Controller` interface. - This modularity helped contain storage changes and enabled incremental refactoring. - Existing tests and clear boundaries provided confidence in generated implementations while production traffic continued. The central lesson is that AI was most effective as an accelerator inside a disciplined engineering process. A well-designed relational schema, modular storage abstraction, and failing tests provided the safety mechanisms; AI helped implement the resulting changes faster, but did not replace human architectural judgment.

Read original(opens in new tab)
slack2 min readCurated summary

How Slack Rebuilt Notifications 📣

Slack rebuilt its notification system to reduce noise by replacing years of inconsistent, tightly coupled behavior with a unified model. The redesign separates what activity users see from how they receive interruptions, while aligning desktop and mobile settings. By combining backend migration strategies, auto-saving controls, and shared UI patterns, Slack aims to make notifications predictable and easier to manage. ## Diagnosing Notification Overload - Notification frustration is common, especially for users in many channels. - Notification issues are among Slack’s top three sources of Customer Experience tickets. - The underlying problem was architectural as well as behavioral: - Desktop and mobile used conflicting preference systems. - Equivalent settings, such as “Nothing” and “Off,” behaved differently across clients. - Activity preferences were coupled to push delivery. - Settings could fall out of sync between desktop and mobile. - Advanced options were scattered or difficult to discover. ## A Unified Notification Model Slack introduced a simpler set of controls: - Channel notifications now offer: - **All new posts** - **Mentions** - **Mute** - Push notifications have separate on/off controls across desktop and mobile. - Advanced features, including mobile “badge all unreads,” are easier to find. - Global preferences use consistent structure and language. - Simplified preference logic improves synchronization between clients. ## Refactoring Preferences Safely - Slack migrated users from four conflicting preference systems to a unified model. - The new model separates: - Desktop activity: **Everything** or **Mentions** - Desktop push: `desktop_push_enabled` set to `true` or `false` - Mobile activity and push behavior: **Everything**, **Mentions**, or **Nothing** - Rather than changing millions of database records directly, Slack used read-time interpretation to preserve backward compatibility and allow rollback. - Existing “Off” settings now behave as “Mentions” with push disabled. - A backfill populated the new desktop push preference based on users’ previous settings. - This preserves in-app awareness while allowing push interruptions to be controlled independently. ## Auto-Saving and Clearer Controls - The previous modal required users to press **Save**, which caused accidental abandoned changes. - The redesigned interface applies changes immediately through auto-save. - Users can independently choose what activity to see and how they want to receive it. - Shared React components replaced legacy mobile-specific UI code, improving consistency across platforms. - Users can now, for example, view all activity while receiving push notifications only for mentions. Slack’s approach demonstrates that reducing notification noise requires more than a visual redesign. Separating activity from delivery, simplifying preference states, and keeping clients synchronized gives users clearer and more reliable control over interruptions.

Read original(opens in new tab)
netflix3 min readCurated summary

Automating RDS Postgres to Aurora Postgres Migration

Netflix standardized on Amazon Aurora PostgreSQL after finding that PostgreSQL already supported most relational workloads and that Aurora offered stronger scalability, availability, elasticity, and ecosystem alignment. To migrate nearly 400 RDS PostgreSQL clusters efficiently, Netflix built a self-service workflow that automates replication, traffic quiescence, validation, and cutover while minimizing downtime and eliminating data loss. The Aurora read-replica method is preferred over snapshot migration because it keeps the target nearly synchronized while production continues running. ## Why Netflix Chose Aurora PostgreSQL - PostgreSQL already supported the majority of Netflix’s relational workloads. - Internal evaluations found Aurora PostgreSQL could support more than 95% of workloads running on other relational database systems. - PostgreSQL benefits from: - A broad open-source ecosystem - Strong community adoption - Compatibility with modern data platforms - Aurora’s distributed, cloud-native architecture provides: - Better scalability and elasticity - High availability - Support for globally distributed applications - The migration effort began with RDS PostgreSQL and is intended to expand to other relational systems. ## Database Migration Requires More Than Data Copying A safe migration must move both data and database functionality while preserving correctness, availability, and performance. - **Data replication:** Copy existing data and continuously apply source changes to the destination. - **Quiescence:** Stop writes to the source so the destination can catch up completely. - **Validation:** Confirm that source and destination data are synchronized. - **Cutover:** Redirect applications to the new Aurora database as the system of record. ## Operational and Technical Challenges - Manually migrating almost 400 PostgreSQL clusters would be slow, error-prone, and operationally expensive. - Coordinating downtime across dependent services is difficult. - Netflix therefore created a self-service workflow that handles orchestration, safety checks, and correctness guarantees automatically. - The system must guarantee: - Zero data loss - Extremely short downtime, especially for critical services - No performance degradation during or after migration - Migration of related resources such as parameter groups, read replicas, and replication slots - Application teams control database clients, so the platform cannot depend on them manually pausing writes. - The migration system must provide control-plane mechanisms to halt traffic safely during validation and cutover. - The workflow must operate without obtaining RDS credentials from users, since databases may be tightly secured and the migration platform may lack direct database access. - Because non-experts operate the process, the experience must be self-guided and require minimal user effort. ## Snapshot-Based Migration The snapshot approach is straightforward but requires stopping writes before migration. - Halt write traffic to the RDS PostgreSQL source. - Create a manual snapshot. - Convert the snapshot into an Aurora-compatible format. - Create an Aurora PostgreSQL cluster from the converted snapshot. - Validate the new cluster. - Redirect applications to the Aurora endpoint. This method is simple but can involve a longer interruption because the target is not continuously updated while the snapshot is created and converted. ## Aurora Read-Replica Migration The read-replica approach reduces downtime by continuously replicating the RDS database into Aurora. - Create an Aurora PostgreSQL read replica from the RDS source. - Stream changes asynchronously from RDS to Aurora while applications continue using the source. - Provision and validate Aurora configuration, connectivity, and performance in advance. - When replication lag is sufficiently low, briefly pause writes. - Allow the replica to catch up fully. - Promote it to a standalone Aurora PostgreSQL cluster. - Redirect application traffic to the Aurora endpoint. This approach keeps the destination nearly synchronized before cutover, making it substantially less disruptive than snapshot-based migration. Netflix’s automation focuses on making the read-replica migration process safe, repeatable, and self-service, with the platform handling replication, traffic control, validation, and cutover rather than relying on manual application-team coordination.

Read original(opens in new tab)
naverOriginal article

Smart Store Center's Zero- (opens in new tab)

Smart Store Center successfully migrated its legacy platform from Oracle to MySQL to overcome performance instability caused by resource contention and to reduce high licensing costs. By implementing a "dual write" strategy, the team achieved a zero-downtime transition while maintaining the ability to roll back immediately without data loss. This technical journey highlights the use of proxy data sources and transaction synchronization to ensure data integrity across disparate database environments. ## Zero-Downtime Migration via Dual Writing * The migration strategy relied on "dual writing," where all Create, Update, and Delete (CUD) operations are performed on both the legacy Oracle and the new MySQL databases. * In the pre-migration phase, Oracle served as the primary source for all traffic while MySQL recorded writes in the background to build a synchronized state. * Once data was fully migrated and verified, the primary traffic was shifted to MySQL, with background writes continuing to Oracle to allow for an instantaneous rollback if performance issues occurred. * This approach decoupled the database switch from application deployment, providing a safety net against critical failures that a simple redeploy could not fix. ## Technical Implementation for JPA * To capture and replicate queries, the team utilized the `datasource-proxy` library, which allowed them to intercept Oracle queries and execute them against a separate MySQL DataSource. * To prevent MySQL write failures from impacting the primary Oracle transactions, writes to the secondary database were managed using `TransactionSynchronizationManager`. * By executing MySQL queries during the `afterCommit` phase, the team ensured that the primary service remained stable even if the secondary database encountered errors or performance bottlenecks. * The transition required modifying JPA Entity configurations, such as changing primary key generation from Oracle Sequences to MySQL’s `IDENTITY` (auto-increment) and adjusting `columnDefinition` for types like `text`, `longtext`, and `decimal`. ## Centralized MyBatis Strategy * To avoid modifying thousands of business logic points in a 10-year-old codebase, the team sought a way to implement dual writing for MyBatis at the architectural level. * The implementation focused on the MyBatis `Configuration` and `MappedStatement` objects to capture SQL execution without requiring manual updates to individual repository interfaces. * This centralized approach maintained the purity of the business logic and ensured that the dual-write logic could be easily removed once the migration was fully stabilized. For organizations managing large-scale legacy migrations, the dual-write pattern combined with asynchronous transaction synchronization is a highly recommended safety mechanism. Prioritizing the isolation of secondary database failures ensures that the user experience remains unaffected while technical validation is performed in real-time.

airbnb3 min readCurated summary

Building a Next-Generation Key-Value Store at Airbnb

Airbnb rebuilt Mussel, its key-value store for derived data, from a complex EC2-based system into a cloud-native NewSQL platform. Mussel v2 combines bulk ingestion, streaming writes, low-latency reads, flexible consistency, and automated operations while supporting more than 100 existing use cases. A gradual, reversible blue/green migration moved production workloads without data loss or customer-visible downtime. ## Why Airbnb Rebuilt Mussel - New use cases—including real-time fraud detection, personalization, and dynamic pricing—required both streaming updates and large-scale bulk ingestion. - Mussel v1 had become difficult to operate and scale: - Node changes required multi-step Chef scripts on EC2. - Static hash partitioning created hotspots and latency spikes. - Consistency options were limited. - Resource consumption and costs were difficult to track. - Mussel v2 provides Kubernetes-based automation, dynamic range sharding, configurable consistency, namespace tenancy, quotas, and usage dashboards. ## Mussel v2 Architecture ### Stateless Dispatcher - A horizontally scalable Kubernetes service translates client requests into backend queries and mutations. - It supports: - Dual writes and shadow reads during migration - Retries, rate limiting, and dynamic throttling - Service-mesh security and discovery - Point lookups, range queries, prefix queries, and low-latency stale reads - Each dataname maps to a logical table, simplifying access patterns. ### Kafka-Based Write Pipeline - Writes are first persisted to Kafka for durability. - The Replayer and Write Dispatcher apply them to the backend in order. - Kafka absorbs traffic bursts and supports consistency, migrations, bootstrapping, and upgrades. - Airbnb plans to eventually rely more directly on the distributed database for ingestion and replication to reduce latency and operational complexity. ### Bulk Loading - Mussel retains support for both: - **Merge** jobs, which add data to existing tables - **Replace** jobs, which swap in a new dataset - Existing Airflow onboarding workflows transform warehouse data into a standard format and upload it to S3. - A stateless controller coordinates ingestion, while Kubernetes StatefulSet workers load data in parallel. - Deduplication, delta merges, and insert-on-duplicate-key-ignore improve throughput and reduce unnecessary writes. ## Scalable Data Expiration - Mussel v1 depended on storage-engine compaction for TTL expiration, which became inefficient at scale. - V2 uses a topology-aware expiration service: - Namespaces are divided into range-based subtasks. - Multiple workers scan and delete expired records concurrently. - Scheduling limits interference with live queries. - Max-version enforcement and targeted deletes help manage write-heavy tables. - The result is faster, more visible, and more scalable retention management. ## Blue/Green Migration - The migration had to handle massive datasets, thousands of tables, and mission-critical traffic with zero data loss and no availability impact. - Because v1 lacked table-level snapshots and CDC, Airbnb built a custom migration pipeline. - Tables were selected and migrated individually according to usage and risk. ### Migration Stages - **Blue:** All production traffic continued serving from v1. - **Shadowing:** Bootstrapped v2 tables processed parallel reads and writes, but v1 still served responses. - **Reverse:** V2 served live traffic while v1 remained available as a fallback. - **Cutover:** After validation, traffic was permanently moved to v2 one dataname at a time. - Automatic circuit breakers and fallback logic enabled rapid rollback if v2 showed errors or replication lag. - Kafka’s replication stream maintained eventual consistency between the two systems throughout the transition. ## Practical Takeaway Mussel v2 demonstrates that large datastore rearchitectures can be made safe through incremental migration, durable event logs, shadow traffic, and reversible per-table cutovers. The key recommendation is to combine a more scalable backend with strong operational automation and migration tooling, rather than attempting a single disruptive replacement.

Read original(opens in new tab)
lineOriginal article

Replacing the Payment System DB Handling (opens in new tab)

The LINE Billing Platform successfully migrated its large-scale payment database from Nbase-T to Vitess to handle high-traffic global transactions. While initially exploring gRPC for its performance reputation, the team transitioned to the MySQL protocol to ensure stability and reduce CPU overhead within their Java-based environment. This implementation demonstrates how Vitess can manage complex sharding requirements while maintaining high availability through automated recovery tools. ### Protocol Selection and Implementation - The team initially attempted to use the gRPC protocol but encountered `http2: frame too large` errors and significant CPU overhead during performance testing. - Manual mapping of query results to Java objects proved cumbersome with the Vitess gRPC client, leading to a shift toward the more mature and recommended MySQL protocol. - Using the MySQL protocol allowed the team to leverage standard database drivers while benefiting from Vitess's routing capabilities via VTGate. ### Keyspace Architecture and Data Routing - The system utilizes a dual-keyspace strategy: a "Global Keyspace" for unsharded metadata and a "Service Keyspace" for sharded transaction data. - The Global Keyspace manages sharding keys using a "sequence" table type to ensure unique, auto-incrementing identifiers across the platform. - The Service Keyspace is partitioned into $N$ shards using a hash-based Vindex, which distributes coin balances and transaction history. - VTGate automatically routes queries to the correct shard by analyzing the sharding key in the `WHERE` clause or `INSERT` statement, minimizing cross-shard overhead. ### MySQL Compatibility and Transaction Logic - Vitess maintains `REPEATABLE READ` isolation for single-shard transactions, while multi-shard transactions default to `READ COMMITTED`. - Advanced features like Two-Phase Commit (2PC) are available for handling distributed transactions across multiple shards. - Query execution plans are analyzed using `VEXPLAIN` and `VTEXPLAIN`, often managed through the VTAdmin web interface for better visibility. - Certain limitations apply, such as temporary tables only being supported in unsharded keyspaces and specific unsupported SQL cases documented in the Vitess core. ### Automated Operations and Monitoring - The team employs VTOrc (based on Orchestrator) to automatically detect and repair database failures, such as unreachable primaries or replication stops. - Monitoring is centralized via Prometheus, which scrapes metrics from VTOrc, VTGate, and VTTablet components at dedicated ports (e.g., 16000). - Real-time alerts are routed through Slack and email, using `tablet_alias` to specifically identify which MySQL node or VTTablet is experiencing issues. - A web-based recovery dashboard provides a history of automated fixes, allowing operators to track the health of the cluster over time. For organizations migrating high-traffic legacy systems to a cloud-native sharding solution, prioritizing the MySQL protocol over gRPC is recommended for better compatibility with existing application frameworks and reduced operational complexity.

lineOriginal article

Replacing a Payment System Database That Processes (opens in new tab)

The LINE Billing Platform team recently migrated its core payment database from Nbase-T to Vitess to address rising licensing costs while maintaining the high availability required for financial transactions. After a rigorous Proof of Concept (PoC) evaluating Apache ShardingSphere, TiDB, and Vitess, the team selected Vitess for its mature sharding capabilities and its ability to provide a stable, scalable environment on bare-metal infrastructure. This migration ensures the platform can handle large-scale traffic efficiently without the financial burden of proprietary license fees. ### Evaluation of Alternative Sharding Solutions Before settling on Vitess, the team analyzed other prominent distributed database technologies to determine their fit for a high-stakes payment system: * **Apache ShardingSphere:** While it offers flexible Proxy and JDBC layers, it was excluded because it requires significant manual effort for data resharding and rebalancing. The management overhead for implementing shard-key logic across various components (API, batch, admin) was deemed too high. * **TiDB:** This MySQL-compatible distributed database uses a decoupled architecture consisting of TiDB (SQL layer), PD (metadata management), and TiKV (row-based storage). Its primary advantage is automatic rebalancing and the lack of a required shard key, which significantly reduces DBA operational costs. * **Nbase-T:** The legacy system provided the highest performance efficiency per resource unit; however, the shift from a free to a paid licensing model necessitated the move to an open-source alternative. ### Vitess Architecture and Core Components Vitess was chosen for its proven track record at companies like YouTube and GitHub, offering a robust abstraction layer that makes a clustered database appear as a single instance to the application. The system relies on several specialized components: * **VTGate:** A proxy server that routes queries to the correct VTTablet, manages distributed transactions, and hides the physical topology of the database from the application. * **VTTablet:** A sidecar process running alongside each MySQL instance that manages query execution, data replication, and connection pooling. * **VTorc and Topology Server:** High availability is managed by VTorc (an automated failover tool), while metadata regarding shard locations and node status is synchronized via a topology server using ZooKeeper or etcd. ### PoC Performance and Environment Setup The team conducted performance testing by simulating real payment API scenarios (a mix of reads and writes) on standardized hardware (8vCPU, 16GB RAM). * **Comparison Metrics:** The tests focused on Transactions Per Second (TPS) and resource utilization as thread counts increased. * **Infrastructure Strategy:** Because payment systems cannot tolerate even brief failover delays, the team opted for a bare-metal deployment rather than a containerized one to ensure maximum stability and performance. * **Resource Efficiency:** While Nbase-T showed the best raw efficiency, Vitess demonstrated the necessary scalability and management features required to replace the legacy system effectively within the new cost constraints. ### Practical Recommendation For organizations managing critical core systems that require horizontal scaling without proprietary lock-in, Vitess is a highly recommended solution. While it requires a deep understanding of its various components (like VTGate and VTTablet) and careful configuration of its topology server, the trade-off is a mature, cloud-native-ready architecture that supports massive scale and automated failover on both bare-metal and cloud environments.