Breaking up a monolith: How we’re unwinding a shared database at scale
Datadog is moving away from a large shared relational database because its benefits eventually give way to coordination costs, schema fragility, noisy-neighbor problems, and scaling limits. Splitting the database is difficult and expensive, but platform investments in service development and managed Postgres can make independently owned databases practical. The key is to establish functional boundaries, provide safe cross-domain access, and automate migrations. ## Why Shared Databases Persist - Shared databases reduce operational overhead for small or fast-moving organizations. - A single database enables simple, low-latency joins across all data. - Workload isolation and access management often matter less when systems are small. - Because the cost of splitting a database is high, organizations commonly keep the shared model longer than they should. ## Signs It Is Time to Split the Database - Data grows beyond the capacity of one machine, or replication becomes too slow. - Noisy-neighbor effects make performance unpredictable. - Schema changes by one team unexpectedly affect others. - Security requirements such as access-control lists are difficult to enforce. - These issues create engineering costs, incidents, and degraded user experiences across teams. ## What Database Decomposition Requires - Identify functional ownership boundaries. - Build services for cross-domain queries where necessary. - Require consumers to use those services instead of querying another domain’s tables directly. - Provision new database instances. - Migrate data and traffic carefully from the shared database to the new instances. Datadog had previously split off large portions of its database into only a few separate databases. The experience showed that finding boundaries, enforcing them, and migrating without incidents is difficult and highly manual. ## Why Teams Resist Leaving Shared Infrastructure - Building a service may jeopardize existing product goals. - Operating a service can introduce significant maintenance and on-call work. - Cross-domain data access may be unclear or cause unacceptable latency or user impact. - Owning a database creates additional operational responsibility. - Migrations are often handcrafted, risky, and difficult to repeat. - Forcing the transition can cost more than tolerating the existing problems and create organizational resistance. ## Platform Investments That Enable Change Datadog addressed these obstacles through two major initiatives: - **Rapid:** An opinionated framework for building and operating API and gRPC services. - **OrgStore:** A managed platform for Postgres databases. Rapid reduces the cost of creating and maintaining services by providing shared configuration, common data-access patterns, and operational support. OrgStore reduces the burden of owning separate database instances. Together, these platforms make it more attractive for new projects to avoid the legacy shared database and allow existing domains to migrate incrementally. The broader lesson is that database decomposition becomes realistic when platform engineering makes service ownership, database operations, cross-domain access, and migrations safe enough to fit into normal product development.
Read original(opens in new tab)