Replication redefined: How we built a low-latency, multi-tenant data replication platform

Datadog built a managed, multi-tenant data replication platform to move data reliably across thousands of services without brittle, point-to-point integrations. The effort began by separating analytical search workloads from a shared PostgreSQL database, then evolved into automated pipeline provisioning with Temporal. The platform favors asynchronous replication to improve scalability and resilience, accepting limited replication lag in exchange for lower application latency and reduced operational coupling.

Scaling Search Beyond PostgreSQL

• A shared PostgreSQL database initially provided low-latency access, ACID guarantees, and low operational cost.
• As data volumes grew, complex joins and aggregations became increasingly slow.
• Datadog’s Metrics Summary page had to join:
  • 82,000 active metrics
  • 817,000 metric configurations
• Page latency reached approximately 7 seconds at p90, while repeated facet changes generated additional expensive queries.
• Index and disk bloat, memory pressure, VACUUM and ANALYZE overhead, and rising I/O wait further reduced throughput.
• Rather than continuing to optimize PostgreSQL for analytical search, Datadog moved search and aggregation workloads to a dedicated search platform.
• Data was denormalized during replication, producing document-oriented indexes better suited to faceted search.
• The resulting system reduced page-load times by as much as 97%—from roughly 30 seconds to 1 second—while maintaining about 500 ms of replication lag.

Automating Pipeline Provisioning with Temporal

Provisioning a replication pipeline required coordinating multiple systems and configuration steps:

• Enabling PostgreSQL logical replication with wal_level.
• Creating users and assigning replication permissions.
• Configuring publishers and replication slots.
• Deploying Debezium instances to capture PostgreSQL changes.
• Creating Kafka topics and mapping them to Debezium instances.
• Adding heartbeat tables to monitor replication and prevent excessive WAL retention.
• Configuring sink connectors to write Kafka data into the search platform.

Manual management became increasingly difficult across many pipelines and data centers. Datadog used Temporal workflows to split provisioning into modular, repeatable tasks and combine them into higher-level orchestrations. This reduced errors, improved consistency, and allowed engineers to create and modify pipelines without repeating complex operational procedures.

Choosing Asynchronous Replication

• Synchronous replication provides strong consistency by waiting for replicas to acknowledge each write.
• However, it increases latency and operational complexity, particularly across distributed environments.
• Asynchronous replication allows the primary system to acknowledge writes immediately while replicas catch up afterward.
• Datadog selected the asynchronous model because it decouples application performance from network latency and replica availability.
• The trade-off is temporary replication lag during failures or periods of pressure, but the model offers better scalability and resilience for high-throughput systems.

Datadog’s experience suggests that replication should be treated as a managed platform rather than a collection of custom integrations. Separating workloads, automating provisioning, and choosing asynchronous delivery can improve performance and reliability while reducing the operational burden on individual engineering teams.