When failover isn’t safe: Building high-availability PostgreSQL on Kubernetes (opens in new tab)
Datadog’s gameday testing exposed a PostgreSQL failure mode in which network latency caused replication lag to grow until no standby could be safely promoted. Although the clusters remained writable, they could not fail over without risking data loss, forcing operators to wait for connectivity and replicas to recover. Datadog’s solution was to redesign failover candidates around synchronous replication coordinated by Patroni, balancing stronger durability with acceptable write latency. ## The Zonal Failure That Exposed the Weakness - A simulated availability-zone failure introduced network latency in a staging environment. - Several Kubernetes-based PostgreSQL clusters had primary nodes in the affected zone. - Communication between primaries and replicas degraded, causing: - Rapidly increasing replication lag - Stalled writes - Applications serving stale data - No replica being current enough for safe promotion - The clusters prioritized continued writes over durability, leaving them writable but unable to fail over safely. ## Baseline PostgreSQL Architecture - Each cluster uses a single-writer design: - One active leader handles writes. - Two standby nodes are reserved for failover and do not serve application traffic. - A separate read-replica pool handles read-only traffic and scales independently. - Read replicas are intentionally excluded from failover candidates. - Patroni manages replication, leader elections, and failover. - ZooKeeper acts as Patroni’s distributed configuration store, tracking: - The current leader lock - Cluster configuration - Member replication state and latest LSN - ZooKeeper’s ephemeral leader key ensures that only one node can become primary. - During partitions, Patroni favors safety by pausing or demoting nodes that cannot verify cluster state. ## Why Failover Was Not Safe - Patroni checks replication lag before promoting a standby using `maximum_lag_on_failover`. - During the gameday, all eligible standbys exceeded that threshold. - Patroni correctly rejected promotion because each candidate could have been missing committed transactions. - The cluster therefore had no safe writable primary, even though the original leader was impaired. - The failure was a consequence of asynchronous replication and network latency, not a failure in Patroni’s safety mechanisms. ## Asynchronous Versus Synchronous Replication - **Asynchronous replication**, used originally: - Lets the leader commit and respond without waiting for replicas. - Provides low write latency and high throughput. - Can lose transactions committed on the leader but not yet copied to a standby. - **Synchronous replication**: - Requires the leader to receive acknowledgment from at least one replica before confirming a transaction. - Reduces the chance that a failover candidate is significantly behind. - Provides stronger durability, but may increase write latency when replicas experience network or availability problems. ## The Redesigned Approach - Datadog reworked its PostgreSQL deployment so failover candidates use synchronous replication. - Patroni coordinates these replicas and continues to enforce safe leader election. - The design aims to make failover both automatic and safe while limiting performance impact. - Benchmarking and failure testing were used to evaluate the trade-off between durability and latency. Datadog’s experience demonstrates that asynchronous replication can leave a system operational but unable to fail over during network disruption. For clusters where data durability and automatic recovery are critical, synchronous replication for designated failover candidates offers a safer architecture, provided its latency and availability costs are measured carefully.