pgbouncer

2 posts

figma

PGKeeper: Building the Bouncer We Needed for Postgres | Figma Blog (opens in new tab)

Figma built PGKeeper to replace PgBouncer as its PostgreSQL connection and load-management layer. Growing traffic, sharding, and stricter reliability requirements exposed PgBouncer’s limits in scalability, prioritization, backpressure, connection protection, and extensibility. PGKeeper is a custom Go service positioned between Figma’s DBProxy routing layer and PostgreSQL, designed to protect databases from overload and connection churn. ## Figma’s Database Architecture - PostgreSQL powers Figma’s OLTP workloads. - Figma scales through horizontal and vertical sharding across multiple database instances. - DBProxy hides sharding complexity from application code by: - Parsing and analyzing queries. - Selecting the appropriate PostgreSQL instances. - Rewriting requests into queries for the selected targets. - A dedicated set of connection-pooler replicas serves each PostgreSQL machine, creating an n-to-one relationship between poolers and databases. ## Why PgBouncer Was No Longer Enough - **Limited scalability** - PgBouncer’s single-threaded architecture created a vertical scaling ceiling. - Adding replicas helped, but uneven load distribution caused performance degradation. - **Insufficient load management** - PgBouncer could not prioritize critical traffic over lower-priority or misbehaving requests. - It lacked effective backpressure and advanced load-shedding algorithms such as Controlled Delay (CoDel). - CoDel sheds work based on how long requests wait, rather than simply counting queued requests. - **Unsafe connection behavior** - PostgreSQL connections are expensive resources. - Rapid connection creation and churn could destabilize database nodes. - Recovery after overload could trigger another surge of connections, creating cascading failures and prolonged overload. - **Limited extensibility and control** - Figma needed deep observability, feature-flagged rollouts, admission control, and fair resource sharing. - Even maintaining small PgBouncer patches proved costly. - Extending PgBouncer substantially would create an ongoing maintenance burden. ## Why Connection Pooling Could Not Live in DBProxy - Figma generally limits each PostgreSQL instance to roughly 100 pooled connections. - Hundreds of stateless DBProxy replicas sit in front of those databases. - Giving every DBProxy replica its own pool would either exceed database connection limits or require complex coordination. - Centralizing pooling in a separate service provided a better fit for the mismatch between many routers and a small fixed connection budget. ## Why Figma Built PGKeeper - PGCat addressed PgBouncer’s single-threaded scalability problem, but customizing it would require deep changes to its core execution paths. - Those changes would likely require Figma to maintain a long-term fork. - Figma therefore created PGKeeper as a Go-based service tailored to its infrastructure and operational requirements. - Its role is to act like a goalkeeper: protecting PostgreSQL from harmful traffic and protecting connections from uncontrolled churn. PGKeeper was chosen because Figma needed more than a basic connection pooler: it needed a scalable, observable, controllable layer capable of prioritizing traffic and preventing database overload.

figma

The growing pains of database architecture | Figma Blog (opens in new tab)

Figma outgrew its single Amazon RDS PostgreSQL database as traffic increased roughly threefold annually, pushing peak CPU utilization above 65% and making latency unpredictable. Initial fixes—larger hardware, read replicas, new databases, and PgBouncer—provided temporary relief but could not adequately reduce write load or handle replication-sensitive reads. Figma ultimately chose vertical partitioning, moving groups of related tables into separate databases as a lower-risk, incremental path to scalability. ## The Limits of a Single Database - Figma stored metadata such as permissions, file information, and comments in one large RDS instance. - Increasing users, new features, and preparation for a second product drove database traffic sharply upward. - Peak CPU utilization reached more than 65%, with latency becoming less predictable as the database approached its limits. - Full saturation would have made Figma unavailable, so the infrastructure team addressed the risk before it became an outage. ## Tactical Measures for More Headroom Figma introduced several short-term improvements: - Upgraded the database from an `r5.12xlarge` to an `r5.24xlarge` instance. - Added multiple read replicas to distribute read traffic. - Created separate databases for new use cases to prevent further growth of the original database. - Added PgBouncer to pool connections and reduce the impact of thousands of application connections. - These changes provided approximately another year of runway, but writes still consumed substantial resources. - Some reads could not be moved to replicas because the application was sensitive to replication lag. ## Evaluating Horizontal Scaling Figma considered horizontally sharding the database but found substantial technical and operational risks: - Many managed horizontally scalable databases were not natively compatible with PostgreSQL. - Migrating to NoSQL or Vitess would require complex double-read and double-write migration strategies. - NoSQL would also require significant application changes. - A managed distributed PostgreSQL system could make Figma an unusually large customer, exposing it to untested scaling limits. - Self-hosting would require new expertise, training, and considerable operational investment, diverting attention from the core scalability problem. ## Choosing Vertical Partitioning Instead of splitting individual tables across many database nodes, Figma chose vertical partitioning: - Groups of related tables would be moved to separate databases. - This approach immediately reduced load on the original database. - It preserved a future path toward horizontal sharding for particularly large or demanding table groups. - The strategy was considered more incremental and operationally manageable than replacing PostgreSQL or adopting a self-hosted distributed system. ## Selecting Tables to Move Figma evaluated candidate tables using two criteria: - **Impact:** Moving the tables should remove a meaningful portion of the database workload. - **Isolation:** The tables should have limited dependency on tables that remained in the original database. - To measure impact, the team analyzed average active sessions (AAS), which estimates the average number of active threads handling a query. - They gathered query activity from PostgreSQL’s `pg_stat_activity` view at 10-millisecond intervals to identify CPU waits associated with individual queries. Figma’s experience shows that database scaling does not always require an immediate move to distributed infrastructure. Carefully selected vertical partitioning can reduce pressure on a primary database while limiting migration risk and preserving more ambitious scaling options for the future.