From Multi-Day Latency to Near Real-Time Insights: Figma’s Data Pipeline Upgrade | Figma Blog
Figma replaced a daily full-table export system that could take hours or days with an incremental synchronization pipeline designed for near real-time analytics. The new architecture combines database snapshots, change data capture (CDC), and Snowflake merge logic to transfer only recent changes. By building the system in-house, Figma gained greater flexibility, lower projected costs, and a design that can scale with continued growth. ## Why the Legacy Pipeline Failed - Since 2020, a daily cron job ran `SELECT * FROM <TABLE>`, exported results to S3, and loaded them into Snowflake. - As Figma’s tables and insert volume grew: - Daily syncs reached roughly six hours by 2023. - The largest tables took several days or longer. - Additional database replicas were required for exports. - Replica maintenance cost millions of dollars annually. - The delays limited access to timely company KPIs and analytical insights. ## Choosing Incremental Synchronization Figma evaluated three options: - Continue using the legacy process, which was increasingly expensive and too slow. - Add parallelism, which might improve throughput temporarily but would not scale sustainably. - Rebuild the synchronization system around incremental updates. Incremental synchronization transfers only new and changed records instead of repeatedly copying entire tables, reducing data movement, processing time, and infrastructure usage. ## Buy vs. Build Figma decided to build the pipeline internally because available proprietary tools did not meet its requirements. - **Flexibility:** Generic SQL tools did not take advantage of capabilities such as Amazon RDS for PostgreSQL snapshot exports. - **Cost:** Commercial solutions were projected to cost five to ten times more than an in-house implementation. - **Scale:** Building internally allowed Figma to optimize the system for its infrastructure and adapt it as the company grows. ## Pipeline Components The bespoke system combines several lower-level technologies: - **Snapshots:** Amazon RDS exports initial table copies to S3. - **Change data capture:** Kafka Connect streams database changes through Amazon MSK. - **Warehouse ingestion:** A Snowflake Connector loads CDC events into Snowflake. - **Incremental merging:** Custom Snowflake stored procedures and scheduled tasks merge changes into base tables. ## Architecture Principles The redesign was guided by four goals: - Reduce end-to-end synchronization latency. - Control costs as data volume increases. - Meet regulatory and compliance requirements. - Preserve data accuracy, completeness, consistency, and trustworthiness. The resulting architecture uses two workflows: a bootstrap workflow for onboarding tables and a validation workflow for checking data correctness. ## Bootstrap Workflow The automated onboarding process includes: - The CDC service begins capturing the new Postgres table and publishes events to a per-table Kafka topic. - Amazon RDS exports the latest database snapshot to S3. - Snowflake’s `COPY INTO <table>` loads the snapshot into a per-entity base table. - An MSK Connect Snowflake Sink Connector streams Kafka events into a separate CDC table, with offsets arranged so changes before the snapshot timestamp are retained. - A scheduled Snowflake task runs a custom `MERGE` procedure to combine the snapshot and CDC data. - Once the process catches up with current changes, Figma creates a lightweight user-facing view over the base table. ## Zero-Downtime Re-Bootstrapping - Bootstrap artifacts are versioned, while the final user-facing view remains stable. - New versions can be built in parallel without interrupting queries. - Promotion is completed through an atomic view update. - This supports schema evolution and other situations requiring a fresh bootstrap without downtime. ## Data Validation - Even well-designed pipelines can suffer corruption from partial failures, configuration errors, software bugs, or unexpected source-data anomalies. - Figma therefore added a validation workflow to verify correctness as data moves through snapshot exports, CDC capture, and incremental merging. Figma’s experience shows that incremental synchronization is a more sustainable alternative to repeated full-table exports. Combining managed infrastructure with custom orchestration can deliver lower latency, better cost control, and stronger operational flexibility than a one-size-fits-all commercial pipeline.
Read original(opens in new tab)