pinterestFeb 24, 2026

Piqama: Pinterest Quota Management Ecosystem (opens in new tab)

Piqama is Pinterest’s generic quota management ecosystem for controlling physical resources, service limits, and application-specific capacity. It centralizes quota definition, validation, authorization, distribution, enforcement, usage tracking, and optimization while allowing individual applications to customize implementation details. Its integrations demonstrate how the same platform can support both capacity management for Big Data and rate limiting for online services. ## Platform Architecture - Provides a centralized management portal accessible through REST and Thrift. - Supports multiple quota types and platforms. - Applications may use Piqama’s default enforcement mechanisms or supply their own. - Manages quotas throughout their lifecycle, from creation and updates to usage feedback and optimization. ## Quota Lifecycle Management - **Schema management:** Defines quota identifiers and hierarchical relationships, such as workloads within projects. - **Validation:** Supports pluggable schema and semantic validation, including remote checks to ensure quotas do not exceed cluster capacity. - **Authorization:** Requires ownership-based authorization for quota updates and deletions; owners may be individuals or groups. - **Update dispatch:** Can distribute changes through Piqama clients, Pinterest’s PinConf system, or custom dispatchers. - **Enforcement:** Default clients can make real-time decisions, such as serving or dropping requests when usage exceeds limits. - Applications can customize schema handling, validation, update delivery, and enforcement logic. ## Governance and Auto-Rightsizing - Piqama clients collect quota enforcement and usage statistics transparently. - Non-client applications can submit data through system-based or storage-based feedback loops. - Data is stored in Apache Iceberg on Amazon S3 using predefined schemas and pre-aggregation to reduce storage costs. - An independent rightsizing service consumes historical data from Presto, Iceberg, and other sources. - Rightsizing strategies account for organic growth, traffic bursts, and underutilization. - Pinterest has developed a capacity-quota strategy intended to maximize resource allocation without saturating Big Data systems. ## Quotas and Budgets - Budgets assign dollar amounts to organizations, teams, or projects, while quotas define the resources available within those financial constraints. - Chargeback systems convert resource consumption into costs. - Projects that exceed their budgets may receive reduced resource allocations based on their tier. - Teams may need additional funding or workload prioritization when resources are restricted. - Piqama is expected to integrate further with Pinterest’s Entitlement system. ## Capacity-Based Quotas in Big Data - Pinterest’s Moka platform uses Apache YuniKorn to schedule batch-processing resources such as memory, CPU, and GPUs. - Piqama manages project-level quotas including: - Guaranteed memory and vcore allocations. - Maximum memory and vcore consumption. - Maximum concurrent applications. - Quota values are generated through: - **Auto-rightsizing:** Uses historical usage within a sliding window to estimate future needs. - **Manual adjustments:** Allows development teams to make immediate quota changes. - Pinterest is also developing a budget-based method for generating quota values. Piqama provides a flexible foundation for governing resource consumption across Pinterest. Organizations adopting it can combine centralized policy and visibility with application-specific enforcement, while usage data enables more efficient and financially aligned quota allocation.

apache-icebergamazon-s3rate-limitingresource-scheduling+4

pinterestFeb 5, 2026

Next Generation DB Ingestion at Pinterest (opens in new tab)

Pinterest replaced fragmented, batch-oriented database ingestion with a unified Change Data Capture (CDC) framework. The new architecture uses Debezium/TiCDC, Kafka, Flink, Spark, and Iceberg to process only changed records, reducing latency from over 24 hours to minutes while lowering infrastructure costs. It also provides native row-level deletion, scalable operations, and improved compliance. ## Problems with the Legacy System - Batch workflows often delayed updates by more than 24 hours. - Full-table processing was inefficient because many tables changed by less than 5% each day. - Lack of row-level deletion support complicated data compliance. - Multiple independently maintained pipelines created operational complexity and inconsistent data quality. ## Unified CDC-Based Architecture - Supports MySQL, TiDB, and KVStore. - Captures database changes through a generic CDC service and publishes them to Kafka, typically in under one second. - Flink processes events in near real time and stores them in append-only CDC Iceberg tables on S3. - Spark jobs run periodically—often every 15 minutes—to merge recent changes into base Iceberg tables. - A bootstrap pipeline initializes base tables from historical database dumps. - Maintenance jobs handle compaction and snapshot expiration. - The framework is designed for at-least-once processing, petabyte-scale data, thousands of pipelines, and YAML-based configuration. ## CDC Tables and Base Tables - CDC tables act as time-series ledgers containing every change event. - CDC data typically becomes available within five minutes. - Base tables mirror the current state of the source database while retaining historical records. - Base-table latency is generally between 15 minutes and one hour. ## Upserting Changes into Base Tables - Spark first identifies the newest event for each primary key. - Events are ranked by timestamp and GTID, then deduplicated. - Iceberg’s `MERGE INTO` applies the resulting changes: - Deletes matching records when the event represents a deletion. - Updates existing records. - Inserts new records unless the event is a deletion. - The process uses a recent CDC window and a processing watermark to avoid reprocessing unnecessary data. ## Choosing Merge-on-Read - Pinterest standardized on Iceberg’s Merge-on-Read (MOR) strategy. - Copy-on-Write (COW) was rejected for most workloads because: - It requires more computation during writes. - It produces substantially larger replacement files, increasing storage costs. - MOR better balances update performance and storage efficiency for frequent incremental changes. ## Partitioning for Faster Upserts - Large base tables can be partitioned using a hash bucket of the primary key. - For example, `bucket(100, id)` distributes records across 100 partitions. - This allows Spark to process partitions in parallel and reduces the data scanned or rewritten during merges. - Iceberg tables are configured with format version 2, identifier fields, merge-on-read update and delete modes, and target file sizes. ## Small-File Challenge - Bucketing improved parallelism but caused each upsert to generate many small files within partitions. - The article indicates that Pinterest investigated this bottleneck and introduced further optimizations, though the supplied excerpt ends before describing them. Pinterest’s CDC-based design provides a substantially faster and more efficient alternative to full-table batch ingestion. Teams adopting a similar system should combine incremental CDC processing with partitioning, merge-on-read storage, bootstrapping, and ongoing file-maintenance strategies.

apache-icebergapache-kafkaapache-sparkmysql+4