Piqama: Pinterest Quota Management Ecosystem (opens in new tab)
Piqama is Pinterest’s generic quota management ecosystem for controlling physical resources, service limits, and application-specific capacity. It centralizes quota definition, validation, authorization, distribution, enforcement, usage tracking, and optimization while allowing individual applications to customize implementation details. Its integrations demonstrate how the same platform can support both capacity management for Big Data and rate limiting for online services. ## Platform Architecture - Provides a centralized management portal accessible through REST and Thrift. - Supports multiple quota types and platforms. - Applications may use Piqama’s default enforcement mechanisms or supply their own. - Manages quotas throughout their lifecycle, from creation and updates to usage feedback and optimization. ## Quota Lifecycle Management - **Schema management:** Defines quota identifiers and hierarchical relationships, such as workloads within projects. - **Validation:** Supports pluggable schema and semantic validation, including remote checks to ensure quotas do not exceed cluster capacity. - **Authorization:** Requires ownership-based authorization for quota updates and deletions; owners may be individuals or groups. - **Update dispatch:** Can distribute changes through Piqama clients, Pinterest’s PinConf system, or custom dispatchers. - **Enforcement:** Default clients can make real-time decisions, such as serving or dropping requests when usage exceeds limits. - Applications can customize schema handling, validation, update delivery, and enforcement logic. ## Governance and Auto-Rightsizing - Piqama clients collect quota enforcement and usage statistics transparently. - Non-client applications can submit data through system-based or storage-based feedback loops. - Data is stored in Apache Iceberg on Amazon S3 using predefined schemas and pre-aggregation to reduce storage costs. - An independent rightsizing service consumes historical data from Presto, Iceberg, and other sources. - Rightsizing strategies account for organic growth, traffic bursts, and underutilization. - Pinterest has developed a capacity-quota strategy intended to maximize resource allocation without saturating Big Data systems. ## Quotas and Budgets - Budgets assign dollar amounts to organizations, teams, or projects, while quotas define the resources available within those financial constraints. - Chargeback systems convert resource consumption into costs. - Projects that exceed their budgets may receive reduced resource allocations based on their tier. - Teams may need additional funding or workload prioritization when resources are restricted. - Piqama is expected to integrate further with Pinterest’s Entitlement system. ## Capacity-Based Quotas in Big Data - Pinterest’s Moka platform uses Apache YuniKorn to schedule batch-processing resources such as memory, CPU, and GPUs. - Piqama manages project-level quotas including: - Guaranteed memory and vcore allocations. - Maximum memory and vcore consumption. - Maximum concurrent applications. - Quota values are generated through: - **Auto-rightsizing:** Uses historical usage within a sliding window to estimate future needs. - **Manual adjustments:** Allows development teams to make immediate quota changes. - Pinterest is also developing a budget-based method for generating quota values. Piqama provides a flexible foundation for governing resource consumption across Pinterest. Organizations adopting it can combine centralized policy and visibility with application-specific enforcement, while usage data enables more efficient and financially aligned quota allocation.