pinterestFeb 27, 2026

Bridging the Gap: Diagnosing Online–Offline Discrepancy in Pinterest’s L1 Conversion Models (opens in new tab)

Pinterest found that strong offline gains in L1 conversion-rate models did not translate into online improvements because training and serving environments were not aligned. Although experimental models reduced LogMAE by roughly 20–45% and improved calibration, online A/B tests showed neutral or worse CPA and unexpected oCPM mix shifts. The investigation identified feature coverage gaps and embedding version skew as structural causes rather than problems with offline evaluation or serving reliability. ## How L1 Models Are Evaluated - L1 filters and prioritizes ads under strict latency limits before downstream ranking and auction stages. - Offline evaluation focused on: - LogMAE and calibration - Performance across candidate pools and pCVR percentiles - Multiple data sources, including auction winners and candidates - Online evaluation focused on: - CPA and other business metrics - Candidate counts and recall across funnel stages - Differences among optimization types, especially oCPM traffic ## Hypotheses That Were Ruled Out - **Offline evaluation errors** - The experimental model consistently outperformed production across three log sources. - Gains remained across pCVR buckets, including after outlier handling. - **Exposure bias** - Increasing treatment traffic from approximately 20% to 70% did not resolve the online over-calibration issue. - **Serving failures** - Control and treatment had comparable success rates and p50/p90/p99 latency. - Timeouts and tail latency were therefore unlikely to explain the discrepancy. ## Missing Features in L1 Serving - Offline training used rich logged features, while online L1 embeddings only included features explicitly onboarded into the embedding pipeline. - Important feature families were absent online, including: - Targeting specification flags - Offsite conversion visit counts over 1-, 7-, 30-, and 90-day windows - Annotations and MediaSage image embeddings - Models learned to depend on these signals during training, but received a substantially thinner feature set when serving many oCPM and performance-oriented ads. - Pinterest updated UFR configurations to add the missing features to L1 embeddings. - Online feature coverage recovered, and online loss improved for CVR and engagement models, particularly on shopping traffic. - UFR tooling was also changed so features onboarded for L2 are automatically considered for L1 embedding usage. ## Query–Pin Embedding Version Skew - Pinterest’s two-tower architecture requires query and Pin embeddings to be generated from compatible model checkpoints. - Offline evaluation generally uses one fixed checkpoint for both towers. - Online pipelines could instead serve query and Pin embeddings produced from different model versions, creating a mismatch between training assumptions and production behavior. - This version skew was identified as a second structural source of online–offline inconsistency. ## Practical Conclusion Offline model quality is not sufficient for launching L1 improvements. Teams must verify feature coverage in serving artifacts such as ANN indices, enforce synchronized query and Pin embedding versions, and monitor funnel behavior and online feature coverage alongside standard offline metrics.

machine-learningfeature-storeranking-modelstwo-tower-architecture+4

pinterestFeb 24, 2026

Piqama: Pinterest Quota Management Ecosystem (opens in new tab)

Piqama is Pinterest’s generic quota management ecosystem for controlling physical resources, service limits, and application-specific capacity. It centralizes quota definition, validation, authorization, distribution, enforcement, usage tracking, and optimization while allowing individual applications to customize implementation details. Its integrations demonstrate how the same platform can support both capacity management for Big Data and rate limiting for online services. ## Platform Architecture - Provides a centralized management portal accessible through REST and Thrift. - Supports multiple quota types and platforms. - Applications may use Piqama’s default enforcement mechanisms or supply their own. - Manages quotas throughout their lifecycle, from creation and updates to usage feedback and optimization. ## Quota Lifecycle Management - **Schema management:** Defines quota identifiers and hierarchical relationships, such as workloads within projects. - **Validation:** Supports pluggable schema and semantic validation, including remote checks to ensure quotas do not exceed cluster capacity. - **Authorization:** Requires ownership-based authorization for quota updates and deletions; owners may be individuals or groups. - **Update dispatch:** Can distribute changes through Piqama clients, Pinterest’s PinConf system, or custom dispatchers. - **Enforcement:** Default clients can make real-time decisions, such as serving or dropping requests when usage exceeds limits. - Applications can customize schema handling, validation, update delivery, and enforcement logic. ## Governance and Auto-Rightsizing - Piqama clients collect quota enforcement and usage statistics transparently. - Non-client applications can submit data through system-based or storage-based feedback loops. - Data is stored in Apache Iceberg on Amazon S3 using predefined schemas and pre-aggregation to reduce storage costs. - An independent rightsizing service consumes historical data from Presto, Iceberg, and other sources. - Rightsizing strategies account for organic growth, traffic bursts, and underutilization. - Pinterest has developed a capacity-quota strategy intended to maximize resource allocation without saturating Big Data systems. ## Quotas and Budgets - Budgets assign dollar amounts to organizations, teams, or projects, while quotas define the resources available within those financial constraints. - Chargeback systems convert resource consumption into costs. - Projects that exceed their budgets may receive reduced resource allocations based on their tier. - Teams may need additional funding or workload prioritization when resources are restricted. - Piqama is expected to integrate further with Pinterest’s Entitlement system. ## Capacity-Based Quotas in Big Data - Pinterest’s Moka platform uses Apache YuniKorn to schedule batch-processing resources such as memory, CPU, and GPUs. - Piqama manages project-level quotas including: - Guaranteed memory and vcore allocations. - Maximum memory and vcore consumption. - Maximum concurrent applications. - Quota values are generated through: - **Auto-rightsizing:** Uses historical usage within a sliding window to estimate future needs. - **Manual adjustments:** Allows development teams to make immediate quota changes. - Pinterest is also developing a budget-based method for generating quota values. Piqama provides a flexible foundation for governing resource consumption across Pinterest. Organizations adopting it can combine centralized policy and visibility with application-specific enforcement, while usage data enables more efficient and financially aligned quota allocation.

amazon-s3apache-icebergrate-limitingresource-scheduling+4

pinterestFeb 17, 2026

Drastically Reducing Out-of-Memory Errors in Apache Spark at Pinterest (opens in new tab)

Pinterest developed **Auto Memory Retries** to reduce Spark out-of-memory failures without permanently assigning oversized executors to every task. The system detects OOM failures and retries affected tasks with progressively larger resource profiles, reducing both on-call incidents and wasted compute. Instead of tuning every job for its peak memory demand, Pinterest can size jobs around typical usage while handling exceptional tasks elastically. ## Pinterest’s Spark Environment - Pinterest processes more than **90,000 Spark jobs daily** across tens of thousands of nodes. - Its infrastructure includes: - Kubernetes clusters - Spark 3.2, with Spark 3.5 adoption underway - Apache Celeborn for shuffle - Apache YuniKorn for scheduling - Apache Gluten and Meta’s Velox for acceleration - Archer, Pinterest’s internal submission service - More than **4.6% of job failures** were caused by OOM errors. ## Why Manual Memory Tuning Was Insufficient - Pinterest’s clusters are memory-bound, so simply increasing executor sizes is expensive and difficult. - Automatic tuning generally reduces executor memory to match historical usage and improve resource efficiency. - Manual tuning can work, but requires substantial expertise because: - Different stages perform different operations. - Individual tasks may have very different memory needs because of data skew. - Configurations that work for most tasks may fail for a small number of high-memory tasks. - Auto Memory Retries allow jobs to target approximately their **P90 memory usage**, while automatically giving unusually demanding tasks more capacity. ## How Spark Executor Memory Works - An executor’s memory and CPU capacity determine how many tasks can run concurrently. - By default, each CPU core provides a task slot. - For example, with `spark.task.cpus=2`, an executor with two usable task slots and 8 GB of memory provides roughly 4 GB per task on average. - Memory is shared, so one task may temporarily use more than its average allocation if another uses less. - An OOM occurs when the combined memory usage of concurrent tasks exceeds the executor’s available memory. ## Auto Memory Retries Design Pinterest modified Spark’s scheduling loop so individual tasks can use resource profiles different from their parent `TaskSet`. - Each task can store an optional `taskRpId` identifying its retry resource profile. - Pinterest creates immutable retry profiles at **2x, 3x, and 4x** the base profile. - If off-heap memory is enabled, it is scaled as well. - Retries use a hybrid strategy: - **First retry:** Double `cpus per task`, allowing the task to run on an existing executor with fewer concurrent tasks. - **Later retry:** Launch a physically larger executor if the task still fails or already requires the entire executor. - The approach prioritizes reusing existing executors before provisioning larger ones. ## Changes to Spark Internals Pinterest extended core Spark components through Pinterest-specific subclasses rather than using a listener-only implementation. - **Task** - Stores the optional task resource profile ID. - **TaskSetManager** - Tracks tasks with non-default profiles. - Assigns the next larger retry profile after an OOM. - **TaskSchedulerImpl** - Allows tasks with increased CPU requirements to run on standard executors. - **ExecutorAllocationManager** - Tracks pending tasks by retry profile. - Requests larger executors when physical memory is required. - The feature-specific classes are loaded only when Auto Memory Retries is enabled. - The Spark UI was updated to display each task’s resource profile ID. ## Handling Tasks After an OOM - When a task fails on an executor with more than one core, its first retry doubles `spark.task.cpus`. - Other tasks in the same stage or future stages are unaffected. - Spark cannot reliably determine which concurrent task caused the executor-level OOM. - As a result, Pinterest treats **all tasks running on the terminated executor** as having failed due to OOM and routes them to retries that do not share the executor with other tasks. ## Practical Conclusion Pinterest’s approach makes executor sizing elastic at the task level: configure jobs for normal memory usage, then progressively increase resources only for tasks that need them. This can reduce OOM-related failures and operational load while avoiding the cost of running every task on oversized executors.

k8sapache-sparkmemory-managementresource-allocation+4

pinterestFeb 13, 2026

GPU-Serving Two-Tower Models for Lightweight Ads Engagement Prediction (opens in new tab)

Pinterest replaced its CPU-served two-tower model for ads lightweight ranking with a GPU-serving architecture based on MMOE and DCN. The more expressive model maintained latency comparable to the CPU baseline while reducing offline CTR loss by 5–10%. Separating standard and shopping ad models produced another 5–10% loss reduction and doubled offline iteration speed, with online improvements in CPC and CTR. ## Role of Lightweight Ranking - Lightweight ranking serves as an intermediate stage in Pinterest’s ads recommendation pipeline. - It filters a large pool of candidate ads before more complex downstream ranking models process them. - The two-tower design balances quality and latency: - The Pin tower generates ad embeddings offline through batch updates. - The query tower generates real-time user embeddings. - The prediction score is the sigmoid of the embeddings’ dot product. ## MMOE-DCN Model Architecture - The new system replaces the previous Multi-Task Multi-Domain (MTMD) model. - It combines: - Multi-gate Mixture-of-Experts (MMOE) with MLP-based gating. - Deep & Cross Network (DCN) layers for modeling feature interactions. - Each expert uses both full-rank and low-rank DCN layers. - Unlike MTMD, MMOE handles multi-task and multi-domain learning without relying on separate domain-specific modules. - GPU serving makes it practical to deploy this larger and more computationally demanding model while preserving CPU-baseline latency. ## Scenario-Specific Modeling - Standard and shopping ad scenarios are served as separate models. - Each model is trained only on data relevant to its scenario. - This specialization delivered an additional 5–10% reduction in offline loss. - Separating the models also doubled the speed of offline model iteration. ## Training Efficiency Improvements - **Dataloader optimization** - GPU prefetching prepares the next batch while the current batch is processed. - Additional worker threads take advantage of the 1 TB of CPU memory available on p4d instances. - **Model code optimization** - Operations that previously allocated zero-filled tensors on the CPU were moved to the GPU. - Fused kernels replaced multiple individual kernels to reduce execution overhead. - **Training configuration** - BF16 precision improved processing speed. - Larger batch sizes increased GPU memory utilization. ## Evaluation Results - The model uses downstream ranking scores as labels and optimizes KL divergence between those labels and its predictions. - Evaluation covers both: - Auction winners—ads ultimately inserted and shown to users. - Auction candidates—ads passed to downstream ranking. - Offline loss decreased significantly across all evaluated slices. - Online experiments showed: - Lower cost per click (CPC), which is favorable. - Higher click-through rate (CTR). GPU-serving a more complex MMOE-DCN two-tower model allowed Pinterest to improve ad engagement prediction without sacrificing serving latency. The results support using GPU infrastructure, scenario-specific models, and targeted training optimizations to scale lightweight ranking systems.

recommendation-systemsgpu-servingctr-predictiontwo-tower-models+4

pinterestFeb 5, 2026

Next Generation DB Ingestion at Pinterest (opens in new tab)

Pinterest replaced fragmented, batch-oriented database ingestion with a unified Change Data Capture (CDC) framework. The new architecture uses Debezium/TiCDC, Kafka, Flink, Spark, and Iceberg to process only changed records, reducing latency from over 24 hours to minutes while lowering infrastructure costs. It also provides native row-level deletion, scalable operations, and improved compliance. ## Problems with the Legacy System - Batch workflows often delayed updates by more than 24 hours. - Full-table processing was inefficient because many tables changed by less than 5% each day. - Lack of row-level deletion support complicated data compliance. - Multiple independently maintained pipelines created operational complexity and inconsistent data quality. ## Unified CDC-Based Architecture - Supports MySQL, TiDB, and KVStore. - Captures database changes through a generic CDC service and publishes them to Kafka, typically in under one second. - Flink processes events in near real time and stores them in append-only CDC Iceberg tables on S3. - Spark jobs run periodically—often every 15 minutes—to merge recent changes into base Iceberg tables. - A bootstrap pipeline initializes base tables from historical database dumps. - Maintenance jobs handle compaction and snapshot expiration. - The framework is designed for at-least-once processing, petabyte-scale data, thousands of pipelines, and YAML-based configuration. ## CDC Tables and Base Tables - CDC tables act as time-series ledgers containing every change event. - CDC data typically becomes available within five minutes. - Base tables mirror the current state of the source database while retaining historical records. - Base-table latency is generally between 15 minutes and one hour. ## Upserting Changes into Base Tables - Spark first identifies the newest event for each primary key. - Events are ranked by timestamp and GTID, then deduplicated. - Iceberg’s `MERGE INTO` applies the resulting changes: - Deletes matching records when the event represents a deletion. - Updates existing records. - Inserts new records unless the event is a deletion. - The process uses a recent CDC window and a processing watermark to avoid reprocessing unnecessary data. ## Choosing Merge-on-Read - Pinterest standardized on Iceberg’s Merge-on-Read (MOR) strategy. - Copy-on-Write (COW) was rejected for most workloads because: - It requires more computation during writes. - It produces substantially larger replacement files, increasing storage costs. - MOR better balances update performance and storage efficiency for frequent incremental changes. ## Partitioning for Faster Upserts - Large base tables can be partitioned using a hash bucket of the primary key. - For example, `bucket(100, id)` distributes records across 100 partitions. - This allows Spark to process partitions in parallel and reduces the data scanned or rewritten during merges. - Iceberg tables are configured with format version 2, identifier fields, merge-on-read update and delete modes, and target file sizes. ## Small-File Challenge - Bucketing improved parallelism but caused each upsert to generate many small files within partitions. - The article indicates that Pinterest investigated this bottleneck and introduced further optimizations, though the supplied excerpt ends before describing them. Pinterest’s CDC-based design provides a substantially faster and more efficient alternative to full-table batch ingestion. Teams adopting a similar system should combine incremental CDC processing with partitioning, merge-on-read storage, bootstrapping, and ongoing file-maintenance strategies.

apache-kafkaapache-sparkmysqlapache-iceberg+4

pinterestFeb 2, 2026

Beyond Two Towers: Re-architecting the Serving Stack for Next-Gen Ads Lightweight Ranking Models… (opens in new tab)

Two-Tower models make retrieval and lightweight ranking highly efficient by scoring user and item embeddings with a dot product, but they cannot represent rich user-item interactions or deep feature crossings. This post describes an ads-serving redesign that introduces general-purpose GPU models while preserving end-to-end latency. The main strategy is to reduce data movement, move filtering logic onto the GPU, and optimize inference from an initial 4-second p90 latency to about 20 milliseconds. ## Why Move Beyond Two-Tower Models - Two-Tower architectures independently encode users and items, enabling fast scoring across millions of candidates. - Their decoupled structure limits: - User-item interaction features - Target attention - Early feature crossing - Deep architectures requiring simultaneous access to user and candidate data - More expressive models require GPU-based general-purpose inference rather than specialized dot-product or ANN retrieval. - The existing retrieval stack was not designed to transfer large candidate and feature sets to a GPU, creating a major latency challenge. ## Restructuring the Serving Funnel The traditional funnel consisted of: - Feature expansion for thousands of candidates - Retrieval and Two-Tower lightweight ranking - Heavy ranking and auction processing for the top documents Adding GPU inference directly to this flow would require fetching, serializing, transferring, and returning features for tens of thousands of documents. The authors therefore redesigned the entire early-stage serving pipeline instead of optimizing the model alone. ## Segmenting the Inventory for Feature Fetching Feature retrieval was a major latency source, often taking longer than model inference for workloads ranging from 10,000 to 100,000 documents. - **High-value inventory:** Roughly 1 million documents responsible for a substantial share of revenue have their features embedded in the PyTorch model as registered buffers. - Features become part of the model state, similar to weights. - They remain in GPU high-bandwidth memory. - Requests avoid remote feature-service calls and host-to-device transfers. - The model file must be periodically updated to refresh features. - Future work may include GPU-based caching. - **Long-tail inventory:** The remaining roughly 1 billion documents use a high-performance key-value store with in-host caching. - The post focuses on the first strategy, which is already running in production. ## Moving Business Logic onto the GPU Previously, the model returned scores for approximately 100,000 candidates, while CPU-side code handled utility calculation, filtering, diversity, deduplication, and top-k selection. - The new PyTorch model performs these operations directly: - Combines pCTR, pCVR, bid, and other signals into utility scores. - Applies diversity and filtering rules. - Performs top-k selection. - The GPU returns only the final winners—typically around 1,000 documents—instead of all candidate scores. - This reduces device-to-host data transfer and takes advantage of GPU parallelism. - The approach works because lightweight-ranking business rules are sufficiently simple to express with tensor operations. ## Reducing GPU Inference Latency Initial GPU inference measured roughly 4,000 ms at p90, far too slow for real-time serving. Several systems optimizations reduced this to approximately 20 ms: - **Multiple CUDA streams:** Separate streams for workers allow host-to-device transfers, computation, and device-to-host transfers to overlap. - **Worker alignment:** Worker threads are matched and pinned to physical CPU cores to reduce context switching and lock contention. - **Kernel fusion:** Triton kernels combine operations such as linear layers and activations, reducing memory traffic. - **BF16 computation:** Brain Floating Point 16 lowers memory usage and accelerates arithmetic compared with FP32. - **Profiling tools:** PyTorch Profiler and NVIDIA Nsight Systems were used to identify bottlenecks. ## Practical Recommendation Deploying more expressive ranking models requires rethinking the serving architecture around data movement and execution placement. Embedding frequently used features, executing business logic on the GPU, and applying low-level CUDA and kernel optimizations can make complex neural ranking feasible without increasing end-to-end latency.

machine-learningrecommendation-systemspytorchgpu-inference+4

pinterestJan 28, 2026

Ads Candidate Generation using Behavioral Sequence Modeling (opens in new tab)

Pinterest’s Ads team uses behavioral sequence modeling to improve ad candidate generation by predicting what users are likely to convert on next. Transformer-based two-tower models first predict relevant advertisers and then specific products, using offsite activity such as views, purchases, and add-to-cart events. The advertiser model is already in production, while item-level modeling addresses Pinterest’s rapidly growing catalog and enables more precise, scalable personalization. ## Predicting Advertiser Interaction - A bidirectional Transformer encodes each user’s behavioral event sequence. - An MLP-based advertiser tower represents candidate advertisers. - Training uses: - In-batch negative samples - Sampled softmax loss - Positive events consisting of checkout, add-to-cart, or signup conversions within a future K-day window - Log-Q bias correction to avoid excessively penalizing popular advertisers - The model is evaluated with Recall@K by comparing user and advertiser embedding similarity against an indexed set of roughly 2 million advertisers. - An offline batch job generates each user’s top 100 advertisers and publishes them to the online feature store. - During ad serving, eligible ads from those advertisers are passed to the L1 ranker, blended with other candidate sources, and scored by heavier downstream models and the marketplace auction. - Online experiments produced higher conversion volume and lower cost per action. - The advertiser-level model has served production traffic for Standard ads since Spring 2024. ## Moving from Advertisers to Products - Pinterest next sought to predict the specific products a user would interact with, rather than only the likely advertiser. - Item-level prediction better matches the item-based ad delivery funnel and avoids forcing downstream models to score an impractically large set of products from selected advertisers. - The approach aims to capture both immediate intent and longer-term interests. ## Item-Level Model Architecture - The model retains the two-tower design: - A user tower encodes behavioral sequences. - An item tower represents individual shopping product Pins. - Item representations combine: - Internal Pin embeddings learned from Pinterest’s engagement graph - Product metadata from the merchant catalog - Because the catalog exceeds 1 billion items, training uses both in-batch negatives and a randomly sampled negative set of 20 million Pins. - The model uses the same conversion labels as the advertiser model. - Label weights and log-Q parameters are tuned to balance retrieval quality with diversity across both products and advertisers. - Daily inference updates user embeddings only for users with new activity, appending them to a previous feature-store snapshot to reduce computation. - The trained item tower indexes hundreds of millions of ad items. ## Evaluation and Diversity - Item retrieval is evaluated using cosine similarity and hit rates at different K values. - Final model selection considers both: - Item-level Recall@K - Advertiser-level Recall@K - Qualitative review is also important because offsite activity is sparse and noisy. - The model is compared with max-pooling and mean-pooling baselines that use aggregated embeddings without Transformer-based sequence modeling. - The evaluation emphasizes that strong retrieval must also produce semantically relevant and sufficiently diverse recommendations. Pinterest’s progression from advertiser prediction to item prediction shows how behavioral sequence models can make ad retrieval more personalized while remaining scalable. A practical system should combine sequence-aware user representations, large-scale approximate retrieval, and explicit controls for popularity, diversity, and computational efficiency.

transformerembeddingsfeature-storecontrastive-learning+4

pinterestJan 13, 2026

PinLanding: Turn Billions of Products into Instant Shopping Collections with Multimodal AI (opens in new tab)

PinLanding is a production pipeline for turning billions of products into searchable shopping collections using multimodal AI. Rather than relying mainly on historical queries or manual curation, it derives structured product attributes from images and metadata, then aligns those attributes with real user search behavior. The system combines multimodal LLMs, embedding-based consolidation, a CLIP-style classifier, and distributed infrastructure to produce scalable, precise shopping feeds. ## Understanding Shopping Intent - Pinterest analyzes search history, autocomplete use, filters, and browsing paths to estimate shopping demand. - Existing systems handle high-volume queries such as “black cocktail dress” well, but provide weaker coverage for: - Long-tail queries - Conversational requests - Contextual intents such as “what to wear for an Italian summer vacation” - The analysis identifies: - Product areas with strong demand but poor collection coverage - Important attribute dimensions, including color, occasion, style, fit, price, and brand - The goal is to expand and improve collection coverage, not replace query understanding. ## Generating and Curating Shopping Topics - Each product is represented by an image plus metadata such as title, description, merchant tags, and price. - A vision-language model generates normalized key-value attributes rather than free-form descriptions. - Raw model output has high recall but produces: - Excessively specific attributes - Near-duplicates such as “boho,” “bohemian,” and “boho-chic” - Sparse attributes that apply to very few products - PinLanding builds a compact vocabulary through: - Frequency filtering to remove rarely useful attributes - Embedding-based clustering to merge semantically similar terms - Manual and LLM-assisted review - An LLM judge evaluates generated topics for semantic coherence, realistic shopping intent, and alignment with natural search phrasing. ## Scalable Attribute Assignment - Running the vision-language model over every product is too expensive and operationally fragile. - PinLanding trains a CLIP-inspired dual encoder: - One encoder embeds product images and text - Another embeds attribute phrases - Matching product-attribute pairs are trained as positives, while mismatches are negatives - A bidirectional contrastive loss aligns related products and attributes. - At inference, products and attributes are embedded once, and attributes are assigned when similarity exceeds a calibrated threshold. - This produces fewer distinct attributes while increasing the average number assigned to each product, creating a denser and more consistent attribute graph. ## Distributed Feed Construction - Ray handles large-scale batch inference across millions of products and topics. - The pipeline separates: - CPU-based image and metadata loading, tokenization, and serialization - GPU-based classifier inference - Streaming allows preprocessing and inference to overlap, while heterogeneous CPU and GPU clusters can scale independently. - The classifier pipeline reportedly completes in about 12 hours using eight NVIDIA A100 GPUs, at an estimated cost of roughly $500 per training run. - Feed construction uses approximate-nearest-neighbor techniques and strict attribute matching. - Topics are represented as attribute tuples, such as: - Category: dress - Color: yellow - Season: summer - Occasion: party - Apache Spark computes topic-product relevance using shared attributes and confidence weights, with partitioning and overlap filters reducing unnecessary candidate comparisons. The core recommendation is to combine user-behavior signals with content-first multimodal modeling. This approach can expand shopping coverage into conversational and long-tail intents while remaining practical through attribute consolidation, contrastive retrieval, and distributed inference.

multimodal-aiapache-sparkcontrastive-learningclip+4

pinterestDec 10, 2025

LLM-Powered Relevance Assessment for Pinterest Search (opens in new tab)

Pinterest Search uses fine-tuned multilingual LLMs to assess search-result relevance at a much larger scale than human labeling allows. The approach combines five-level relevance classification, stratified query sampling, and paired A/B-test evaluation to detect smaller overall effects and differences across query types. XLM-RoBERTa-large provides a practical balance of accuracy and cost, achieving strong agreement with human judgments while enabling substantially faster labeling. ## Relevance Measurement Challenges - Search relevance measures how well Pins satisfy a user’s query, rather than merely reflecting past engagement. - Human annotations are expensive and limited in volume. - Previous sampling designs could detect only relatively large topline changes, with minimum detectable effects (MDEs) around 1.3%–1.5%. - Limited labels also made it difficult to measure heterogeneous effects across query interests or popularity segments. ## Fine-Tuned LLM Relevance Model - Pinterest defines relevance using five labels: - L5: Highly Relevant - L4: Relevant - L3: Marginally Relevant - L2: Irrelevant - L1: Highly Irrelevant - A cross-encoder model predicts the relevance of each Pin for a query. - Open-source multilingual models are fine-tuned on human-annotated examples using multiclass cross-entropy loss. - Pin representations include: - Titles and descriptions - BLIP-generated image captions - Linked-page titles and descriptions - Board titles where Pins were saved - Highly engaged query tokens associated with the Pin - Models tested included multilingual BERT, T5, mDeBERTa, XLM-RoBERTa, and Llama 3. - The final relevance label is selected from the model’s five output scores using argmax. ## Stratified Query Sampling - Lower LLM labeling costs allow Pinterest to use much larger and more detailed samples. - Queries are stratified using: - A DistilBERT-based query-to-interest model - Query popularity, based on how many users issue each query - Stratification improves representativeness and reduces variance by grouping similar queries. - Pinterest moved from simple random sampling to stratified sampling with optimal allocation across strata. - Most of the MDE improvement came from variance reduction through stratification. - The redesigned process reduced MDEs from approximately 1.3%–1.5% to 0.25% or less. ## LLM-Based A/B-Test Measurement - Pinterest samples paired queries from control and treatment groups. - Pairing controls for differences between queries, which are a major source of relevance variance. - For each query, the top 25 results are retained and labeled by the LLM. - Query-level relevance is measured using sDCG@25, a variant of nDCG that assumes an unlimited supply of highly relevant L5 results. - Results are aggregated into topline experiment metrics. - Heterogeneous effects are analyzed by query popularity and interest categories such as beauty, fashion, and art. - The Benjamini–Hochberg procedure controls the false discovery rate when testing multiple segments. ## Model Choice and Validation - XLM-RoBERTa-large was selected for its balance of quality and efficiency. - On a single A10G GPU, it can label 150,000 rows in about 30 minutes. - Llama 3–8B produced slightly better accuracy but required roughly six times the inference time and cost. - LLM labels matched human labels exactly for 73.7% of Pins. - A total of 91.7% of predictions differed from human ratings by no more than one relevance point. Pinterest’s approach makes relevance evaluation cheaper, faster, and more statistically sensitive. Fine-tuned LLMs paired with stratified sampling are recommended for search experimentation when human labeling cannot provide enough coverage to detect small or heterogeneous ranking effects.

llmnlpfine-tuningab-testing+4

pinterestDec 8, 2025

How Pinterest Built a Real‑Time Radar for Violative Content using AI (opens in new tab)

Pinterest built an AI-assisted prevalence measurement system to estimate how often users actually see policy-violating content, rather than relying only on user reports. The system samples daily impressions, uses production risk scores to improve efficiency, labels content with a multimodal LLM, and applies statistical reweighting to preserve unbiased estimates. This enables daily, segmented monitoring with substantially lower cost and latency than human-only review. ## Why Prevalence Matters - User reports miss important harms because: - Some sensitive issues, such as self-harm, are under-reported. - Users seeking harmful content may not report it. - Rare policy categories provide too few reports for reliable trend detection. - Human review of reports is expensive and slow. - Prevalence measures exposure: the share of total views directed to violating content. - This helps Pinterest identify under-reported harms, evaluate interventions, and detect changes earlier. - Human-only prevalence studies were previously conducted only about every six months and required multiple reviewers plus adjudication. ## What Pinterest Measures - Daily prevalence is calculated as: - **Views of content violating a policy ÷ total views** - For example, 10 violating views in a sample of 100,000 produces an estimated prevalence of 0.01%. - Results include 95% confidence intervals to communicate statistical precision. - Metrics can be segmented by: - Policy area, such as Adult Content, Self-harm, or Graphic Violence - Sub-policy, such as nudity versus explicit sexual content - Surface, including Homefeed, Search, and Related Pins - Content age, geography, and user-age groups where relevant ## Risk-Aware, Unbiased Sampling - Pinterest samples from the daily user-impressions stream. - Production enforcement risk scores are used to prioritize likely high-risk and high-exposure content, but they are not treated as labels or eligibility rules. - Missing scores are replaced with the day’s median so that new content remains eligible. - Weighted reservoir sampling approximates probability-proportional-to-size sampling, considering impressions and risk scores. - Inverse-probability weighting removes the bias introduced by risk-based sampling, ensuring estimates represent impressions rather than model thresholds. - Pinterest uses Hansen–Hurwitz ratio estimators for sampling with replacement and Horvitz–Thompson ratio estimators for sampling without replacement. - Pure random sampling is also available for validation studies. ## LLM-Based Labeling - A multimodal LLM analyzes sampled content using both images and text. - Prompts are reviewed by policy subject-matter experts and can return structured label hierarchies such as `safe`, `not_safe`, and `unsure`. - Each decision records: - The label and brief rationale - Policy version - Prompt and model identifiers - Token usage and run cost - Human validation is performed on strategically selected samples to identify edge cases and AI blind spots. - The LLM is tested against human-reviewed gold sets before launch and periodically afterward to detect drift. - The workflow is reportedly 15 times faster and far cheaper than human-only labeling while maintaining comparable decision quality and statistical governance. ## Production System and Monitoring - Inputs include entity-by-day engagement data such as impressions, clicks, hides, and reports, alongside current production risk scores. - The system stores prevalence estimates, sampling weights, labels, diagnostics, and lineage for audits. - Dashboards display: - Daily prevalence and 95% confidence intervals - Confidence-interval width and effective sample size - Sample positive rate - Risk-score distributions - Prompt, model, taxonomy, and metric versions - Teams can pivot results by policy, sub-policy, and surface. - Validation samples and run-health information help monitor both statistical quality and operational reliability. Pinterest’s approach combines probability sampling, inverse-probability estimation, and continuously calibrated multimodal AI labeling to create a daily radar for harmful exposure. The practical recommendation is to use AI to scale measurement, but retain rigorous sampling, human validation, confidence intervals, and full model and policy lineage so that faster estimates remain trustworthy.

llmmachine-learningmultimodal-aitrust-and-safety+3

pinterestDec 5, 2025

Improving Quality of Recommended Content through Pinner Surveys (opens in new tab)

Pinterest uses Pinner surveys to measure visual quality and incorporate user preferences into recommendation systems, rather than optimizing solely for engagement. The company surveyed 5,000 Pins, trained a lightweight neural network to predict average perceived quality, and applied the resulting model across Homefeed, Related Pins, and Search. This approach aims to reduce clickbait and promote content that supports positive, long-term user experiences. ## Why Engagement Alone Is Insufficient - High engagement does not necessarily indicate high-quality content; optimizing for clicks can promote clickbait or harmful material. - Pinterest defines quality as content that feels good, inspires further exploration, and encourages fulfilling long-term engagement. - Direct user feedback helps recommendation systems prioritize content that Pinners actually value. - The work supports Pinterest’s Inspired Internet Pledge principles, especially listening to users and tuning the platform for wellbeing. ## Collecting Pinner Quality Ratings - Pinners rated images from 1 to 5 in response to: “How visually pleasing or displeasing is this Pin?” - Pinterest collected ratings for 5,000 Pins, sampling 1,000 from each of five major interest categories: - Art - Beauty - DIY & Crafts - Home Decor - Women’s Fashion - Pins were sampled based on impressions and were generally mid-to-high quality rather than deliberately exposing users to poor content. - Each image received at least 10 ratings, allowing Pinterest to average responses and reduce noise from subjectivity or accidental misclicks. - Surveys were considered appropriate for visual appeal, which is subjective but still measurable across many users. More objective issues should be evaluated by trained reviewers, while highly contextual judgments such as personal relevance are harder to capture with a single Pin-level score. - Highly rated content included makeup, grooming styles, maximalist interiors, landscapes, sunsets, and baby animals. - Home Decor images tended to receive higher ratings overall, while Art showed the greatest variation, reflecting its subjective nature. ## Training a Visual-Quality Model - Pinterest trained a model to estimate the average Pinner’s perception of visual quality from image embeddings. - Embeddings encode visual, textual, and behavioral information, including relationships between images and the boards where they are saved. - The model produces a score from 0 to 1, with higher values representing greater perceived quality. - Pinterest chose a small fully connected neural network with approximately 92,000 parameters: - The limited size helps prevent overfitting to the 5,000-image dataset. - It also makes large-scale inference faster and less expensive. - Instead of predicting an exact rating, the model uses pairwise ranking: - It learns which of two images Pinners would consider better. - The comparison is based on each image’s mean survey rating. - Training comparisons are restricted to images within the same top-level interest category, encouraging the model to learn visual quality rather than simply recognizing that one topic is more popular than another. Pinterest’s approach demonstrates how survey-based quality signals can complement engagement metrics. Training recommendation systems on what users perceive as appealing can help the platform promote more satisfying content while reducing incentives to favor attention-grabbing but low-quality material.

machine-learningrecommendation-systemsembeddingsneural-networks+3

pinterestDec 4, 2025

On the (re)-prioritization of open-source AI (opens in new tab)

Pinterest argues that AI competition is shifting beyond ever-larger proprietary models. Open-source models now deliver comparable quality at a fraction of the cost, while compact models fine-tuned for specific tasks can outperform general-purpose systems. The company’s strategy is to combine open-source models with Pinterest-specific data, internal systems, and deep product integration. ## Open-Source Models and Cost Efficiency - Pinterest reports achieving performance comparable to leading proprietary AI models at less than 10% of the cost. - The company is shifting more investment toward fine-tuned open-source models, especially for visual and multimodal applications. - As core LLM architectures become increasingly commoditized, competitive advantage is moving toward: - Domain-specific data - Personalization - Product integration - End-to-end system optimization ## Choosing What to Build, Buy, or Adapt Pinterest evaluates foundation-model strategy by modality: - **User modeling and recommendation** - These systems are tightly coupled to a product’s behavior and are generally built internally. - Pinterest uses long-term user-action sequences and a graph containing hundreds of billions of user, board, and content nodes. - Examples include PinFM for representation learning and PinRec for generative recommendations. - **Visual models** - Pinterest largely trains visual encoders and diffusion models in-house. - Its visual-search data and image-board collections provide the weakly supervised datasets needed for large-scale training. - Internal models benefit from Pinterest’s specialized visual domain. - **Text models** - Pinterest has historically relied more on open-source and proprietary third-party LLMs. - Progress in reasoning and language modeling depends heavily on enormous datasets and compute resources, making external models practical. ## Domain-Specific Data as the Differentiator - Open-source multimodal architectures are narrowing the capability gap with proprietary models. - Pinterest’s experience reflects an older machine-learning pattern: model architectures become broadly available, while value comes from specialized data and fine-tuning. - Its visual encoders, including UVE and PinCLIP, improved retrieval by training on Pinterest image and visual-search data rather than using generic embeddings. - Pinterest Canvas similarly adapts an internally trained diffusion model for image editing and enhancement, outperforming larger general-purpose visual-generation models in those use cases. ## Pinterest Assistant and Specialized Tools - Pinterest Assistant combines: - Multimodal retrieval systems - Recommendation services - Specialized generative models - A core multimodal LLM - Most recommendation and agentic capabilities are handled by Pinterest-native tools built on its user and visual foundation models. - The central LLM acts primarily as an intelligent router, handling query understanding, planning, and tool calling rather than performing every task itself. - This architecture allows Pinterest to improve the overall product by optimizing smaller, specialized components instead of relying solely on a larger general-purpose model. Pinterest’s recommendation is to use open-source models as adaptable building blocks, then differentiate through proprietary data, specialized models, and tight integration with the product. The most effective AI systems may therefore be smaller, cheaper, and more purpose-built than frontier general-purpose models.

llmgen-aimultimodal-aifine-tuning+4