pinterestJan 13, 2026

PinLanding: Turn Billions of Products into Instant Shopping Collections with Multimodal AI (opens in new tab)

PinLanding is a production pipeline for turning billions of products into searchable shopping collections using multimodal AI. Rather than relying mainly on historical queries or manual curation, it derives structured product attributes from images and metadata, then aligns those attributes with real user search behavior. The system combines multimodal LLMs, embedding-based consolidation, a CLIP-style classifier, and distributed infrastructure to produce scalable, precise shopping feeds. ## Understanding Shopping Intent - Pinterest analyzes search history, autocomplete use, filters, and browsing paths to estimate shopping demand. - Existing systems handle high-volume queries such as “black cocktail dress” well, but provide weaker coverage for: - Long-tail queries - Conversational requests - Contextual intents such as “what to wear for an Italian summer vacation” - The analysis identifies: - Product areas with strong demand but poor collection coverage - Important attribute dimensions, including color, occasion, style, fit, price, and brand - The goal is to expand and improve collection coverage, not replace query understanding. ## Generating and Curating Shopping Topics - Each product is represented by an image plus metadata such as title, description, merchant tags, and price. - A vision-language model generates normalized key-value attributes rather than free-form descriptions. - Raw model output has high recall but produces: - Excessively specific attributes - Near-duplicates such as “boho,” “bohemian,” and “boho-chic” - Sparse attributes that apply to very few products - PinLanding builds a compact vocabulary through: - Frequency filtering to remove rarely useful attributes - Embedding-based clustering to merge semantically similar terms - Manual and LLM-assisted review - An LLM judge evaluates generated topics for semantic coherence, realistic shopping intent, and alignment with natural search phrasing. ## Scalable Attribute Assignment - Running the vision-language model over every product is too expensive and operationally fragile. - PinLanding trains a CLIP-inspired dual encoder: - One encoder embeds product images and text - Another embeds attribute phrases - Matching product-attribute pairs are trained as positives, while mismatches are negatives - A bidirectional contrastive loss aligns related products and attributes. - At inference, products and attributes are embedded once, and attributes are assigned when similarity exceeds a calibrated threshold. - This produces fewer distinct attributes while increasing the average number assigned to each product, creating a denser and more consistent attribute graph. ## Distributed Feed Construction - Ray handles large-scale batch inference across millions of products and topics. - The pipeline separates: - CPU-based image and metadata loading, tokenization, and serialization - GPU-based classifier inference - Streaming allows preprocessing and inference to overlap, while heterogeneous CPU and GPU clusters can scale independently. - The classifier pipeline reportedly completes in about 12 hours using eight NVIDIA A100 GPUs, at an estimated cost of roughly $500 per training run. - Feed construction uses approximate-nearest-neighbor techniques and strict attribute matching. - Topics are represented as attribute tuples, such as: - Category: dress - Color: yellow - Season: summer - Occasion: party - Apache Spark computes topic-product relevance using shared attributes and confidence weights, with partitioning and overlap filters reducing unnecessary candidate comparisons. The core recommendation is to combine user-behavior signals with content-first multimodal modeling. This approach can expand shopping coverage into conversational and long-tail intents while remaining practical through attribute consolidation, contrastive retrieval, and distributed inference.

multimodal-aiapache-sparkcontrastive-learningclip+4

pinterestDec 8, 2025

How Pinterest Built a Real‑Time Radar for Violative Content using AI (opens in new tab)

Pinterest built an AI-assisted prevalence measurement system to estimate how often users actually see policy-violating content, rather than relying only on user reports. The system samples daily impressions, uses production risk scores to improve efficiency, labels content with a multimodal LLM, and applies statistical reweighting to preserve unbiased estimates. This enables daily, segmented monitoring with substantially lower cost and latency than human-only review. ## Why Prevalence Matters - User reports miss important harms because: - Some sensitive issues, such as self-harm, are under-reported. - Users seeking harmful content may not report it. - Rare policy categories provide too few reports for reliable trend detection. - Human review of reports is expensive and slow. - Prevalence measures exposure: the share of total views directed to violating content. - This helps Pinterest identify under-reported harms, evaluate interventions, and detect changes earlier. - Human-only prevalence studies were previously conducted only about every six months and required multiple reviewers plus adjudication. ## What Pinterest Measures - Daily prevalence is calculated as: - **Views of content violating a policy ÷ total views** - For example, 10 violating views in a sample of 100,000 produces an estimated prevalence of 0.01%. - Results include 95% confidence intervals to communicate statistical precision. - Metrics can be segmented by: - Policy area, such as Adult Content, Self-harm, or Graphic Violence - Sub-policy, such as nudity versus explicit sexual content - Surface, including Homefeed, Search, and Related Pins - Content age, geography, and user-age groups where relevant ## Risk-Aware, Unbiased Sampling - Pinterest samples from the daily user-impressions stream. - Production enforcement risk scores are used to prioritize likely high-risk and high-exposure content, but they are not treated as labels or eligibility rules. - Missing scores are replaced with the day’s median so that new content remains eligible. - Weighted reservoir sampling approximates probability-proportional-to-size sampling, considering impressions and risk scores. - Inverse-probability weighting removes the bias introduced by risk-based sampling, ensuring estimates represent impressions rather than model thresholds. - Pinterest uses Hansen–Hurwitz ratio estimators for sampling with replacement and Horvitz–Thompson ratio estimators for sampling without replacement. - Pure random sampling is also available for validation studies. ## LLM-Based Labeling - A multimodal LLM analyzes sampled content using both images and text. - Prompts are reviewed by policy subject-matter experts and can return structured label hierarchies such as `safe`, `not_safe`, and `unsure`. - Each decision records: - The label and brief rationale - Policy version - Prompt and model identifiers - Token usage and run cost - Human validation is performed on strategically selected samples to identify edge cases and AI blind spots. - The LLM is tested against human-reviewed gold sets before launch and periodically afterward to detect drift. - The workflow is reportedly 15 times faster and far cheaper than human-only labeling while maintaining comparable decision quality and statistical governance. ## Production System and Monitoring - Inputs include entity-by-day engagement data such as impressions, clicks, hides, and reports, alongside current production risk scores. - The system stores prevalence estimates, sampling weights, labels, diagnostics, and lineage for audits. - Dashboards display: - Daily prevalence and 95% confidence intervals - Confidence-interval width and effective sample size - Sample positive rate - Risk-score distributions - Prompt, model, taxonomy, and metric versions - Teams can pivot results by policy, sub-policy, and surface. - Validation samples and run-health information help monitor both statistical quality and operational reliability. Pinterest’s approach combines probability sampling, inverse-probability estimation, and continuously calibrated multimodal AI labeling to create a daily radar for harmful exposure. The practical recommendation is to use AI to scale measurement, but retain rigorous sampling, human validation, confidence intervals, and full model and policy lineage so that faster estimates remain trustworthy.

multimodal-aillmmachine-learningtrust-and-safety+3

pinterestDec 4, 2025

On the (re)-prioritization of open-source AI (opens in new tab)

Pinterest argues that AI competition is shifting beyond ever-larger proprietary models. Open-source models now deliver comparable quality at a fraction of the cost, while compact models fine-tuned for specific tasks can outperform general-purpose systems. The company’s strategy is to combine open-source models with Pinterest-specific data, internal systems, and deep product integration. ## Open-Source Models and Cost Efficiency - Pinterest reports achieving performance comparable to leading proprietary AI models at less than 10% of the cost. - The company is shifting more investment toward fine-tuned open-source models, especially for visual and multimodal applications. - As core LLM architectures become increasingly commoditized, competitive advantage is moving toward: - Domain-specific data - Personalization - Product integration - End-to-end system optimization ## Choosing What to Build, Buy, or Adapt Pinterest evaluates foundation-model strategy by modality: - **User modeling and recommendation** - These systems are tightly coupled to a product’s behavior and are generally built internally. - Pinterest uses long-term user-action sequences and a graph containing hundreds of billions of user, board, and content nodes. - Examples include PinFM for representation learning and PinRec for generative recommendations. - **Visual models** - Pinterest largely trains visual encoders and diffusion models in-house. - Its visual-search data and image-board collections provide the weakly supervised datasets needed for large-scale training. - Internal models benefit from Pinterest’s specialized visual domain. - **Text models** - Pinterest has historically relied more on open-source and proprietary third-party LLMs. - Progress in reasoning and language modeling depends heavily on enormous datasets and compute resources, making external models practical. ## Domain-Specific Data as the Differentiator - Open-source multimodal architectures are narrowing the capability gap with proprietary models. - Pinterest’s experience reflects an older machine-learning pattern: model architectures become broadly available, while value comes from specialized data and fine-tuning. - Its visual encoders, including UVE and PinCLIP, improved retrieval by training on Pinterest image and visual-search data rather than using generic embeddings. - Pinterest Canvas similarly adapts an internally trained diffusion model for image editing and enhancement, outperforming larger general-purpose visual-generation models in those use cases. ## Pinterest Assistant and Specialized Tools - Pinterest Assistant combines: - Multimodal retrieval systems - Recommendation services - Specialized generative models - A core multimodal LLM - Most recommendation and agentic capabilities are handled by Pinterest-native tools built on its user and visual foundation models. - The central LLM acts primarily as an intelligent router, handling query understanding, planning, and tool calling rather than performing every task itself. - This architecture allows Pinterest to improve the overall product by optimizing smaller, specialized components instead of relying solely on a larger general-purpose model. Pinterest’s recommendation is to use open-source models as adaptable building blocks, then differentiate through proprietary data, specialized models, and tight integration with the product. The most effective AI systems may therefore be smaller, cheaper, and more purpose-built than frontier general-purpose models.

multimodal-aillmgen-aifine-tuning+4