techlist.io
KREN
pinterest

How Pinterest Built a Real‑Time Radar for Violative Content using AI (opens in new tab)

llmmachine-learningmultimodal-aitrust-and-safetycontent-moderationstatistical-modelingsampling-algorithms

Pinterest built an AI-assisted prevalence measurement system to estimate how often users actually see policy-violating content, rather than relying only on user reports. The system samples daily impressions, uses production risk scores to improve efficiency, labels content with a multimodal LLM, and applies statistical reweighting to preserve unbiased estimates. This enables daily, segmented monitoring with substantially lower cost and latency than human-only review.

Why Prevalence Matters

  • User reports miss important harms because:
    • Some sensitive issues, such as self-harm, are under-reported.
    • Users seeking harmful content may not report it.
    • Rare policy categories provide too few reports for reliable trend detection.
    • Human review of reports is expensive and slow.
  • Prevalence measures exposure: the share of total views directed to violating content.
  • This helps Pinterest identify under-reported harms, evaluate interventions, and detect changes earlier.
  • Human-only prevalence studies were previously conducted only about every six months and required multiple reviewers plus adjudication.

What Pinterest Measures

  • Daily prevalence is calculated as:

    • Views of content violating a policy ÷ total views

  • For example, 10 violating views in a sample of 100,000 produces an estimated prevalence of 0.01%.

  • Results include 95% confidence intervals to communicate statistical precision.

  • Metrics can be segmented by:

    • Policy area, such as Adult Content, Self-harm, or Graphic Violence
    • Sub-policy, such as nudity versus explicit sexual content
    • Surface, including Homefeed, Search, and Related Pins
    • Content age, geography, and user-age groups where relevant

Risk-Aware, Unbiased Sampling

  • Pinterest samples from the daily user-impressions stream.
  • Production enforcement risk scores are used to prioritize likely high-risk and high-exposure content, but they are not treated as labels or eligibility rules.
  • Missing scores are replaced with the day’s median so that new content remains eligible.
  • Weighted reservoir sampling approximates probability-proportional-to-size sampling, considering impressions and risk scores.
  • Inverse-probability weighting removes the bias introduced by risk-based sampling, ensuring estimates represent impressions rather than model thresholds.
  • Pinterest uses Hansen–Hurwitz ratio estimators for sampling with replacement and Horvitz–Thompson ratio estimators for sampling without replacement.
  • Pure random sampling is also available for validation studies.

LLM-Based Labeling

  • A multimodal LLM analyzes sampled content using both images and text.
  • Prompts are reviewed by policy subject-matter experts and can return structured label hierarchies such as safe, not_safe, and unsure.
  • Each decision records:
    • The label and brief rationale
    • Policy version
    • Prompt and model identifiers
    • Token usage and run cost
  • Human validation is performed on strategically selected samples to identify edge cases and AI blind spots.
  • The LLM is tested against human-reviewed gold sets before launch and periodically afterward to detect drift.
  • The workflow is reportedly 15 times faster and far cheaper than human-only labeling while maintaining comparable decision quality and statistical governance.

Production System and Monitoring

  • Inputs include entity-by-day engagement data such as impressions, clicks, hides, and reports, alongside current production risk scores.
  • The system stores prevalence estimates, sampling weights, labels, diagnostics, and lineage for audits.
  • Dashboards display:
    • Daily prevalence and 95% confidence intervals
    • Confidence-interval width and effective sample size
    • Sample positive rate
    • Risk-score distributions
    • Prompt, model, taxonomy, and metric versions
  • Teams can pivot results by policy, sub-policy, and surface.
  • Validation samples and run-health information help monitor both statistical quality and operational reliability.

Pinterest’s approach combines probability sampling, inverse-probability estimation, and continuously calibrated multimodal AI labeling to create a daily radar for harmful exposure. The practical recommendation is to use AI to scale measurement, but retain rigorous sampling, human validation, confidence intervals, and full model and policy lineage so that faster estimates remain trustworthy.