Improving Quality of Recommended Content through Pinner Surveys

Pinterest uses Pinner surveys to measure visual quality and incorporate user preferences into recommendation systems, rather than optimizing solely for engagement. The company surveyed 5,000 Pins, trained a lightweight neural network to predict average perceived quality, and applied the resulting model across Homefeed, Related Pins, and Search. This approach aims to reduce clickbait and promote content that supports positive, long-term user experiences.

Why Engagement Alone Is Insufficient

• High engagement does not necessarily indicate high-quality content; optimizing for clicks can promote clickbait or harmful material.
• Pinterest defines quality as content that feels good, inspires further exploration, and encourages fulfilling long-term engagement.
• Direct user feedback helps recommendation systems prioritize content that Pinners actually value.
• The work supports Pinterest’s Inspired Internet Pledge principles, especially listening to users and tuning the platform for wellbeing.

Collecting Pinner Quality Ratings

• Pinners rated images from 1 to 5 in response to: “How visually pleasing or displeasing is this Pin?”
• Pinterest collected ratings for 5,000 Pins, sampling 1,000 from each of five major interest categories:
  • Art
  • Beauty
  • DIY & Crafts
  • Home Decor
  • Women’s Fashion
• Pins were sampled based on impressions and were generally mid-to-high quality rather than deliberately exposing users to poor content.
• Each image received at least 10 ratings, allowing Pinterest to average responses and reduce noise from subjectivity or accidental misclicks.
• Surveys were considered appropriate for visual appeal, which is subjective but still measurable across many users. More objective issues should be evaluated by trained reviewers, while highly contextual judgments such as personal relevance are harder to capture with a single Pin-level score.
• Highly rated content included makeup, grooming styles, maximalist interiors, landscapes, sunsets, and baby animals.
• Home Decor images tended to receive higher ratings overall, while Art showed the greatest variation, reflecting its subjective nature.

Training a Visual-Quality Model

• Pinterest trained a model to estimate the average Pinner’s perception of visual quality from image embeddings.
• Embeddings encode visual, textual, and behavioral information, including relationships between images and the boards where they are saved.
• The model produces a score from 0 to 1, with higher values representing greater perceived quality.
• Pinterest chose a small fully connected neural network with approximately 92,000 parameters:
  • The limited size helps prevent overfitting to the 5,000-image dataset.
  • It also makes large-scale inference faster and less expensive.
• Instead of predicting an exact rating, the model uses pairwise ranking:
  • It learns which of two images Pinners would consider better.
  • The comparison is based on each image’s mean survey rating.
• Training comparisons are restricted to images within the same top-level interest category, encouraging the model to learn visual quality rather than simply recognizing that one topic is more popular than another.

Pinterest’s approach demonstrates how survey-based quality signals can complement engagement metrics. Training recommendation systems on what users perceive as appealing can help the platform promote more satisfying content while reducing incentives to favor attention-grabbing but low-quality material.