google A collaborative approach to image generation (opens in new tab)
Google Research has introduced PASTA (Preference Adaptive and Sequential Text-to-image Agent), a reinforcement learning agent designed to transform image generation from a single-prompt task into a collaborative, multi-turn dialogue. By learning individual user preferences through sequential interactions, the system eliminates the frustration of trial-and-error prompting to achieve a specific creative vision. ## Data Strategy and User Simulation * Researchers collected a foundational dataset featuring over 7,000 human interactions, using Gemini Flash for prompt expansion and Stable Diffusion XL (SDXL) for image generation. * To overcome the scarcity of real-world interaction data, the team developed a user simulator that generated over 30,000 additional interaction trajectories. * The simulator is built on two primary components: a utility model that predicts how much a user will like an image, and a choice model that predicts which image a user will select from a given set. ## Latent Preference Discovery * The architecture utilizes pre-trained CLIP encoders paired with user-specific components to capture nuanced aesthetic tastes. * An expectation-maximization (EM) algorithm is employed to identify "user types," allowing the system to cluster users with similar interests, such as a preference for specific artistic styles or subject matter like "Food" or "Animals." * This approach enables the model to generalize preferences quickly, allowing it to adapt to new users based on minimal initial feedback. ## The Collaborative Generation Loop * PASTA operates as a value-based reinforcement learning model that aims to maximize cumulative user satisfaction across an entire interaction session. * The workflow begins with a candidate generator creating diverse prompt expansions; a candidate selector then picks an optimal "slate" of four variations to present to the user. * Each user selection provides a feedback signal that guides the agent’s next set of suggestions, iteratively narrowing the gap between the generated output and the user's intent. ## Training and Performance Validation * The agent was trained using Implicit Q-learning (IQL) to optimize decision-making without requiring online interaction during the training phase. * Performance was measured using several metrics, including Pick-a-Pic accuracy, Spearman’s rank correlation, and cross-turn accuracy. * Results indicated that agents trained on a combination of real-world and simulated data significantly outperformed baseline models and versions trained on only one data type. PASTA demonstrates that integrating iterative feedback loops and reinforcement learning can effectively bridge the "intent gap" in generative AI. For developers building creative tools, this research suggests that move-away from static prompting toward adaptive, simulation-trained agents can provide a more satisfying and intuitive user experience.