ConvApparel: Measuring and bridging the realism gap in user simulators
ConvApparel addresses the “realism gap” between LLM-based user simulators and genuine human behavior. It combines over 4,000 human-AI shopping conversations with a controlled Good-versus-Bad agent setup and evaluates simulators through statistical alignment, human-likeness, and counterfactual adaptation. The framework aims to determine whether simulators genuinely model human reactions or merely reproduce patterns from their training data. ## Why User Simulator Realism Matters - Conversational agents often fail during long, multi-turn interactions by forgetting constraints or producing irrelevant responses. - Human testing provides valuable feedback but is expensive, slow, and difficult to scale. - LLM-based user simulators offer a scalable alternative, but often behave unlike real users: - They may be excessively verbose. - They can lack consistent personas or coherent preferences. - They may possess unrealistic, encyclopedic knowledge. - They are often unusually patient and assistant-like. - Training systems only against unrealistic simulators may cause them to fail with real users. ## The Need for Counterfactual Validation - A simulator should respond plausibly not only to situations represented in its training data, but also to novel assistant behaviors. - The authors introduce **counterfactual validation**: training a simulator on helpful-agent conversations, then testing it against an unexpectedly frustrating agent. - A realistic simulator should recognize poor assistance and show increased frustration, reduced satisfaction, and behavior changes similar to those of real users. - This tests whether the simulator has learned general human behavior rather than memorized training patterns. ## The ConvApparel Dataset - ConvApparel contains more than 4,000 human-AI multi-turn conversations and nearly 15,000 total turns in the apparel-shopping domain. - Participants were unknowingly assigned to one of two recommendation agents: - **Good agent:** Helpful, efficient, and supported by robust search. - **Bad agent:** Intentionally confusing, tangential, and based on degraded search retrieval. - The dataset captures reactions ranging from satisfaction to significant annoyance. - Participants provided turn-by-turn retrospective annotations, including: - Satisfaction - Frustration - Likelihood of making a purchase ## Three-Part Evaluation Framework ### Population-Level Statistical Alignment - Simulated conversations are compared with human conversations using aggregate measures such as: - Conversation length - Words per turn - Dialogue acts, including rejecting recommendations - This reveals whether simulators reproduce broad behavioral distributions. ### Human-Likeness Score - An automated discriminator is trained on human and simulated conversations. - It produces a probability indicating how human-like a conversation appears. - The score is intended to detect subtle stylistic differences that simple statistics may miss. ### Counterfactual Validation - A simulator is trained only on conversations with the Good agent. - It then interacts with the unseen Bad agent. - High-fidelity simulation should produce a human-like increase in frustration and decline in satisfaction when the assistant behaves poorly. ## Simulator Configurations The experiments compare three Gemini-based user simulators: - **Prompted simulator:** Uses high-level behavioral instructions without additional task-specific training. - **In-context learning (ICL) simulator:** Retrieves semantically similar human conversations from ConvApparel and supplies them as examples at each turn. - **Supervised fine-tuning (SFT) simulator:** Trains a Gemini 2.5 Flash model directly on the dataset. The post presents ConvApparel as a structured way to measure simulator realism and test whether simulated users can adapt to assistant behavior outside their training distribution. Its central recommendation is to evaluate user simulators not only by surface-level similarity, but also by how naturally they react to unexpected failures.
Read original(opens in new tab)