Conversational Ai | Techlist.io

googleJul 22, 20263 min readCurated summary

SymptomAI: Towards a conversational AI agent for everyday symptom assessment

SymptomAI explores whether conversational AI can conduct realistic symptom interviews and generate useful differential diagnoses outside curated medical vignettes. In a randomized national study of 13,917 participants, SymptomAI agents often performed as well as or better than clinician-generated differentials according to expert reviewers, particularly when they actively asked follow-up questions. The study also found that diagnoses associated with infectious illnesses corresponded with shifts in participants’ Fitbit biosignals, suggesting potential for large-scale health research. ## Moving Beyond Curated Medical Cases - Existing language-model evaluations often use detailed, synthetic, or highly structured patient vignettes. - Real patients may provide incomplete information, have varying medical literacy, or describe symptoms unpredictably during conversation. - SymptomAI was designed to test end-to-end symptom assessment in a more natural setting, while making clear that its outputs were research results rather than clinical diagnoses. ## National-Scale Study Design - 13,917 consenting participants were randomly assigned to one of five Gemini Flash 2.0 SymptomAI agents. - Participants described their symptoms, answered follow-up questions, received a differential diagnosis (DDx), and were given next-step recommendations. - Two weeks later, participants reported diagnoses received from healthcare providers. - Three board-certified clinicians reviewed the conversations, created their own differentials, and blindly ranked SymptomAI’s DDx against clinician-generated alternatives. ## SymptomAI Compared Favorably with Clinicians - Clinical reviewers preferred SymptomAI’s differential diagnosis over those from other clinicians in more than 50% of cases. - SymptomAI’s DDx was more likely to be ranked as the highest-quality option. - Using top-five accuracy—whether the eventual provider diagnosis appeared among five proposed diagnoses—reviewers found SymptomAI’s differentials accurate more often than the comparison clinician differentials. ## Follow-Up Questions Improved Accuracy - The study tested five interview strategies: - Dynamic Live and Dynamic Final agents could ask unrestricted follow-up questions. - Fixed Canonical and Flexible Canonical agents used standardized medical history questions. - The Base condition represented a user-led interaction with an unprompted language model. - Every agent-driven strategy significantly outperformed the Base condition. - The findings indicate that actively eliciting additional information is more effective than relying solely on what users initially choose to disclose. ## Strongest Results in Uncertain Cases - SymptomAI’s advantage over clinician baselines was greatest when clinicians expressed low confidence in their own differentials. - This suggests conversational AI may be especially useful as a second opinion or support tool in ambiguous cases, though the study does not establish that it can replace professional diagnosis. ## Connecting Diagnoses with Wearable Data - The researchers used SymptomAI’s diagnostic outputs as potential reference labels for analyzing population-scale physiological data. - Participants provided up to 30 days of Fitbit biometric data before their SymptomAI interaction. - Acute respiratory infection cases showed noticeable biosignal changes in the days leading up to symptom reporting. - These shifts appeared consistent with symptom onset and possible immune responses, although the provided text ends before presenting the full analysis. SymptomAI’s results support building conversational systems that ask structured follow-up questions and assist with differential diagnosis. Any practical deployment should retain clinician oversight, communicate uncertainty clearly, and treat AI-generated assessments as decision support rather than confirmed medical diagnoses.

Read original

Conversational Ai Gemini Differential Diagnosis Language Models+3

tossJun 17, 20263 min readCurated summary

How AI Has Changed the Product Design Process

Toss redesigned its customer-service chatbot after finding that 60% of users abandoned the existing menu-based experience. Instead of forcing customers to identify the correct category, the team used AI to interpret natural-language requests and connect users directly to solutions. The project’s key insight was to prototype and validate experiences early, then improve the system through reusable rules rather than endlessly editing individual scenarios. ## Replacing Menu Navigation with Natural Language - The customer center serves roughly 600,000 visitors monthly, including about 170,000 chat users. - Users often know their problem—such as a failed payment or mistaken transfer—but not how Toss categorizes it. - The team therefore aimed to build a chatbot that understood intent and guided users directly to the appropriate resolution. ## Generating Scenario Drafts from Real 상담 Data - AI created initial scenarios for the 20 most common customer-service issues. - Personal information was removed and data was anonymized or pseudonymized before use. - Instead of relying primarily on policy documents, the team trained the AI on real consultation data. - This revealed: - The language customers actually use - The questions agents ask to narrow down causes - The order in which problems are diagnosed and resolved - Ways to explain complex policies in understandable language - As a result, the chatbot behaved more like a human 상담사 than a system that simply listed policies. ## Validating Many Situations with a Scenario Hub - A single request can hide dozens of possible conditions. For example, membership cancellation may depend on whether: - The monthly fee was already paid - Benefits were used - Cancellation was already scheduled - The team built a scenario hub where these conditions could be saved and selected. - Each selection immediately opened a prototype conversation with the chosen conditions applied. - This enabled rapid testing after scenario changes, new branches, or rule updates. - The prototype became an experimentation environment rather than a static mockup, allowing the team to judge realistic conversations instead of relying on assumptions. ## Replacing Individual Fixes with General Rules - Testing revealed recurring problems, including: - Asking for information the chatbot already knew - Repeating the same explanation - Guessing when it lacked reliable information - Rather than fixing each scenario separately, the team created general rules such as: - Resolve the customer’s problem before giving explanations - Admit uncertainty instead of guessing - Escalate to a human agent only under defined conditions - Clearly limit the AI’s authority - Improving one scenario helped only that scenario, while improving a rule improved many scenarios at once. ## Designing the Experience Before the System - The team first envisioned the desired customer experience, then worked backward to determine the required: - Data structures - APIs - Operational tools - System architecture - This reversed the conventional process of designing infrastructure first and fitting the user experience onto it. - Within approximately three weeks, the team completed analysis, experience design, scenario generation, prototyping, testing, and refinement. - AI changed not only productivity but also the order of product design: designers could validate more possibilities before committing to implementation. ## Practical Lessons - Build an imperfect prototype early instead of waiting for a complete design. - Study real user data before relying solely on guides or documentation. - When similar problems recur, identify a shared rule instead of repeatedly patching individual cases. The broader recommendation is to use AI as a fast experimentation tool. Its greatest value is not simply producing artifacts, but making it cheaper and faster to test which experiences and design principles actually work.

Read original

Conversational Ai Natural Language Processing Chatbot Rapid Prototyping+3

googleApr 9, 20263 min readCurated summary

ConvApparel: Measuring and bridging the realism gap in user simulators

ConvApparel addresses the “realism gap” between LLM-based user simulators and genuine human behavior. It combines over 4,000 human-AI shopping conversations with a controlled Good-versus-Bad agent setup and evaluates simulators through statistical alignment, human-likeness, and counterfactual adaptation. The framework aims to determine whether simulators genuinely model human reactions or merely reproduce patterns from their training data. ## Why User Simulator Realism Matters - Conversational agents often fail during long, multi-turn interactions by forgetting constraints or producing irrelevant responses. - Human testing provides valuable feedback but is expensive, slow, and difficult to scale. - LLM-based user simulators offer a scalable alternative, but often behave unlike real users: - They may be excessively verbose. - They can lack consistent personas or coherent preferences. - They may possess unrealistic, encyclopedic knowledge. - They are often unusually patient and assistant-like. - Training systems only against unrealistic simulators may cause them to fail with real users. ## The Need for Counterfactual Validation - A simulator should respond plausibly not only to situations represented in its training data, but also to novel assistant behaviors. - The authors introduce **counterfactual validation**: training a simulator on helpful-agent conversations, then testing it against an unexpectedly frustrating agent. - A realistic simulator should recognize poor assistance and show increased frustration, reduced satisfaction, and behavior changes similar to those of real users. - This tests whether the simulator has learned general human behavior rather than memorized training patterns. ## The ConvApparel Dataset - ConvApparel contains more than 4,000 human-AI multi-turn conversations and nearly 15,000 total turns in the apparel-shopping domain. - Participants were unknowingly assigned to one of two recommendation agents: - **Good agent:** Helpful, efficient, and supported by robust search. - **Bad agent:** Intentionally confusing, tangential, and based on degraded search retrieval. - The dataset captures reactions ranging from satisfaction to significant annoyance. - Participants provided turn-by-turn retrospective annotations, including: - Satisfaction - Frustration - Likelihood of making a purchase ## Three-Part Evaluation Framework ### Population-Level Statistical Alignment - Simulated conversations are compared with human conversations using aggregate measures such as: - Conversation length - Words per turn - Dialogue acts, including rejecting recommendations - This reveals whether simulators reproduce broad behavioral distributions. ### Human-Likeness Score - An automated discriminator is trained on human and simulated conversations. - It produces a probability indicating how human-like a conversation appears. - The score is intended to detect subtle stylistic differences that simple statistics may miss. ### Counterfactual Validation - A simulator is trained only on conversations with the Good agent. - It then interacts with the unseen Bad agent. - High-fidelity simulation should produce a human-like increase in frustration and decline in satisfaction when the assistant behaves poorly. ## Simulator Configurations The experiments compare three Gemini-based user simulators: - **Prompted simulator:** Uses high-level behavioral instructions without additional task-specific training. - **In-context learning (ICL) simulator:** Retrieves semantically similar human conversations from ConvApparel and supplies them as examples at each turn. - **Supervised fine-tuning (SFT) simulator:** Trains a Gemini 2.5 Flash model directly on the dataset. The post presents ConvApparel as a structured way to measure simulator realism and test whether simulated users can adapt to assistant behavior outside their training distribution. Its central recommendation is to evaluate user simulators not only by surface-level similarity, but also by how naturally they react to unexpected failures.

Read original

Conversational Ai Database Design Large Language Models Gemini+4

grammarlyMar 27, 20263 min readCurated summary

What Is AI Chat? Definition, How It Works, and Key Benefits

AI chat enables open-ended, context-aware conversations with systems that generate responses dynamically rather than following fixed scripts. Powered by large language models (LLMs), it supports tasks such as writing, brainstorming, learning, summarizing, planning, and coding. Its flexibility comes with limitations: responses reflect learned patterns rather than true understanding, so users should provide clear context and verify results. ## What AI Chat Is - AI chat allows users to ask questions naturally and refine requests through follow-up messages. - It can answer questions, explain complex subjects, draft and revise text, summarize documents, generate code, and provide feedback. - Unlike fixed chatbot flows, it handles unstructured requests and evolving conversations without requiring users to restart. ## How AI Chat Works - **LLM training:** Models learn language patterns from massive text datasets rather than memorizing a fixed set of answers. - **Natural language processing:** The system analyzes prompts to infer meaning, intent, tone, and context beyond exact keyword matches. - **Response generation:** The model predicts and selects text one word at a time based on the prompt and patterns learned during training. - **Conversation context:** Recent messages help the system interpret follow-up requests, such as understanding that “make it shorter” refers to a previously generated summary. - **Ongoing refinement:** Fine-tuning and human feedback improve safety, accuracy, and alignment. Models generally do not learn from individual conversations in real time. ## AI Chat Compared with Traditional Chatbots - Traditional chatbots commonly use rules, decision trees, and scripted responses. - They work well for narrow, repeatable tasks such as FAQs, appointment booking, and order tracking. - AI chat is better suited to open-ended activities including brainstorming, drafting, explanations, and problem-solving. - “Conversational AI chatbot” usually describes a chatbot interface powered by generative AI, making it more flexible than a fully rules-based system. ## Common Uses - **Writing and editing:** Draft emails, rewrite passages, adjust tone, improve clarity, and revise reports or presentations. - **Brainstorming:** Generate ideas, outlines, alternatives, and new perspectives through iterative discussion. - **Learning and planning:** Explore unfamiliar topics, simplify complex information, and develop plans. - **Coding support:** Generate code, explain technical concepts, and help troubleshoot problems. ## Effective Use - Write clear prompts and provide relevant context. - State the goal, desired format, audience, and preferences. - Use follow-up questions to refine the response. - Review outputs for factual errors, bias, and inappropriate assumptions. AI chat is most useful as a flexible assistant rather than an unquestionable authority. Use it for exploration and productivity, but verify important information and apply human judgment before relying on its output.

Read original

Conversational Ai Large Language Models Machine Learning Automation+4

grammarlyMar 26, 20263 min readCurated summary

What Is a Chatbot? Definition, Types, and Examples

Chatbots are conversational interfaces that use text or voice to answer questions, provide information, and help users complete tasks. They range from predictable rule- and keyword-based systems to flexible AI-powered tools that generate responses dynamically. Their main advantages are speed, consistency, and scalability, but flexibility and accuracy depend on how they are designed. ## What Chatbots Are - Chatbots simulate human conversation through text or voice. - They let users ask questions or make requests without navigating menus or fixed workflows. - Common applications include websites, mobile apps, messaging platforms, customer support, and help centers. - A chatbot is the user-facing interface; conversational AI provides language-understanding capabilities; and virtual assistants are broader tools that use conversation to perform tasks. ## Main Types of Chatbots ### Rule-Based Chatbots - Follow predefined decision trees and fixed conversation paths. - Commonly use buttons or menus such as “Billing” and “Technical support.” - Provide consistent, predictable responses. - Struggle with unexpected questions or requests outside their programmed workflows. ### Keyword-Based Chatbots - Detect specific words or phrases and return associated responses. - For example, the word “refund” might trigger a returns-policy link. - Allow free-text input but do not truly understand intent. - Can fail when users phrase requests differently from expected keywords. ### AI Chatbots - Use machine learning, natural language processing, and large language models to interpret requests. - Generate responses dynamically rather than selecting only from predefined answers. - Can handle loosely phrased questions, follow-up messages, complex explanations, and tone adjustments. - Responses may vary and should be checked for accuracy and relevance. ### Hybrid Chatbots - Combine structured rules with AI-generated responses. - May use menus to route common requests and AI for more complex follow-up questions. - Balance predictable task handling with conversational flexibility. ## How Chatbots Work - **Receive input:** The system captures a typed message or spoken request. - **Interpret the request:** Rule-based systems follow pathways, keyword systems match terms, and AI systems analyze intent and context. - **Generate a response:** The chatbot provides information, a next step, a predefined reply, or an AI-generated answer. - The overall process is similar across chatbot types, but the method used to interpret messages and produce responses differs significantly. ## Benefits and Limitations - Chatbots can deliver fast responses, provide consistent information, scale across many users, and automate routine interactions. - They can guide users through tasks, answer common questions, and reduce reliance on human support. - Rule- and keyword-based systems are reliable within narrow, predefined scenarios but lack flexibility. - AI chatbots handle broader conversations more naturally but may produce inaccurate or inconsistent answers. - Choosing the right chatbot type depends on whether predictability, flexibility, task automation, or open-ended conversation is most important. A practical chatbot strategy matches the technology to the task: use structured systems for predictable workflows, AI for nuanced conversations, and hybrid designs when both reliability and flexibility are needed.

Read original

Conversational Ai Machine Learning Natural Language Processing Chatbots+4

googleMar 11, 20263 min readCurated summary

Exploring the feasibility of conversational diagnostic AI in a real-world clinical study

The study evaluated Google’s conversational medical AI, AMIE, in a real-world primary care workflow rather than simulated cases. In a prospective, IRB-approved study at Beth Israel Deaconess Medical Center, AMIE conducted supervised pre-visit history-taking with 100 patients. Results suggested that supervised deployment was feasible and conversationally safe, while AMIE’s diagnostic and management-plan quality was broadly comparable to that of primary care physicians, with physicians performing better on practicality and cost effectiveness. ## Study Design and Clinical Workflow - Patients with new, non-emergency, episodic complaints used AMIE through a secure web link before an in-person or telehealth appointment. - A physician supervised each AI-patient interaction through live video and screen-sharing. - AMIE produced a transcript and summary for the patient’s primary care physician. - Independent clinical evaluators assessed: - The quality of the AMIE conversation - AMIE’s differential diagnoses - AMIE’s management plans - Comparable outputs from physicians - The study was prospective, single-center, single-arm, pre-registered, and IRB approved. ## Participants - 100 adults completed the AMIE interaction. - 98 attended their scheduled primary care appointments. - Participants represented varied ages, racial and ethnic groups, health literacy, technology literacy, and prior chatbot experience. - Compared with all 1,452 urgent care visits during the study period, participants tended to be younger, although the sample reflected the broader population’s female and white demographic skew. ## Safety Oversight - Human supervisors could stop an interaction if they observed: - Immediate risk of harm to the patient or others - Significant emotional distress related to the AI interaction - Potential clinical harm - A patient’s explicit request to end the session - No safety stops were required across the study. - The authors interpret this as evidence that supervised AMIE interactions were conversationally safe in this setting. ## Clinical Reasoning Performance - Three independent clinical evaluators reviewed each case using blinded, randomized assessments. - AMIE and physicians showed similar overall quality for: - Differential diagnoses - Management plans - Management-plan appropriateness and safety - Physicians performed better on the practicality and cost effectiveness of management plans. - AMIE’s differential-diagnosis accuracy was reported as high, including cases where the final diagnosis was confirmed through diagnostic testing. ## Patient and Clinician Experience - The study measured trust, perceptions, and acceptance among both patients and clinicians. - Patient trust in AI increased after interacting with AMIE. - Overall findings indicated that the system was well received within the supervised pre-visit workflow. The study supports cautious, supervised testing of conversational diagnostic AI in clinical environments. It does not establish that AMIE can independently replace clinicians; rather, it suggests that pre-visit information gathering may be a practical early use case, provided rigorous oversight, safety protocols, and further evaluation in larger and more diverse settings.

Read original

Conversational Ai Chatbot Telehealth Differential Diagnosis+4

googleFeb 10, 20263 min readCurated summary

Beyond one-on-one: Authoring, simulating, and testing dynamic human-AI group conversations

DialogLab is an open-source research prototype for designing, simulating, and evaluating dynamic human–AI group conversations. It addresses the tension between rigid scripts and unpredictable generative dialogue by combining structured conversational phases with real-time improvisation. Its evaluation with 14 participants suggests that human-guided simulation offers the strongest balance of realism, engagement, and control. ## A Framework for Multi-Party Conversations - DialogLab separates a conversation’s social structure from its progression over time. - **Group dynamics** define: - Groups, such as a conference or social event - Parties, such as presenters and audiences - Elements, including human or AI participants and shared content - **Conversation-flow dynamics** define: - Snippets, or distinct phases such as opening, debate, and consensus - Participants and turn sequences within each snippet - Interaction styles, including collaborative or argumentative modes - Rules for interruptions and backchanneling - This separation makes complex conversation designs modular and easier to revise. ## The Author–Test–Verify Workflow ### Authoring with Visual Tools - Designers use a drag-and-drop canvas to arrange avatars and shared content. - Inspector panels configure personas, roles, interaction patterns, and snippet behavior. - Automatically generated prompts can be customized for specific narrative or conversational goals. ### Human-in-the-Loop Simulation - A live preview displays the evolving transcript. - In human-control mode, an audit panel suggests possible AI responses. - Designers can edit, accept, or reject suggestions, retaining control over the agents’ contributions. - The system supports both structured interactions and more improvisational conversations. ### Verification and Analytics - A verification dashboard provides post-hoc analysis of the conversation. - Visualizations show turn-taking distributions and sentiment flows. - These tools help creators diagnose interaction patterns without manually reviewing entire transcripts. ## Prototype Evaluation - Fourteen participants from game design, education, and social science research evaluated DialogLab. - They designed an academic social event and tested AI group discussions under three conditions: - **Human control:** Users prompted agents to shift topics, introduce perspectives, ask probing questions, or generate emotional responses. - **Autonomous:** Agents participated proactively according to predefined random or sequential orders. - **Reactive:** A simulated human agent responded only when directly addressed. - Human control was rated significantly more engaging and was generally considered more effective and realistic. - Participants also described the interface as intuitive, flexible, and enjoyable. - Users valued the combination of automated prompt generation, detailed customization, and support for different moderation strategies. DialogLab demonstrates that effective multi-party conversational design benefits from combining explicit structure with controlled improvisation. For developers and researchers building group-based human–AI experiences, a visual authoring workflow paired with human-guided simulation and analytics can provide a practical foundation for rapid iteration and more realistic interactions.

Read original

Conversational Ai Large Language Models Generative AI Multi Party Dialogue+3

googleFeb 3, 20263 min readCurated summary

Collaborating on a nationwide randomized study of AI in real-world virtual care

Google and Included Health plan to launch, pending IRB approval, a nationwide randomized study of conversational AI in real-world virtual care. Unlike prior simulated or small feasibility studies, it will prospectively evaluate AI with consented patients across varied conditions and locations, comparing it with standard clinical practice. The goal is to generate rigorous evidence about safety, usefulness, limitations, and impact on patients and clinicians. ## Moving from Simulation to Real-World Evaluation - Earlier research demonstrated clinician-level capabilities in simulated consultations and retrospective analyses. - A feasibility study with Beth Israel Deaconess Medical Center began testing conversational AI in clinical workflows, using measures such as safety-supervisor interruptions. - The new study will advance beyond feasibility through: - A randomized controlled design - Nationwide recruitment - Consented participants - Real patients, clinical concerns, and virtual-care workflows - Controlled comparison with standard practice ## A Phased Approach to Medical AI Research - Google argues that medical AI should be evaluated with evidence standards similar to other medical interventions. - Each research phase adds information about: - Patient and clinician experiences - Safety - Usefulness - The AI system’s capabilities and limitations - Results from each stage are intended to guide safer, more responsible development and deployment. ## Foundational Research Behind the Study ### Diagnostic and Management Reasoning - The AMIE system was developed to handle medical interviews and clinical reasoning. - Studies with patient actors and synthetic cases found that AMIE could match or exceed primary care physicians in simulated diagnostic accuracy and conversation quality. - Later work expanded the system to: - Longitudinal disease management - Clinical-guideline and patient-history reasoning - Investigation and treatment planning - Interpretation of multimodal evidence ### Personalized Health Insights - Research on the Personal Health Agent examined how AI could interpret personal health data, including sleep and activity information from wearables. - Its multi-agent architecture combined the roles of: - Data scientist - Medical domain expert - Health coach - This work informed Fitbit Labs tools such as Symptom Checker and Medical Records Navigator and Plan for Care. ### Navigating Health Information - Google’s “wayfinding” AI research explored how conversational agents can help people find and understand health information. - The system uses proactive guidance, goal recognition, and tailored conversations to make health information searches more practical and useful. ## Practical Conclusion The partnership with Included Health represents a transition from demonstrating what medical AI can do in controlled environments to measuring how it performs at scale in actual care. A nationwide randomized trial could provide the evidence needed to determine whether conversational AI can safely improve virtual care and expand access to medical expertise.

Read original

Conversational Ai Healthcare Ai Diagnostic Reasoning Multimodal Models+3

dropboxOct 2, 20253 min readCurated summary

A practical blueprint for evaluating conversational AI at scale

Conversational AI systems depend on many probabilistic stages, so even small changes can cause unexpected regressions. Dropbox Dash’s experience shows that evaluation should be treated like production engineering: systematic, repeatable, and required before changes are shipped. The approach combines curated datasets, actionable metrics, LLM-based judging, and human review. ## Evaluation as a Development Discipline - AI pipelines include intent classification, retrieval, ranking, prompt construction, inference, and safety filtering. - Changes to any stage can affect final answer quality in unpredictable ways. - Dropbox initially used ad-hoc testing, but shifted to a standardized process in which every model, prompt, or retrieval change had to pass evaluation before merging. - The evaluation framework covers datasets, metrics, tools, and workflows. - Future-proof evaluation must extend beyond text to images, video, and audio. ## Curating Public and Internal Datasets - Public datasets established baseline retrieval and question-answering performance: - **Natural Questions** tested retrieval from very large documents. - **MS MARCO** emphasized queries requiring multiple document hits. - **MuSiQue** tested multi-hop reasoning. - Internal datasets captured real-world usage from anonymized Dropbox employee queries and content. - Representative query sets reflected actual user behavior, using proxy labels or internal annotators. - Representative content sets focused on shared files, documentation, and connected data sources. - LLM-generated synthetic questions and answers covered tables, images, tutorials, and factual lookups. - These datasets became the foundation for automated pass/fail checks and experiment monitoring. ## Why Traditional Metrics Fall Short - Metrics such as BLEU, ROUGE, METEOR, BERTScore, and embedding similarity are fast and useful for detecting major regressions. - They often fail to measure production requirements, including: - Correct source citations - Factual accuracy - Valid file names and references - Reliable table parsing - Clear formatting - High ROUGE or BERTScore can coexist with hallucinations, missing citations, or buried factual errors. - Production evaluation therefore requires metrics tied directly to task requirements. ## Using LLMs as Evaluation Judges - LLM judges can assess dimensions traditional metrics miss, including: - Whether an answer addresses the query - Whether claims are supported by context - Citation correctness - Clarity, formatting, and tone - Judges receive the query, model answer, source context, and sometimes a hidden reference answer. - They return structured scores—scalar or categorical—alongside explanations. - Rubrics and judge models must themselves be tested, calibrated, versioned, and improved. - Specialized judges may be needed for particular languages or technical domains. ## Calibration and Human Review - Periodic manual labeling of sampled outputs created calibration sets for comparing human and judge-model decisions. - The team tracked agreement and judge drift over time. - Engineers manually reviewed 5–10% of each release’s regression suite. - Disagreements were investigated as either judge-prompt defects or model errors. - Recurring problems led to prompt revisions or more detailed scoring criteria. Dropbox’s evaluation-first approach treats AI changes like code changes: test them against realistic datasets, score them with task-specific rubrics, validate the evaluators, and retain human oversight. This makes conversational AI development more reliable as systems and modalities grow more complex.

Read original

Conversational Ai Large Language Models Natural Language Processing Prompt Engineering+4

googleSep 24, 2025Original article

Towards better health conversations: Research insights on a “wayfinding” AI agent based on Gemini (opens in new tab)

Google Research has developed "Wayfinding AI," a research prototype based on Gemini designed to transform health information seeking from a passive query-response model into a proactive, context-seeking dialogue. By prioritizing clarifying questions and iterative guidance, the agent addresses the common struggle users face when attempting to articulate complex or ambiguous medical concerns. User studies indicate that this proactive approach results in health information that participants find significantly more helpful, relevant, and tailored to their specific needs than traditional AI responses. ### Challenges in Digital Health Navigation * Formative research involving 33 participants highlighted that users often struggle to articulate health concerns because they lack the clinical background to know which details are medically relevant. * The study found that users typically "throw words" at a search engine and sift through generic, impersonal results that do not account for their unique context. * Initial UX testing revealed a strong user preference for a "deferred-answer" approach, where the AI mimics a medical professional by asking clarifying questions before jumping to a conclusion. ### Core Design Principles of Wayfinding AI * **Proactive Conversational Guidance:** At every turn, the agent asks up to three targeted questions to reduce ambiguity and help users systematically share their "health story." * **Best-Effort Answers:** To ensure immediate utility, the AI provides the best possible information based on the data available at that moment, while noting that the answer will improve as the user provides more context. * **Transparent Reasoning:** The system explicitly explains how the user’s most recent answers have helped refine the previous response, making the AI’s internal logic understandable. ### Split-Stream User Interface * To prevent clarifying questions from being buried in long paragraphs, the prototype uses a two-column layout. * The left column is dedicated to the interactive chat and specific follow-up questions to keep the user focused on the dialogue. * The right column displays the "best information so far" and detailed explanations, allowing users to dive into the technical content only when they feel enough context has been established. ### Comparative Evaluation and Performance * A randomized study with 130 participants compared the Wayfinding AI against a baseline Gemini 2.5 Flash model. * Participants interacted with both models for at least three minutes regarding a personal health question and rated them across six dimensions: helpfulness, question relevance, tailoring, goal understanding, ease of use, and efficiency. * The proactive agent outperformed the baseline significantly, with participants reporting that the context-seeking behavior felt more professional and increased their confidence in the AI's suggestions. The research suggests that for sensitive and complex topics like health, AI should move beyond being a passive knowledge base. By adopting a "wayfinding" strategy that guides users through their own information needs, AI agents can provide more personalized and empowering experiences that better mirror expert human consultation.

Conversational Ai AI Large Language Models AI Agents+4

googleJun 26, 2025Original article

REGEN: Empowering personalized recommendations with natural language (opens in new tab)

Google Research has introduced REGEN, a benchmark dataset designed to evolve recommender systems from simple item predictors into conversational agents capable of natural language interaction. By augmenting the Amazon Product Reviews dataset with synthetic critiques and narratives using Gemini 1.5 Flash, the researchers provide a framework for training models to understand user feedback and explain their suggestions. The study demonstrates that integrating natural language critiques significantly improves recommendation accuracy while enabling models to generate personalized, context-aware content. ### Composition of the REGEN Dataset * The dataset enriches the existing Amazon Product Reviews archive by adding synthetic conversational elements, specifically targeting the gap in datasets that support natural language feedback. * **Critiques** are generated for similar item pairs within hierarchical categories, allowing users to guide the system by requesting specific changes, such as a different color or increased storage. * **Narratives** provide contextual depth through purchase reasons, product endorsements, and concise user summaries, helping the system justify its recommendations to the end-user. ### Unified Generative Modeling Approaches * The researchers framed a "jointly generative" task where models must process a purchase history and optional critique to output both a recommended item ID and a supporting narrative. * The **FLARE (Hybrid)** architecture uses a sequential recommender for item prediction based on collaborative filtering, which then feeds into a Gemma 2B LLM to generate the final text narrative. * The **LUMEN (Unified)** model functions as an end-to-end system where item IDs and text tokens are integrated into a single vocabulary, allowing one LLM to handle critiques, recommendations, and narratives simultaneously. ### Performance and Impact of User Feedback * Incorporating natural language critiques consistently improved recommendation metrics across different architectures, demonstrating that language-guided refinement is a powerful tool for accuracy. * In the Office domain, the FLARE hybrid model's Recall@10—a measure of how often the desired item appears in the top 10 results—increased from 0.124 to 0.1402 when critiques were included. * Results indicate that models trained on REGEN can achieve performance comparable to state-of-the-art specialized recommenders while maintaining high-quality natural language generation. The REGEN dataset and the accompanying LUMEN architecture provide a path forward for building more transparent and interactive AI assistants. For developers and researchers, utilizing these conversational benchmarks is essential for moving beyond "black box" recommendations toward systems that can explain their logic and adapt to specific user preferences in real time.

Conversational Ai AI Large Language Models Gemini+4

figmaMay 4, 20253 min readCurated summary

How Headspace Built an AI Companion that Fosters Trust and Transparency | Figma Blog

Headspace built Ebb as an AI companion for reflection between therapy sessions—not as a replacement for human care. Because mental-health AI carries significant safety and trust risks, the team prioritized clinical grounding, transparency, user autonomy, and careful cross-functional design. Early alignment workshops, iterative prototyping, and explicit interface guidelines helped shape Ebb into a friendly but clearly non-human companion. ## Defining Ebb’s Role - Headspace created Ebb in response to growing use of general-purpose AI for emotional support. - The companion was intended to: - Support reflective practices between therapy sessions. - Help people who may be unable to access or afford therapy. - Complement, rather than replace, human care. - The team first clarified the complete user experience and business objectives through FigJam workshops. - This early alignment helped address internal concerns and ambiguity around using AI in mental health. ## Building a Non-Human, Approachable Identity - The team explored names including Odom, Ibo, and Scribe before choosing **Ebb**. - The name suggests the fluidity and changing nature of emotions. - Designers avoided a gendered human name to reduce stereotypes that associate caregiving with women. - Ebb was designed as a friendly, human-adjacent entity without a specific gender. - Brand, product, illustration, animation, and copy teams used FigJam “playgrounds” to explore how Ebb could look and sound. ## Iterative Collaboration and Prototyping - Headspace followed a “build-to-learn” approach, bringing brand and product teams together early. - The teams tested approximately six brand identities before selecting a direction. - Figma allowed designers to apply different visual treatments directly to product screens. - Sharing prototypes in one workspace kept teams connected and enabled rapid iteration. - The team deliberately stress-tested ideas before rejecting them. ## Designing for Trust and Safety - Ebb was trained with input from clinical psychologists, providing a strong scientific foundation. - Designers focused on reducing adoption barriers by helping users understand: - That Ebb is an AI system. - How it can support them. - How their information and conversations are handled. - The interface was designed to make users feel safe expressing themselves. - Users retain agency to exit and delete conversations at any time. - A central principle was that AI should never be invisible: members should always know whether they are interacting with AI or a human. - The team’s broader guidelines emphasized differentiating AI from human-delivered care, reinforcing privacy and safety, supporting member choice, and creating a reflective environment. Headspace’s approach suggests that mental-health AI should be designed transparently and collaboratively, with safety and user control treated as foundational product requirements rather than features added later.

Read original

Conversational Ai Figma AI User Experience+2

figmaJun 25, 20243 min readCurated summary

What Would You Ask If No One Could Judge You? | Figma Blog

Perplexity’s founders envision it as an “answer engine” that turns web-scale information into concise, sourced explanations rather than lists of links. The product grew from a personal need for judgment-free learning and was shaped by the shortcomings of early conversational AI, especially outdated knowledge and hallucinations. Its broader goal is to make curiosity easier to express and pursue. ## Building a Judgment-Free Knowledge Tool - Aravind Srinivas was inspired by childhood “Wikipedia rabbit holes” and the evolution from printed encyclopedias to AI-powered knowledge tools. - Perplexity aims to make learning engaging through curiosity rather than attention-grabbing entertainment. - The company wants users to ask anything without worrying about appearing uninformed or being judged. ## From Private Slackbot to Public Product - The founders initially built a Slackbot to answer practical questions about fundraising, employee health insurance, and running a company. - They hesitated to launch because they feared criticism for attempting to compete with Google. - Investor Nat Friedman encouraged them to view the effort as an asymmetric bet: little downside, but potentially enormous upside. - Perplexity launched shortly after ChatGPT, despite the founders having no previous company-building experience. ## An Answer Engine with Sources - ChatGPT highlighted problems with knowledge cutoffs, hallucinations, and unsupported answers. - Perplexity responded by combining: - Natural-language interaction - Web search and indexing - Large language models - Inline sources and footnotes - Its goal is to provide a direct answer while allowing users to verify the underlying information. - Srinivas describes the product as a combination of Wikipedia and conversational chat, with information drawn from across the internet. ## Making Complex Information Approachable - Perplexity follows an 80/20 approach: identify the most important concepts and deliver most of the useful understanding quickly. - It synthesizes information from multiple web pages into a concise explanation instead of requiring users to read extensively. - The product aims to simplify information without reducing it to misleading or overly shallow conclusions. ## Turning Answers into Further Curiosity - Each response includes three related follow-up questions to encourage exploration. - Srinivas argues that people are naturally curious but often lack the confidence, vocabulary, or precision to formulate good questions. - Perplexity’s design assumes that the user is never wrong; the system should help clarify and develop a person’s curiosity rather than blame them for asking imperfectly. Perplexity’s central recommendation is implicit in its design: make knowledge easier to access, verify, and explore, while removing the social fear that prevents people from asking questions in the first place.

Read original

Conversational Ai AI Large Language Models Natural Language Processing+2