SymptomAI: Towards a conversational AI agent for everyday symptom assessment
SymptomAI explores whether conversational AI can conduct realistic symptom interviews and generate useful differential diagnoses outside curated medical vignettes. In a randomized national study of 13,917 participants, SymptomAI agents often performed as well as or better than clinician-generated differentials according to expert reviewers, particularly when they actively asked follow-up questions. The study also found that diagnoses associated with infectious illnesses corresponded with shifts in participants’ Fitbit biosignals, suggesting potential for large-scale health research. ## Moving Beyond Curated Medical Cases - Existing language-model evaluations often use detailed, synthetic, or highly structured patient vignettes. - Real patients may provide incomplete information, have varying medical literacy, or describe symptoms unpredictably during conversation. - SymptomAI was designed to test end-to-end symptom assessment in a more natural setting, while making clear that its outputs were research results rather than clinical diagnoses. ## National-Scale Study Design - 13,917 consenting participants were randomly assigned to one of five Gemini Flash 2.0 SymptomAI agents. - Participants described their symptoms, answered follow-up questions, received a differential diagnosis (DDx), and were given next-step recommendations. - Two weeks later, participants reported diagnoses received from healthcare providers. - Three board-certified clinicians reviewed the conversations, created their own differentials, and blindly ranked SymptomAI’s DDx against clinician-generated alternatives. ## SymptomAI Compared Favorably with Clinicians - Clinical reviewers preferred SymptomAI’s differential diagnosis over those from other clinicians in more than 50% of cases. - SymptomAI’s DDx was more likely to be ranked as the highest-quality option. - Using top-five accuracy—whether the eventual provider diagnosis appeared among five proposed diagnoses—reviewers found SymptomAI’s differentials accurate more often than the comparison clinician differentials. ## Follow-Up Questions Improved Accuracy - The study tested five interview strategies: - Dynamic Live and Dynamic Final agents could ask unrestricted follow-up questions. - Fixed Canonical and Flexible Canonical agents used standardized medical history questions. - The Base condition represented a user-led interaction with an unprompted language model. - Every agent-driven strategy significantly outperformed the Base condition. - The findings indicate that actively eliciting additional information is more effective than relying solely on what users initially choose to disclose. ## Strongest Results in Uncertain Cases - SymptomAI’s advantage over clinician baselines was greatest when clinicians expressed low confidence in their own differentials. - This suggests conversational AI may be especially useful as a second opinion or support tool in ambiguous cases, though the study does not establish that it can replace professional diagnosis. ## Connecting Diagnoses with Wearable Data - The researchers used SymptomAI’s diagnostic outputs as potential reference labels for analyzing population-scale physiological data. - Participants provided up to 30 days of Fitbit biometric data before their SymptomAI interaction. - Acute respiratory infection cases showed noticeable biosignal changes in the days leading up to symptom reporting. - These shifts appeared consistent with symptom onset and possible immune responses, although the provided text ends before presenting the full analysis. SymptomAI’s results support building conversational systems that ask structured follow-up questions and assist with differential diagnosis. Any practical deployment should retain clinician oversight, communicate uncertainty clearly, and treat AI-generated assessments as decision support rather than confirmed medical diagnoses.
Read original(opens in new tab)