Gemini | Techlist.io

googleAug 11, 20263 min readCurated summary

Advancing AMIE towards expert-level audio-visual clinical consultations

AMIE (Video) is Google’s real-time audiovisual medical AI system, designed to overcome the limitations of text-only clinical conversations. Built on Gemini and Project Astra, it observes visual and auditory cues, guides patients through virtual examinations, and performs diagnostic reasoning during live consultations. In a randomized study involving 300 simulated consultations, the system was evaluated against text-only AMIE and board-certified primary care physicians. ## Why Audio-Visual Consultation Matters - Traditional text-based systems lose important clinical information, including: - Gait and visible physical symptoms - Breathing patterns and signs of distress - Vocal and auditory cues - Patient responses during physical examination maneuvers - Requiring patients to describe symptoms in writing can reduce diagnostic accuracy, particularly for people with limited digital or health literacy. - Audiovisual interaction may also improve trust, communication, and access to medical expertise. ## AMIE’s Broader Development - Earlier versions of AMIE demonstrated expert-level performance in: - Text-based diagnostic dialogue - Differential diagnosis support - Disease treatment and longitudinal management - Specialist evaluations in oncology, cardiology, and ophthalmology - Reasoning over medical images and clinical documents - Google has also explored physician oversight and real-world clinical feasibility studies. ## Asynchronous Multi-Agent Architecture AMIE (Video) divides the consultation among three agents operating in parallel: - **Talker agent** - Maintains natural, low-latency spoken conversation. - Incorporates information and recommendations from the other agents. - **Planner agent** - Performs deeper clinical reasoning in the background. - Updates differential diagnoses and management plans. - Identifies missing information and reprioritizes clinical objectives. - **Perception agent** - Continuously analyzes audio and video. - Detects non-verbal findings such as visible distress, physical signs, and auditory abnormalities. - Interprets observations in the context of the conversation. This separation allows AMIE to reason deeply without creating long conversational pauses. Automated tests indicated that the agents contributed to improvements in history-taking, clinical reasoning, treatment recommendations, communication quality, and response latency. ## Automated Evaluation Framework - Google created a taxonomy of audiovisual clinical competencies based on medical literature. - The taxonomy covered: - Non-verbal visual cues - Auditory signals - Physical examination maneuvers - The evaluation suite included: - Single-turn tests targeting specific perception and reasoning abilities - Multi-turn simulated consultations assessing complete conversational performance - Simulations injected visual findings as textual descriptions, such as a patient holding handwriting samples up to the camera. - These tests helped identify capabilities and failure modes before human evaluation. ## Randomized Video Study - The study used a synchronous video consultation interface and an Objective Structured Clinical Examination format. - It included: - 100 clinical scenarios - Five body systems: cardiopulmonary, abdominal, HEENT, neurological/psychiatric, and musculoskeletal - 15 trained patient actors - 300 standardized consultations - Three study arms were compared: - **AMIE (Video):** Real-time audiovisual consultations - **AMIE (Text):** Text-only AMIE used to isolate the value of audiovisual capabilities - **PCP (Video):** Board-certified primary care physicians using the same video interface - An independent panel of 20 experienced primary care physicians assessed the consultations using established clinical rubrics. AMIE (Video) represents a move from text-based medical dialogue toward interactive, multimodal consultations. Its multi-agent design and audiovisual perception are intended to preserve conversational responsiveness while supporting richer clinical reasoning, though the reported findings come from simulated consultations and require further validation in real-world clinical care.

Read original

Gemini Multimodal AI Telehealth Multi Agent Architecture+3

googleJul 22, 20263 min readCurated summary

SymptomAI: Towards a conversational AI agent for everyday symptom assessment

SymptomAI explores whether conversational AI can conduct realistic symptom interviews and generate useful differential diagnoses outside curated medical vignettes. In a randomized national study of 13,917 participants, SymptomAI agents often performed as well as or better than clinician-generated differentials according to expert reviewers, particularly when they actively asked follow-up questions. The study also found that diagnoses associated with infectious illnesses corresponded with shifts in participants’ Fitbit biosignals, suggesting potential for large-scale health research. ## Moving Beyond Curated Medical Cases - Existing language-model evaluations often use detailed, synthetic, or highly structured patient vignettes. - Real patients may provide incomplete information, have varying medical literacy, or describe symptoms unpredictably during conversation. - SymptomAI was designed to test end-to-end symptom assessment in a more natural setting, while making clear that its outputs were research results rather than clinical diagnoses. ## National-Scale Study Design - 13,917 consenting participants were randomly assigned to one of five Gemini Flash 2.0 SymptomAI agents. - Participants described their symptoms, answered follow-up questions, received a differential diagnosis (DDx), and were given next-step recommendations. - Two weeks later, participants reported diagnoses received from healthcare providers. - Three board-certified clinicians reviewed the conversations, created their own differentials, and blindly ranked SymptomAI’s DDx against clinician-generated alternatives. ## SymptomAI Compared Favorably with Clinicians - Clinical reviewers preferred SymptomAI’s differential diagnosis over those from other clinicians in more than 50% of cases. - SymptomAI’s DDx was more likely to be ranked as the highest-quality option. - Using top-five accuracy—whether the eventual provider diagnosis appeared among five proposed diagnoses—reviewers found SymptomAI’s differentials accurate more often than the comparison clinician differentials. ## Follow-Up Questions Improved Accuracy - The study tested five interview strategies: - Dynamic Live and Dynamic Final agents could ask unrestricted follow-up questions. - Fixed Canonical and Flexible Canonical agents used standardized medical history questions. - The Base condition represented a user-led interaction with an unprompted language model. - Every agent-driven strategy significantly outperformed the Base condition. - The findings indicate that actively eliciting additional information is more effective than relying solely on what users initially choose to disclose. ## Strongest Results in Uncertain Cases - SymptomAI’s advantage over clinician baselines was greatest when clinicians expressed low confidence in their own differentials. - This suggests conversational AI may be especially useful as a second opinion or support tool in ambiguous cases, though the study does not establish that it can replace professional diagnosis. ## Connecting Diagnoses with Wearable Data - The researchers used SymptomAI’s diagnostic outputs as potential reference labels for analyzing population-scale physiological data. - Participants provided up to 30 days of Fitbit biometric data before their SymptomAI interaction. - Acute respiratory infection cases showed noticeable biosignal changes in the days leading up to symptom reporting. - These shifts appeared consistent with symptom onset and possible immune responses, although the provided text ends before presenting the full analysis. SymptomAI’s results support building conversational systems that ask structured follow-up questions and assist with differential diagnosis. Any practical deployment should retain clinician oversight, communicate uncertainty clearly, and treat AI-generated assessments as decision support rather than confirmed medical diagnoses.

Read original

Gemini Conversational Ai Differential Diagnosis Language Models+3

gitlabJun 10, 20262 min readCurated summary

GitLab on Google Cloud: Fully managed, compliant, and AI-ready

GitLab is introducing a fully managed deployment on Google Cloud through certified managed service providers such as Beyond and Digital Future. The offering combines data residency and compliance controls with access to Google’s Gemini and Gemma models through GitLab Duo Agent Platform. Organizations can also purchase the platform through Google Cloud Marketplace, applying existing cloud commitments to GitLab, AI inference, and infrastructure costs. ## Fully Managed GitLab on Google Cloud - Certified MSPs operate GitLab on Google Cloud under service-level agreements, removing infrastructure-management responsibilities from customer teams. - Organizations retain control over where code, pipelines, and security data are stored, supporting sovereignty and data-residency requirements. - GitLab’s audit and policy controls provide visibility into agent actions, merge requests, and security findings. ## AI Model Choice for Different Workloads - Gemini models, including Gemini 3.5 Flash, are available in Duo Agent Platform through Google’s Gemini Enterprise Agent Platform. - GitLab’s participation in Google’s early-access program is intended to bring new Gemini models to Duo as they become available. - Regulated or self-hosted teams can use Gemma 4 with GitLab Duo Self-Hosted. - With self-hosted models, the AI Gateway and all requests and responses remain within an organization’s on-premises or private-cloud environment. ## Using Existing Google Cloud Commitments - GitLab and Duo Agent Platform can be purchased through Google Cloud Marketplace. - Existing Google Cloud commitments can fund GitLab subscriptions, model inference, and related infrastructure without starting a new procurement cycle. - Consolidated Google Cloud billing reduces reconciliation across vendors. - GitLab retains its own cost-management features, including usage dashboards, model policies, and GitLab Credits for more predictable AI spending. ## One Governed DevSecOps Platform - GitLab Duo Agent Platform provides software-delivery context that standalone coding assistants lack, including merge requests, pipelines, and deployment targets. - This context helps agents perform multi-step work and supports code review at monorepo scale. - Combining GitLab’s governance and lifecycle data with Google’s models keeps deployment, model selection, compliance, and spending aligned in one platform rather than fragmented across multiple tools. Organizations can start with a Duo Agent Platform trial, enable it through the free GitLab tier, or use included GitLab Credits with Premium and Ultimate subscriptions. Overall, the offering is aimed at teams that want managed GitLab operations, flexible AI model access, and strong control over data location and costs on Google Cloud.

Read original

Gemini Gitlab DevSecOps Gitlab Duo Agent Platform+4

googleMay 28, 20263 min readCurated summary

A New Era of Innovation: Google Research at I/O 2026

Google’s I/O 2026 research announcements present AI as an “agentic” amplifier of human ingenuity, particularly in science and healthcare. New systems such as Gemini for Science, ERA, Co-Scientist, and Gemini Deep Think are designed to generate hypotheses, write and optimize code, evaluate evidence, and solve difficult research problems. Google also highlighted health-focused AI that supports users before, during, and after medical visits, while emphasizing collaboration, validation, and responsible deployment. ## AI-Driven Scientific Discovery - **Gemini for Science** is a suite of experimental tools built from Google Research and developed with Google Cloud, Google DeepMind, and Google Labs. - **Empirical Research Assistance (ERA)** acts as a code-optimizing research engine: - Proposes concepts and writes software. - Evaluates results against a defined scoring system. - Uses tree search to test thousands of code variants. - Has supported work in neuroscience, cosmology, respiratory-illness forecasting, and California runoff prediction. - **Co-Scientist** is a Gemini-based multi-agent collaborator that generates, evaluates, and refines hypotheses. - Researchers have applied it to antimicrobial resistance, plant immunity, and liver fibrosis. - **Computational Discovery**, combining ERA and AlphaEvolve, runs thousands of code variations in parallel to test scientific models and hypotheses more quickly. - **Hypothesis Generation** uses a multi-agent “idea tournament” to debate and rank research ideas, with clickable citations supporting claims. - **Literature Insights**, powered by NotebookLM, helps researchers synthesize large bodies of scientific literature. - **Science Skills** can automate specialist workflows such as structural bioinformatics and genomic analysis on agentic coding platforms. ## AI for Peer Review and Advanced Reasoning - Google is piloting the **Paper Assistant Tool (PAT)** for scientific peer review. - PAT has experimentally reviewed more than 10,000 papers for conferences including ICML, STOC, and NeurIPS. - Its feedback has helped authors identify theoretical gaps and design additional experiments. - **Gemini Deep Think** has been used with mathematicians, physicists, and computer scientists to address open problems involving network deadlocks, optimization, machine-learning behavior, auction theory, and cosmic-string singularities. ## Advancing Health with AI - Google’s health research focuses on supporting people throughout the full healthcare journey, from understanding symptoms and preparing for appointments to interpreting medical records. - Research contributions underpin the **Google Health app** and **Google Health Coach**, with the app beginning rollout to existing Fitbit users. - **Symptom AI** investigates how conversational AI can reason about information relevant to a person’s symptoms. - A Fitbit-based study included 13,917 participants. - In blind comparisons, clinicians preferred Symptom AI’s differential diagnoses roughly twice as often as those produced by other clinicians. - The **Plan for Care** pilot involved 1,779 participants preparing for doctor visits. - Compared with baseline systems, 15% more users felt prepared. - 13% more users felt confident they could make effective use of their appointment. - Google is also studying personal health large language models and the use of personal health record data to improve health guidance. Google’s announcements point toward research systems that actively experiment, collaborate, and reason rather than merely retrieve information. Their practical value will depend on continued scientific validation, clinician involvement, privacy protections, and careful expansion from experimental tools into real-world use.

Read original

Gemini Machine Learning Generative AI Multi Agent Systems+2

googleMay 19, 20262 min readCurated summary

Empirical Research Assistance (ERA): From Nature publication to catalyzing Computational Discovery

Empirical Research Assistance (ERA) is a Google AI system designed to help scientists develop expert-level computational models. Using Gemini, it searches literature, generates and evaluates code, and explores thousands of possible solutions through tree search. A Nature paper reports strong performance across scientific benchmarks, while new applications suggest ERA can accelerate research in health, climate, energy, and economics. ## How ERA Supports Scientific Coding - ERA starts with a scientific problem and a success metric. - It searches relevant research, combines methods, writes code, and iteratively tests and improves solutions. - Its tree-search process evaluates thousands of alternatives to optimize the resulting model. - Benchmarks in genomics, public health, satellite imagery, neuroscience, time-series forecasting, and mathematics showed expert-level performance. ## Applications to Open Scientific Problems - **Epidemiological forecasting** - Predicted U.S. hospital admissions up to four weeks ahead for flu, COVID-19, and RSV. - Forecasts ranked at or near the top of CDC leaderboards. - The techniques can potentially be adapted to other countries and diseases. - **California water-supply forecasting** - Produced seasonal runoff predictions for snow-fed river basins. - Delivered more accurate early forecasts than California’s official Bulletin 120 outlook. - Improved predictions could support water management and agriculture. - **Atmospheric carbon dioxide monitoring** - Combined geostationary weather-satellite data with other inputs to estimate CO₂ concentrations every 10 minutes across broad areas. - Captured urban emissions, plant-driven daytime absorption, and other atmospheric cycles. - Provides higher spatial and temporal coverage than measurements from satellites such as Orbiting Carbon Observatory-2. - **Solar-energy design** - Combined ERA with Google Antigravity to optimize three-dimensional solar-panel geometries. - Identified a 500-triangle volumetric fan design that could capture scattered radiation without backward shading. - **Retail forecasting** - Used economic indicators, Google Trends, historical patterns, and consumer sentiment. - Matched or exceeded commercial consensus forecasts and the Chicago Fed’s monthly retail forecast. ## Computational Discovery - Google is gradually opening access to Computational Discovery through a trusted tester program in Google Labs. - The system combines ERA with AlphaEvolve to support computational scientific investigation. - It complements other Gemini for Science experiments: - **Hypothesis Generation**, built with AI Co-Scientist, supports developing scientific hypotheses. - **Literature Insights** supports research and literature analysis. ERA’s demonstrated value lies in automating the labor-intensive cycle of designing, testing, and refining scientific software. Its expanding applications indicate that AI-assisted computational research could broaden access to advanced modeling while helping experts investigate complex scientific problems more quickly.

Read original

Gemini AI Machine Learning Time Series Forecasting+4

googleApr 9, 20263 min readCurated summary

ConvApparel: Measuring and bridging the realism gap in user simulators

ConvApparel addresses the “realism gap” between LLM-based user simulators and genuine human behavior. It combines over 4,000 human-AI shopping conversations with a controlled Good-versus-Bad agent setup and evaluates simulators through statistical alignment, human-likeness, and counterfactual adaptation. The framework aims to determine whether simulators genuinely model human reactions or merely reproduce patterns from their training data. ## Why User Simulator Realism Matters - Conversational agents often fail during long, multi-turn interactions by forgetting constraints or producing irrelevant responses. - Human testing provides valuable feedback but is expensive, slow, and difficult to scale. - LLM-based user simulators offer a scalable alternative, but often behave unlike real users: - They may be excessively verbose. - They can lack consistent personas or coherent preferences. - They may possess unrealistic, encyclopedic knowledge. - They are often unusually patient and assistant-like. - Training systems only against unrealistic simulators may cause them to fail with real users. ## The Need for Counterfactual Validation - A simulator should respond plausibly not only to situations represented in its training data, but also to novel assistant behaviors. - The authors introduce **counterfactual validation**: training a simulator on helpful-agent conversations, then testing it against an unexpectedly frustrating agent. - A realistic simulator should recognize poor assistance and show increased frustration, reduced satisfaction, and behavior changes similar to those of real users. - This tests whether the simulator has learned general human behavior rather than memorized training patterns. ## The ConvApparel Dataset - ConvApparel contains more than 4,000 human-AI multi-turn conversations and nearly 15,000 total turns in the apparel-shopping domain. - Participants were unknowingly assigned to one of two recommendation agents: - **Good agent:** Helpful, efficient, and supported by robust search. - **Bad agent:** Intentionally confusing, tangential, and based on degraded search retrieval. - The dataset captures reactions ranging from satisfaction to significant annoyance. - Participants provided turn-by-turn retrospective annotations, including: - Satisfaction - Frustration - Likelihood of making a purchase ## Three-Part Evaluation Framework ### Population-Level Statistical Alignment - Simulated conversations are compared with human conversations using aggregate measures such as: - Conversation length - Words per turn - Dialogue acts, including rejecting recommendations - This reveals whether simulators reproduce broad behavioral distributions. ### Human-Likeness Score - An automated discriminator is trained on human and simulated conversations. - It produces a probability indicating how human-like a conversation appears. - The score is intended to detect subtle stylistic differences that simple statistics may miss. ### Counterfactual Validation - A simulator is trained only on conversations with the Good agent. - It then interacts with the unseen Bad agent. - High-fidelity simulation should produce a human-like increase in frustration and decline in satisfaction when the assistant behaves poorly. ## Simulator Configurations The experiments compare three Gemini-based user simulators: - **Prompted simulator:** Uses high-level behavioral instructions without additional task-specific training. - **In-context learning (ICL) simulator:** Retrieves semantically similar human conversations from ConvApparel and supplies them as examples at each turn. - **Supervised fine-tuning (SFT) simulator:** Trains a Gemini 2.5 Flash model directly on the dataset. The post presents ConvApparel as a structured way to measure simulator realism and test whether simulated users can adapt to assistant behavior outside their training distribution. Its central recommendation is to evaluate user simulators not only by surface-level similarity, but also by how naturally they react to unexpected failures.

Read original

Gemini Database Design Large Language Models Conversational Ai+4

googleMar 25, 20263 min readCurated summary

Vibe Coding XR: Accelerating AI + XR prototyping with XR Blocks and Gemini

Vibe Coding XR combines Gemini’s natural-language coding capabilities with the open-source XR Blocks framework to rapidly create interactive, physics-aware WebXR applications. Users can describe an experience—such as a dandelion, physics lab, or educational visualization—and receive a working Android XR prototype in under 60 seconds. The workflow supports both desktop simulation and deployment to Android XR headsets, making spatial prototyping faster and more accessible. ## Bridging AI Prototyping and XR - Traditional XR development requires fragmented perception systems, game engines, and low-level sensor integrations. - Vibe-coded prototypes let developers quickly evaluate 3D interfaces, spatial interactions, and visualizations before investing in full production. - The workflow is designed for both experienced developers and creators without prior XR expertise. - Gemini translates natural-language prompts into functional XR applications with scene setup, perception, interaction, and physics logic. ## The Vibe Coding XR Workflow - Users open the XR Blocks Gem in Chrome on an Android XR headset or desktop. - They provide a prompt by typing or using voice, such as “Create a beautiful dandelion.” - Gemini plans and implements the experience using XR Blocks examples and templates. - On Android XR, users can enter the experience with a pinch gesture and interact naturally—for example, pinching to blow away an animated dandelion. - Applications can be published through a shareable public link. - Desktop Chrome provides a simulated-reality environment for testing before deployment, while Android XR enables advanced features such as hand tracking, depth sensing, and physics. ## Technical Foundation - XR Blocks is built on WebXR, three.js, and LiteRT.js. - Its engine coordinates: - Environmental perception - XR interaction - Spatial computing - AI integration - Gemini receives a specialized system prompt containing: - XR design guidelines for room-scale environments, spatial layout, scale, and interaction distances - Package-management rules and recommended styles - Curated source code, templates, and working samples - Grounding Gemini in valid XR Blocks APIs reduces hallucinated code and encourages consistent implementation patterns. ## Educational and Interactive Applications - **Math tutor:** Visualizes Euler’s theorem using tetrahedra, cubes, and octahedra, with pinch-based highlighting of vertices, edges, and faces. - **Physics lab:** Lets users pick up and place labeled weights on a balance scale to learn about equilibrium. - **Immersive chemistry:** Simulates methane, ethylene, and acetylene combustion with educational cards and volumetric effects, offering a safer mixed-reality alternative to physical experiments. - **Schrödinger’s cat:** Uses pinch and proximity interactions to demonstrate superposition, revealing alive and dead versions of a cat before collapsing the state into one outcome. - **XR sports:** Generates interactive experiences such as hand-based volleyball, including textured balls, environmental collision, and adjustable launch behavior. ## Practical Value - Creators can test spatial ideas in minutes rather than building complete XR pipelines first. - The same prototype can be evaluated on desktop and then experienced with body and hand interactions on Android XR. - The approach is especially useful for education, interaction design, scientific visualization, and early-stage product exploration. Vibe Coding XR is best viewed as a rapid experimentation layer rather than a replacement for production XR engineering. By combining Gemini’s reasoning with XR Blocks’ specialized runtime and templates, it significantly lowers the barrier to creating and validating intelligent spatial experiences.

Read original

Gemini Vibe Coding Spatial Computing Webxr+4

googleMar 16, 20263 min readCurated summary

Testing LLMs on superconductivity research questions

LLMs may help physicists navigate complex research, but their reliability depends heavily on the quality and curation of their sources. In a high-temperature superconductivity study, systems grounded in expert-selected literature—especially NotebookLM and a custom retrieval-augmented generation system—outperformed general web-access models. The results suggest that trustworthy scientific AI requires balanced reasoning, strong evidence, and carefully controlled reference collections. ## Evaluating LLMs on Superconductivity - Researchers from Google Research and Cornell University tested whether LLMs could answer expert-level questions in condensed matter physics. - The study focused on cuprate high-temperature superconductors, whose underlying mechanism remains unresolved despite decades of research. - Understanding superconductivity in these materials could help scientists discover compounds that work at higher temperatures. - The field contains thousands of experimental and theoretical papers and competing explanations, making it difficult for researchers—especially newcomers—to establish a balanced view. ## Study Design and Sources - Six systems were evaluated: - GPT-4o - Perplexity - Claude 3.5 - Gemini Advanced Pro 1.5 - Google NotebookLM - A custom retrieval-augmented generation (RAG) system - Four models had broad web access, including 765 open-access experimental papers and 1,553 theoretical papers. - NotebookLM and the custom RAG system used a curated database: - Twelve superconductivity experts selected 15 review articles. - Those reviews contained approximately 3,300 references. - A final collection of 1,726 experimental papers and reviews was assembled. - Experts created 67 difficult questions, including questions about doping levels and evidence for quantum criticality in cuprates. ## Evaluation Criteria Experts used masked reviews and scored responses from 0 to 2 on: - Balance between competing scientific perspectives - Comprehensiveness and factual depth - Conciseness and clarity - Evidence and links to sources - Relevance of supplied images - Qualitative comments ## Results - NotebookLM achieved the strongest overall performance. - The custom RAG system ranked second overall, showing the value of retrieval from the same expert-curated sources. - NotebookLM, Gemini, and the custom RAG system performed best at presenting balanced and comprehensive answers. - NotebookLM provided the strongest evidence and citations but was less concise than the other systems. - Image quality was generally weaker; the custom RAG system performed best among the models that regularly supplied images. - All systems showed areas needing improvement, particularly when addressing nuanced, unresolved research questions. ## Practical Implication For scientific research, LLMs should be paired with expert-curated, quality-controlled literature rather than relying solely on unrestricted web searches. Such systems can serve as research tutors or thought partners, but their answers still require expert verification, especially in fields with competing theories and rapidly evolving evidence.

Read original

Gemini Database Design Large Language Models Notebooklm+2

googleMar 12, 20263 min readCurated summary

Introducing Groundsource: Turning news reports into data with Gemini

Groundsource is a Google Research methodology that uses Gemini to convert global news reports into structured historical records of natural disasters. Its first dataset contains 2.6 million flash-flood events across more than 150 countries from 2000 onward, addressing major gaps in conventional flood databases. Google reports that the system can support near-global urban flash-flood forecasts up to 24 hours in advance. ## The problem: Limited historical disaster data - Floods lack the standardized global sensor infrastructure available for hazards such as earthquakes. - Existing sources, including the Global Flood Database and Dartmouth Flood Observatory, are limited by cloud cover, satellite revisit times, and their focus on large or long-lasting floods. - GDACS contains roughly 10,000 high-impact disaster records but misses many localized and rapidly developing flash floods. - This shortage of reliable historical data makes global forecasting, model training, and validation difficult. ## How Groundsource processes news - The system analyzes news articles where flooding is the primary subject. - Google Read Aloud extracts article text in 80 languages, which is translated into English using Cloud Translation. - Gemini then applies a verification-oriented prompt to: - Distinguish actual past or ongoing floods from warnings, policy discussions, and general risk reports. - Resolve relative dates such as “last Tuesday” using the article’s publication date. - Identify precise locations, including neighborhoods and streets. - Map locations to standardized geographic polygons through Google Maps Platform. ## Accuracy and scale - Manual evaluation found: - 60% of events were accurate in both timing and location. - 82% were sufficiently accurate for practical analysis, such as identifying the correct administrative district or event day. - The resulting dataset contains 2.6 million flood events, greatly exceeding traditional monitoring archives. - Between 2020 and 2026, Groundsource captured 85%–100% of severe flood events listed by GDACS while also recording smaller local incidents. - Coverage is densest in recent years, particularly from 2020 to 2025, reflecting the growth of digitized news. ## Forecasting and future applications - Groundsource data has enabled near-global urban flash-flood forecasts up to 24 hours ahead. - These forecasts are being integrated into Google Flood Hub. - Google plans to improve rural coverage and incorporate additional data sources. - The same approach could help build historical datasets for droughts, landslides, avalanches, and other hazards with limited ground-truth records. Groundsource demonstrates that news archives can serve as a large-scale source of disaster history when combined with language models, translation, and geographic verification. Its open flash-flood dataset could improve forecasting and resilience planning, though its reported accuracy levels make continued validation and refinement important.

Read original

Gemini Generative AI Flood Forecasting Groundsource+4

googleFeb 16, 20263 min readCurated summary

Teaching AI to read a map

MapTrace addresses a major weakness in multimodal language models: recognizing objects on maps is easier for them than understanding connectivity, obstacles, and valid routes. The authors propose a synthetic-data pipeline that generates maps, identifies walkable areas, constructs navigation graphs, and verifies computed paths with AI critics. They report releasing 2 million map question-answer pairs and show that fine-tuning on a much smaller subset improves route tracing on unseen real-world maps. ## The Challenge: Weak Spatial Grounding - MLLMs may recognize locations and objects in an image but still draw routes through walls, buildings, enclosures, or shops. - Effective navigation requires understanding: - Which regions are traversable - How paths connect - That routes are ordered sequences of connected points - The geometric and topological relationships between map features - Existing image-text training rarely teaches this “spatial grammar.” - Manual pixel-level route annotation would be expensive and difficult to scale. - Many useful maps of malls, museums, and theme parks are proprietary, limiting access to real-world training data. ## A Scalable Synthetic-Data Pipeline MapTrace uses generative AI to create diverse maps and automatically produce valid route annotations. ### Generating Diverse Maps - An LLM creates detailed prompts for environments such as: - Zoos with interconnected habitats - Shopping malls with food courts - Fantasy theme parks with themed areas - A text-to-image model renders the prompts as map images. - This approach provides control over map diversity and complexity. ### Identifying Walkable Areas with a Mask Critic - Pixels are clustered by color to produce candidate masks representing possible walkways. - An MLLM reviews each mask alongside the original map. - The “Mask Critic” rejects masks that do not represent realistic, connected traversable regions. - Accepted areas may include sidewalks, crosswalks, and pedestrian paths. ### Converting Maps into Navigation Graphs - The selected traversable mask is converted into a pixel-based graph. - Walkway intersections become nodes, while connected stretches become edges. - This graph captures the map’s connectivity and enables computational route planning. ### Generating and Validating Routes - Thousands of random start and end points are sampled for each map. - Dijkstra’s algorithm computes the shortest path between each pair. - A “Path Critic” checks the overlaid route to ensure it: - Stays within traversable regions - Avoids obstacles - Follows a logical human route - Routes approved by the critic become training examples. ## Dataset and Evaluation - The pipeline generated 2 million annotated map question-answer pairs. - The authors note that generated maps sometimes contain incorrect text, but the study focuses primarily on path fidelity. - They fine-tuned models including Gemma 3 27B and Gemini 2.5 Flash on 23,000 generated paths. - Performance was evaluated on MapBench, which contains unseen real-world maps. - Route accuracy was measured using normalized dynamic time warping (NDTW), which compares predicted and reference coordinate sequences while accounting for differences in sampling and travel speed. - Lower NDTW scores indicate closer agreement with the reference route. ## Conclusion The work suggests that targeted synthetic training data can teach MLLMs map-based spatial reasoning that is largely missing from general pretraining. The released dataset and pipeline provide a foundation for improving visual navigation, while better image-generation models should reduce remaining typography and rendering artifacts.

Read original

Gemini Synthetic Data Generation Mllm Spatial Reasoning+3

googleFeb 5, 20263 min readCurated summary

How AI tools can redefine universal design to increase accessibility

Google Research proposes Natively Adaptive Interfaces (NAI), a framework that uses multimodal and agentic AI to make interfaces adapt to individual users rather than forcing everyone into a fixed design. Developed through co-design with disability communities, NAI aims to reduce the accessibility gap by embedding assistive capabilities directly into products. Early prototypes suggest that personalized, context-aware interfaces can improve experiences for disabled users while also benefiting the broader population. ## Community-led co-design - Google follows the principle “Nothing About Us, Without Us,” involving people with disabilities as co-designers from the beginning. - Partnerships include RIT/NTID, The Arc of the United States, RNID, and Team Gleason. - These collaborations focus on real-world barriers and recognize the expertise of disability communities. - The approach also aims to create employment and economic opportunities for people who help shape the technology. ## Moving from reactive accessibility to adaptive interfaces - Google identifies an “accessibility gap” between the release of new features and the development of compatible assistive tools. - NAI addresses this by making accessibility native to the interface instead of adding it afterward. - Static navigation is replaced with dynamic, agent-driven modules that can interpret context and adjust the experience. ## Multi-system agents - An Orchestrator maintains shared context and delegates tasks to specialized sub-agents. - A Summarization Agent breaks down complex documents and assigns subtasks to expert agents. - A Settings Agent dynamically adjusts interface elements such as text size. - This structure lets users accomplish tasks without navigating complicated menus or searching for the right control. ## Multimodal interaction - Gemini-based prototypes combine voice, vision, and text rather than limiting accessibility to text-to-speech. - Live video can be converted into interactive audio descriptions. - Users can ask follow-up questions about specific visual details as events unfold. - Conversational interaction provides situational awareness and may reduce cognitive load. ## Proven prototypes - **StreetReaderAI** - Supports blind and low-vision users navigating physical spaces. - Combines an AI Describer that analyzes visual and geographic information with an AI Chat system for questions. - Maintains context so users can ask about previously encountered locations, such as the position of a bus stop. - **Multimodal Agent Video Player (MAVP)** - Makes audio description interactive rather than static. - Users can change the level of detail or ask questions during playback. - Uses an offline “dense index” of visual descriptions and retrieval-augmented generation (RAG) for fast responses. - **Grammar Laboratory** - Developed by RIT/NTID with Google.org support for American Sign Language and English learners. - Provides grammar instruction through ASL videos, English captions, spoken narration, and written transcripts. - Uses adaptive AI to customize lessons according to each student’s language preferences and interactions. ## The curb-cut effect - Accessibility features designed for people with significant constraints can benefit many other users. - Voice interfaces created for blind users may help sighted people who are multitasking. - AI synthesis and learning tools designed for people with learning disabilities can also support users who want information presented more clearly or flexibly. - NAI therefore treats accessibility as a source of better universal design, not as a specialized add-on. NAI’s central recommendation is to build accessibility into interfaces from the start, using multimodal AI, persistent context, and community-led design. The most effective systems will adapt to users while remaining accountable to the people whose needs they are intended to serve.

Read original

Gemini AI Agents Generative AI Accessibility+4

googleJan 22, 20263 min readCurated summary

Small models, big results: Achieving superior intent extraction through decomposition

Small multimodal models can outperform much larger models at extracting user intent from UI interaction trajectories when the task is decomposed. Google’s approach first summarizes each screen and interaction, then derives an overall intent from those summaries. This enables accurate, faster, and more privacy-preserving on-device understanding without sending sensitive UI data to servers. ## Why On-Device Intent Understanding Matters - Understanding what users are doing across mobile and web interfaces can help agents anticipate useful next actions. - Large multimodal models perform well but often require server-side processing, introducing latency, cost, and privacy risks. - The goal is to make intent understanding practical with smaller models running directly on devices. ## Two-Stage Intent Extraction ### Screen and Interaction Summaries For each interaction, a small multimodal model examines a sliding window of three screens: the previous, current, and next screens. It generates information about: - Salient context on the current screen. - Actions the user just performed. - A speculation about what the user is trying to accomplish. This converts raw screenshots and actions into structured, manageable event summaries. ### Intent Extraction from Summaries A fine-tuned small model then processes the sequence of summaries and produces a single concise intent statement. The authors improve this stage through: - **Fine-tuning:** Training on examples of high-quality intent statements helps the model retain relevant details and discard noise. - **Label preparation:** Training intents are stripped of details absent from the summaries, reducing hallucinated information. - **Removing speculation:** Speculative fields help create richer individual summaries but are excluded from the second stage because they can confuse intent extraction. ## Evaluation with Atomic Facts - The authors use the Bi-Fact evaluation method to compare predicted intents with reference intents. - Each intent is split into indivisible “atomic facts,” such as “a one-way flight” or the separate origin and destination in a flight request. - The method measures: - **Recall:** How many reference facts were captured. - **Precision:** How many predicted facts are supported by the reference. - **F1:** The balance between precision and recall. - Tracking facts through both stages also reveals where details are lost or hallucinated. ## Results - The decomposed method outperformed chain-of-thought prompting and end-to-end fine-tuning. - Improvements held across both mobile and web interaction trajectories. - Results were consistent across Gemini and Qwen2 base models. - Gemini 1.5 Flash 8B achieved results comparable to Gemini 1.5 Pro while offering substantially lower cost and faster processing. - On mobile data, the small-model approach approached the performance of the larger Gemini Pro model. The study suggests that decomposing intent understanding into local summarization followed by sequence-level extraction is an effective path toward accurate, private, and efficient on-device assistants. As mobile hardware and small models improve, this technique could support a broad range of assistive features.

Read original

Gemini Natural Language Processing Multimodal AI Fine Tuning+4

lineDec 30, 2025Original article

A Business Trip to Japan After Only One (opens in new tab)

Joining the Developer Relations (DevRel) team at LINE Plus, a new employee was immediately thrust into a high-stakes business trip to Japan just one week after onboarding to support major global tech events. This immersive experience allowed the recruit to rapidly grasp the company’s engineering culture by facilitating cross-border collaboration and managing large-scale technical conferences. Ultimately, the journey highlights how a proactive onboarding strategy and a culture of creative freedom enable DevRel professionals to bridge the gap between complex engineering feats and community engagement. ### Global Collaboration at Tech Week * The trip centered on participating in **Tech-Verse**, a global conference featuring simultaneous interpretation in Korean, English, and Japanese, where the focus was on maintaining operational detail across diverse technical sessions. * Operational support was provided for **Hack Day**, an in-house hackathon that brought together engineers from various countries to collaborate on rapid prototyping and technical problem-solving. * The experience facilitated direct coordination with DevRel teams from Japan, Thailand, Taiwan, and Vietnam, establishing a unified approach to technical branding and regional community support. * Post-event responsibilities included translating live experiences into digital assets, such as "Shorts" video content and technical blog recaps, to maintain engagement after the physical event concluded. ### Modernizing Internal Technical Sharing * The **Tech Talk** series, a long-standing tradition with over 78 sessions, was used as a platform to experiment with "B-grade" humorous marketing—including quirky posters and cup holders—to drive offline participation in a remote-friendly work environment. * To address engineer feedback, the format shifted from passive lectures to **hands-on practical sessions** focusing on AI implementation. * Specific technical workshops demonstrated how to use tools like **Claude Code** and **ChatGPT** to automate workflows, such as generating weekly reports by integrating **Jira tickets with internal Wikis**. * Preparation for these sessions involved creating detailed environment setup guides and troubleshooting protocols to ensure a seamless experience for participating developers. ### Scaling AI Literacy via AI Campus Day * The **AI Campus Day** was a large-scale event designed for over 3,000 participants, aimed at lowering the barrier to entry for AI adoption across all departments. * The "Event & Operation" role involved creating interactive AI photo zones using **Gemini** to familiarize employees with new internal AI tools in a low-pressure setting. * Event production utilized AI-driven assets, including AI-generated voices and icons, to demonstrate the practical utility of these tools within standard business communication and video guides. * The success of the event relied on "participation design," ensuring that even non-technical staff could engage with AI concepts through hands-on play and peer mentoring. For organizations looking to strengthen their technical culture, this experience suggests that integrating new hires into high-impact global projects immediately can be a powerful onboarding tool. Providing DevRel teams the psychological safety to experiment with unconventional marketing and hands-on technical workshops is essential for maintaining developer engagement in a hybrid work era.

Gemini AI Generative AI Claude Code+5

lineDec 19, 2025Original article

We held AI Campus Day to improve (opens in new tab)

LY Corporation recently hosted "AI Campus Day," a large-scale internal event designed to bridge the gap between AI theory and practical workplace application for over 3,000 employees. By transforming their office into a learning campus, the company successfully fostered a culture of "AI Transformation" through peer-led mentorship and task-specific experimentation. The event demonstrated that internal context and hands-on participation are far more effective than traditional external lectures for driving meaningful AI literacy and productivity gains. ## Hands-on Experience and Technical Support * The curriculum featured 10 specialized sessions across three tracks—Common, Creative, and Engineering—to ensure relevance for every job function. * Sessions ranged from foundational prompt engineering for non-developers to advanced technical topics like building Model Context Protocol (MCP) servers for engineers. * To ensure smooth execution, the organizers provided comprehensive "Session Guides" containing pre-configured account settings and specific prompt templates. * The event utilized a high support ratio, with 26 teaching assistants (TAs) available to troubleshoot technical hurdles in real-time and dedicated Slack channels for sharing live AI outputs. ## Peer-Led Mentorship and Internal Context * Instead of hiring external consultants, the program featured 10 internal "AI Mentors" who shared how they integrated AI into their actual daily workflows at LY Corporation. * Training focused exclusively on company-approved tools, including ChatGPT Enterprise, Gemini, and Claude Code, ensuring all demonstrations complied with internal security protocols. * Internal mentors were able to provide specific "company context" that external lecturers lack, such as integrating AI with existing proprietary systems and data. * A rigorous three-stage quality control process—initial flow review, final end-to-end dry run, and technical rehearsal—was implemented to ensure the educational quality of mentor-led sessions. ## Gamification and Cultural Engagement * The event was framed as a "festival" rather than a mandatory training, using campus-themed motifs like "enrollment" and "school attendance" to reduce psychological barriers. * A "Stamp Rally" system encouraged participation by offering tiered rewards, including welcome kits, refreshments, and subscriptions to premium AI tools. * Interactive exhibition booths allowed employees to experience AI utility firsthand, such as an AI photo zone using Gemini to generate "campus-style" portraits and an AI Agent Contest booth. * Strong executive support played a crucial role, with leadership encouraging staff to pause routine tasks for the day to focus entirely on AI experimentation and "playing" with new technologies. To effectively scale AI literacy within a large organization, it is recommended to move away from passive, one-size-fits-all lectures. Success lies in leveraging internal experts who understand the specific security and operational constraints of the business, and creating a low-pressure environment where employees can experiment with hands-on tasks relevant to their specific roles.

Gemini AI Generative AI Model Context Protocol+5

googleDec 14, 2025Original article

Gemini provides automated feedback for theoretical computer scientists at STOC 2026 (opens in new tab)

Google Research launched an experimental program for the STOC 2026 conference using a specialized Gemini model to provide automated, rigorous feedback on theoretical computer science submissions. By identifying critical logical errors and proof gaps within a 24-hour window, the tool demonstrated that advanced AI can serve as a powerful pre-vetting collaborator for high-level mathematical research. The overwhelmingly positive reception from authors indicates that AI can effectively augment the human peer-review process by improving paper quality before formal submission. ## Advanced Reasoning via Inference Scaling - The tool utilized an advanced version of Gemini 2.5 Deep Think specifically optimized for mathematical rigor. - It employed inference scaling methods, allowing the model to explore and combine multiple possible solutions and reasoning traces simultaneously. - This non-linear approach to problem-solving helps the model focus on the most salient technical issues while significantly reducing the likelihood of hallucinations. ## Structured Technical Feedback - Feedback was delivered in a structured format that included a high-level summary of the paper's core contributions. - The model provided a detailed analysis of potential mistakes, specifically targeting errors within lemmas, theorems, and logical proofs. - Authors also received a categorized list of minor corrections, such as inconsistent variable naming and typographical errors. ## Identified Technical Issues and Impact - The pilot saw high engagement, with over 80% of STOC 2026 submitters opting in for the AI-generated review. - The tool successfully identified "critical bugs" and calculation errors that had previously evaded human authors for months. - Survey results showed that 97% of participants found the feedback helpful, and 81% reported that the tool improved the overall clarity and readability of their work. ## Expert Verification and Hallucinations - Because the users were domain experts, they were able to act as a filter, distinguishing between deep technical insights and occasional model hallucinations. - While the model sometimes struggled to parse complex notation or interpret figures, authors valued the "neutral tone" and the speed of the two-day turnaround. - The feedback was used as a starting point for human verification, allowing researchers to refine their arguments rather than blindly following the model's output. ## Future Outlook and Educational Potential - Beyond professional research, 75% of surveyed authors see significant educational value in using the tool to train students in mathematical rigor. - The experiment's success has led to 88% of participants expressing interest in having continuous access to such a tool throughout their entire research and drafting process. The success of the STOC 2026 pilot suggests that researchers should consider integrating specialized LLMs early in the drafting phase to catch "embarrassing" or logic-breaking errors. While the human expert remains the final arbiter of truth, these tools provide a necessary layer of automated verification that can accelerate the pace of scientific discovery.

Gemini AI Large Language Models Generative AI+5