Computer Vision - Google Research

googleJun 16, 20263 min readCurated summary

From pixels to planning: Earth AI for nature restoration

Google Research developed a high-resolution AI system that converts satellite imagery into vector data identifying small ecological features such as hedgerows, stone walls, and copses. These features can store carbon and support biodiversity without taking agricultural land out of production, but they are often missed by conventional forest inventories. The resulting open dataset aims to make fine-scale nature restoration measurable and actionable across the UK. ## The case for fine-scale restoration - Forests support carbon sequestration, water filtration, and biodiversity, but expanding forests can compete with food production. - Conservation efforts can also create “leakage,” shifting environmental damage elsewhere. - Hedgerows, shelterbelts, and other woody features offer ecological benefits while remaining integrated into farmland. - Google’s earlier **Farmscapes 2020** raster map detected these features across England, but pixel data was insufficient for planning, restoration, and carbon accounting. ## Challenges in mapping the countryside - Agricultural features often overlap spatially, such as hedgerows growing alongside or over stone walls. - Processing data in S2-cell tiles can split features at tile boundaries. - A generic “woody” classification does not reveal whether a feature is a forest, wildlife corridor, or isolated copse. - England’s more than 130,000 km² of high-resolution imagery created major computational demands for conventional raster-to-vector processing. ## Deep learning and vectorization - The team fine-tuned Remote Sensing Foundations’ Vision Transformer, pretrained on more than 300 million global satellite images. - Only about 247 km² of annotated British landscape data was available, so the pretrained model helped transfer broad visual knowledge to local conditions. - A dual-layer labeling system combined submeter imagery with 1-meter LiDAR to distinguish: - Ground-level boundaries such as fields and water - Above-ground features such as trees and walls - Geometry-merging algorithms reconnected features split across S2-cell borders. ## Classifying ecological features - The system used the **Polsby–Popper compactness score** to classify detected shapes by geometry. - Woodlands were defined as contiguous canopies at least 30 meters in diameter. - Woody patches included small copses and individual trees. - Linear woody features, including hedgerows and corridors, were identified using a compactness score below 0.5. - This classification turns raw detections into an inventory tied more closely to ecological function. ## Scaling the analysis - Google Earth Engine processed thousands of S2 cells in parallel. - This made it possible to generate vector geometries for millions of features across England without overwhelming traditional systems. - The resulting dataset is intended for landowners, conservationists, scientists, and policymakers. ## Future applications - The researchers plan to apply the method to silvopasture and agrisilviculture. - High-precision monitoring could help detect conservation “leakage” beyond project boundaries. - Open access to the data could support restoration while protecting agricultural productivity and food security. The dataset provides a practical foundation for planning and measuring nature recovery on working lands, demonstrating how AI can translate overlooked landscape details into conservation decisions.

Read original

Computer Vision Deep Learning Remote Sensing Geospatial Data+4

googleJun 4, 20263 min readCurated summary

Towards passive heart health monitoring via smartphone camera

PHRM is a Google Research system that passively estimates heart rate and daily resting heart rate using short facial videos captured by a smartphone’s front-facing camera after face unlock. It applies remote photoplethysmography and efficient deep learning, achieving wearable-comparable accuracy in laboratory and real-world settings. A central contribution is its deliberate validation across skin tones, including substantial representation of participants with dark skin. ## Passive Heart Monitoring with a Smartphone - PHRM records approximately eight seconds of facial video during everyday phone use. - It estimates heart rate from changes in how light reflects from the skin as blood pulses through facial tissue. - The system uses: - Temporal shift convolutional neural networks for efficient on-device prediction. - Confidence scores to filter unreliable measurements. - Kalman filtering and daily aggregation to estimate resting heart rate. - Unlike earlier smartphone approaches requiring a finger over the camera, PHRM works passively in the background. ## Designing for Skin-Tone Inclusivity - Existing remote PPG research often relied on small, controlled studies and underrepresented people with dark skin. - Google trained PHRM on more than 350,000 video clips from nearly 700 participants across laboratory and real-world conditions. - Dataset targets based on the Monk Skin Tone scale ensured: - At least 25% representation of light and medium skin tones. - At least 33% representation of dark skin tones. - The researchers required heart-rate error differences between skin-tone groups to remain below five percentage points. ## Laboratory Validation - The model was trained using synchronized facial video and ECG recordings from 365 diverse participants. - On a separate 104-person test set, PHRM achieved mean absolute percentage error below 10% for every skin-tone group after confidence filtering. - It outperformed 15 leading published remote-PPG models and was the only evaluated model to meet the under-10% target across all groups. ## Real-World Smartphone Study - In an eight-day free-living study, 231 participants used their own phones normally while wearing an ECG chest strap and Fitbit Charge 6. - The research app captured an average of 231 facial clips per participant per day after face unlock. - Participants reviewed and explicitly authorized uploads each day; recordings were sent to secure, encrypted servers. - On a held-out group of 101 participants: - Overall MAPE was 6.09% after confidence gating. - MAPE was 5.04% for light skin, 5.12% for medium skin, and 7.84% for dark skin. - The average underestimation was only 0.64 beats per minute. - The model again outperformed the 15 comparison systems. ## Resting Heart Rate and Research Access - By combining reliable heart-rate estimates throughout the day, PHRM produced daily resting-heart-rate estimates with mean absolute error below five beats per minute compared with a wearable tracker. - Google released the largest and most diverse publicly available smartphone-video dataset for remote PPG research, along with a pre-trained “PHRM-mini” model. - Access is available to qualified researchers. PHRM demonstrates that smartphones could broaden access to passive cardiovascular monitoring without requiring dedicated wearable hardware. Before clinical use, the system would still require continued validation, careful privacy protections, and assessment across additional devices and populations.

Read original

Computer Vision Deep Learning Convolutional Neural Networks Photoplethysmography+3

googleMar 24, 20263 min readCurated summary

Mapping the modern world: How S2Vec learns the language of our cities

S2Vec is a self-supervised framework that converts buildings, roads, businesses, and infrastructure into general-purpose geospatial embeddings. By rasterizing these features into S2 Geometry cells and training a masked autoencoder to reconstruct missing areas, it learns the spatial “character” of neighborhoods without manually labeled data. It performs especially well for socioeconomic predictions in geographically unseen regions, while environmental tasks benefit from combining it with satellite imagery. ## Turning Geospatial Data into Images - Geospatial data is multimodal and unevenly distributed: urban blocks may contain hundreds of features, while rural areas contain few. - S2Vec uses hierarchical S2 Geometry cells to divide the Earth into regions at different resolutions. - It counts feature types within each cell—such as buildings, parks, roads, and businesses—and organizes them into multilayered raster images. - This makes complex geographic information compatible with computer vision methods developed for ordinary images. ## Learning with Masked Autoencoding - S2Vec masks portions of the rasterized map and trains a model to reconstruct the missing features from surrounding context. - Repeated training across global locations teaches relationships among urban elements, such as the likelihood of shops near residential buildings and transit stations. - The resulting embeddings are compact numerical representations of each location’s built environment. - Because training is self-supervised, S2Vec does not require worldwide labels for income, air quality, population, or other metrics. - The model can identify similar neighborhood types without being explicitly told concepts such as “financial district” or “suburban residential area.” ## Evaluation and Socioeconomic Performance - S2Vec was compared with models including SATCLIP, GEOCLIP, RS-MaMMUT, Hex2vec, and GeoVeX. - Tests covered population density, median income, carbon emissions, tree cover, and elevation. - Models were evaluated using mean squared error and both: - Interpolation, using random train/test splits - Extrapolation, predicting conditions in geographically unseen regions - S2Vec was generally the strongest individual model for zero-shot socioeconomic prediction, including population density and median income. - It performed competitively with established image-based approaches and exceeded GEOCLIP in the reported comparisons. ## Benefits of Multimodal Fusion - Combining S2Vec with satellite-image embeddings generally produced better results than either modality alone. - Built-environment data captures structures and infrastructure, while satellite imagery adds information about vegetation, terrain, and transportation patterns. - Fusion was particularly valuable for environmental prediction tasks. ## Limitations on Environmental Tasks - Built-environment features alone do not fully explain factors such as tree cover and elevation. - S2Vec was competitive for carbon-emissions prediction but weaker on some environmental metrics. - Satellite imagery embeddings improved performance by supplying information unavailable from counts of buildings, roads, and businesses. S2Vec points toward scalable geographic foundation models that replace task-specific feature engineering with reusable representations. In practice, it is most effective when its built-environment embeddings are combined with complementary imagery, especially for environmental analysis.

Read original

Computer Vision Foundation Models Geospatial Data Self Supervised Learning+4

googleNov 12, 2025Original article

Separating natural forests from other tree cover with AI for deforestation-free supply chains (opens in new tab)

Researchers from Google DeepMind and Google Research have developed "Natural Forests of the World 2020," an AI-powered global map that distinguishes natural ecosystems from commercial tree plantations. By utilizing high-resolution satellite data and machine learning, the project provides a critical 10-meter resolution baseline to support deforestation-free supply chain regulations like the EUDR. This tool enables governments and companies to monitor biodiversity-rich areas with unprecedented accuracy, ensuring that natural forests are protected from industrial degradation. **The Limitation of Traditional Tree Cover Maps** * Existing maps frequently conflate all woody vegetation into a generic "tree cover" category, leading to "apples-to-oranges" comparisons between different land types. * This lack of distinction makes it difficult to differentiate between the harvesting of short-term plantations and the permanent loss of ancient, biodiversity-rich natural forests. * Precise mapping is now a legal necessity due to regulations like the European Union Regulation on Deforestation-free Products (EUDR), which bans products from land deforested or degraded after December 31, 2020. **The MTSViT Modeling Approach** * To accurately identify forest types, researchers developed the Multi-modal Temporal-Spatial Vision Transformer (MTSViT). * Rather than relying on a single snapshot, the AI "observes" 1280 x 1280 meter patches over the course of a year to identify seasonal, spectral, and textural signatures. * The model integrates multi-modal data, including Sentinel-2 satellite imagery, topographical information (such as elevation and slope), and specific geographical coordinates. * This temporal-spatial analysis allows the AI to recognize the complex patterns of natural forests that distinguish them from the uniform, fast-growing structures of commercial plantations. **Dataset Scale and Global Validation** * The model was trained on a massive dataset comprising over 1.2 million global patches at 10-meter resolution. * The final map provides seamless global coverage, achieving a best-in-class validation accuracy of 92.2% against an independent global dataset. * The research was a collaborative effort involving the World Resources Institute and the International Institute for Applied Systems Analysis to ensure scientific rigor and practical utility. The "Natural Forests of the World 2020" dataset is publicly available via Google Earth Engine and other open repositories. Organizations should leverage this high-resolution baseline to conduct environmental due diligence, support government monitoring, and target conservation efforts in preparation for global climate milestones like COP30.

Computer Vision AI Machine Learning Remote Sensing+3

googleOct 28, 2025Original article

StreetReaderAI: Towards making street view accessible via context-aware multimodal AI (opens in new tab)

StreetReaderAI is a research prototype designed to make immersive street-level imagery accessible to the blind and low-vision community through multimodal AI. By integrating real-time scene analysis with context-aware geographic data, the system transforms visual mapping data into an interactive, audio-first experience. This framework allows users to virtually explore environments and plan routes with a level of detail and independence previously unavailable through traditional screen readers. ### Navigation and Spatial Awareness The system offers an immersive, first-person exploration interface that mimics the mechanics of accessible gaming. * Users navigate using keyboard shortcuts or voice commands, taking "virtual steps" forward or backward and panning their view in 360 degrees. * Real-time audio feedback provides cardinal and intercardinal directions, such as "Now facing North," to maintain spatial orientation. * Distance tracking informs the user how far they have traveled between panoramic images, while "teleport" features allow for quick jumps to specific addresses or landmarks. ### Context-Aware AI Describer At the core of the tool is a subsystem backed by Gemini that synthesizes visual and geographic data to generate descriptions. * The AI Describer combines the current field-of-view image with dynamic metadata about nearby roads, intersections, and points of interest. * Two distinct modes cater to different user needs: a "Default" mode focusing on pedestrian safety and navigation, and a "Tour Guide" mode that provides historical and architectural details. * The system utilizes Gemini to proactively predict and suggest follow-up questions relevant to the specific scene, such as details about crosswalks or building entrances. ### Interactive Dialogue and Session Memory StreetReaderAI utilizes the Multimodal Live API to facilitate real-time, natural language conversations about the environment. * The AI Chat agent maintains a large context window of approximately 1,048,576 tokens, allowing it to retain a "memory" of up to 4,000 previous images and interactions. * This memory allows users to ask retrospective spatial questions, such as "Where was that bus stop I just passed?", with the agent providing relative directions based on the user's current location. * By tracking every pan and movement, the agent can provide specific details about the environment that were captured in previous steps of the virtual walk. ### User Evaluation and Practical Application Testing with blind screen reader users confirmed the system's utility in practical, real-world scenarios. * Participants successfully used the prototype to evaluate potential walking routes, identifying critical environmental features like the presence of benches or shelters at bus stops. * The study highlighted the importance of multimodal inputs—combining image recognition with structured map data—to provide a more accurate and reliable description than image analysis alone could offer. While StreetReaderAI remains a proof-of-concept, it demonstrates that the integration of multimodal LLMs and spatial data can bridge significant accessibility gaps in digital mapping. Future implementation of these technologies could transform how visually impaired individuals interact with the world, turning static street imagery into a functional tool for independent mobility and exploration.

Computer Vision AI Accessibility Gemini+5

googleSep 30, 2025Original article

Introducing interactive on-device segmentation in Snapseed (opens in new tab)

Google has introduced a new "Object Brush" feature in Snapseed that enables intuitive, real-time selective photo editing through a novel on-device segmentation technology. By leveraging a high-performance interactive AI model, users can isolate complex subjects with simple touch gestures in under 20 milliseconds, bridging the gap between professional-grade editing and mobile convenience. This breakthrough is achieved through a sophisticated teacher-student training architecture that prioritizes both pixel-perfect accuracy and low-latency performance on consumer hardware. ### High-Performance On-Device Inference * The system is powered by the Interactive Segmenter model, which is integrated directly into the Snapseed "Adjust" tool to facilitate immediate object-based modifications. * To ensure a fluid user experience, the model utilizes the MediaPipe framework and LiteRT’s GPU acceleration to process selections in less than 20ms. * The interface supports dynamic refinement, allowing users to provide real-time feedback by tracing lines or tapping to add or subtract specific areas of an image. ### Teacher-Student Model Distillation * The development team first created "Interactive Segmenter: Teacher," a large-scale model fine-tuned on 30,000 high-quality, pixel-perfect manual annotations across more than 350 object categories. * Because the Teacher model’s size and computational requirements are prohibitive for mobile use, researchers developed "Interactive Segmenter: Edge" through knowledge distillation. * This distillation process utilized a dataset of over 2 million weakly annotated images, allowing the smaller Edge model to inherit the generalization capabilities of the Teacher model while maintaining a footprint suitable for mobile devices. ### Training via Synthetic User Prompts * To make the model universally capable across all object types, the training process uses a class-agnostic approach based on the Big Transfer (BiT) strategy. * The model learns to interpret user intent through "prompt generation," which simulates real-world interactions such as random scribbles, taps, and lasso (box) selections. * During training, both the Teacher and Edge models receive identical prompts—such as red foreground scribbles and blue background scribbles—to ensure the student model learns to produce high-quality masks even from imprecise user input. This advancement significantly lowers the barrier to entry for complex photo manipulation by moving heavy-duty AI processing directly onto the mobile device. Users can expect a more responsive and precise editing experience that handles everything from fine-tuning a subject's lighting to isolating specific environmental elements like clouds or clothing.

Computer Vision AI Machine Learning On Device Ai+5

googleAug 20, 2025Original article

From massive models to mobile magic: The tech behind YouTube real-time generative AI effects (opens in new tab)

YouTube has successfully deployed over 20 real-time generative AI effects by distilling the capabilities of massive cloud-based models into compact, mobile-ready architectures. By utilizing a "teacher-student" training paradigm, the system overcomes the computational bottlenecks of high-fidelity generative AI while ensuring the output remains responsive on mobile hardware. This approach allows for complex transformations, such as cartoon style transfer and makeup application, to run frame-by-frame on-device without sacrificing the user’s identity. ### Data Curation and Diversity * The foundation of the effects pipeline relies on high-quality, properly licensed face datasets. * Datasets are meticulously filtered to ensure a uniform distribution across different ages, genders, and skin tones. * The Monk Skin Tone Scale is used as a benchmark to ensure the effects work equitably for all users. ### The Teacher-Student Framework * **The Teacher:** A large, powerful pre-trained model (initially StyleGAN2 with StyleCLIP, later transitioning to Google DeepMind’s Imagen) acts as the "expert" that generates high-fidelity visual effects. * **The Student:** A lightweight UNet-based architecture designed for mobile efficiency. It utilizes a MobileNet backbone for both the encoder and decoder to ensure fast frame-by-frame processing. * The distillation process narrows the scope of the massive teacher model into a student model focused on a single, specific task. ### Iterative Distillation and Training * **Data Generation:** The teacher model processes thousands of images to create "before and after" pairs. These are augmented with synthetic elements like AR glasses, sunglasses, and hand occlusions to improve real-world robustness. * **Optimization:** The student model is trained using a sophisticated combination of loss functions, including L1, LPIPS, Adaptive, and Adversarial loss, to balance numerical accuracy with aesthetic quality. * **Architecture Search:** Neural architecture search is employed to tune "depth" and "width" multipliers, identifying the most efficient model structure for different mobile hardware constraints. ### Addressing the Inversion Problem * A major challenge in real-time effects is the "inversion problem," where the model struggles to represent a real face in latent space, leading to a loss of the user's identity (e.g., changes in skin tone or clothing). * YouTube uses Pivotal Tuning Inversion (PTI) to ensure that the user's specific features are preserved during the generative process. * By editing images in the latent space—a compressed numerical representation—the system can apply stylistic changes while maintaining the core characteristics of the original video stream. By combining advanced model distillation with on-device optimization via MediaPipe, YouTube demonstrates a practical path for bringing heavy generative AI research into consumer-facing mobile applications.

Computer Vision AI Generative AI Knowledge Distillation+5

googleApr 30, 2025Original article

AMIE gains vision: A research AI agent for multimodal diagnostic dialogue (opens in new tab)

Google Research and DeepMind have introduced multimodal AMIE, an advanced research AI agent designed to conduct diagnostic medical dialogues that integrate text, images, and clinical documents. By building on Gemini 2.0 Flash and a novel state-aware reasoning framework, the system can intelligently request and interpret visual data such as skin photos or ECGs to refine its diagnostic hypotheses. This evolution moves AI diagnostic tools closer to real-world clinical practice, where visual evidence is often essential for accurate patient assessment and management. ### Enhancing AMIE with Multimodal Perception To move beyond text-only limitations, researchers integrated vision capabilities that allow the agent to process complex medical information during a conversation. * The system uses Gemini 2.0 Flash as its core component to interpret diverse data types, including dermatology images and laboratory reports. * By incorporating multimodal perception, the agent can resolve diagnostic ambiguities that cannot be addressed through verbal descriptions alone. * Preliminary testing with Gemini 2.5 Flash suggests that further scaling the underlying model continues to improve the agent's reasoning and diagnostic accuracy. ### Emulating Clinical Workflows via State-Aware Reasoning A key technical contribution is the state-aware phase transition framework, which helps the AI mimic the structured yet flexible approach used by experienced clinicians. * The framework orchestrates the conversation through three distinct phases: History Taking, Diagnosis & Management, and Follow-up. * The agent maintains a dynamic internal state that tracks known information about the patient and identifies specific "knowledge gaps." * When the system detects uncertainty, it strategically requests multimodal artifacts—such as a photo of a rash or an image of a lab result—to update its differential diagnosis. * Transitions between conversation phases are only triggered once the system assesses that the objectives of the current phase have been sufficiently met. ### Evaluation through Simulated OSCEs To validate the agent’s performance, the researchers developed a robust simulation environment to facilitate rapid iteration and standardized testing. * The system was tested using patient scenarios grounded in real-world datasets, including the SCIN dataset for dermatology and PTB-XL for ECG measurements. * Evaluation was conducted using a modified version of Objective Structured Clinical Examinations (OSCEs), the global standard for assessing medical students and professionals. * In comparative studies, AMIE's performance was measured against primary care physicians (PCPs) to ensure its behavior, accuracy, and tone aligned with clinical standards. This research demonstrates that multimodal AI agents can effectively navigate the complexities of a medical consultation by combining linguistic empathy with the technical ability to interpret visual clinical evidence. As these systems continue to evolve, they offer a promising path toward high-quality, accessible diagnostic assistance that mirrors the multimodal nature of human medicine.

Computer Vision AI Large Language Models AI Agents+5

googleApr 7, 2025Original article

Geospatial Reasoning: Unlocking insights with generative AI and multiple foundation models (opens in new tab)

Google Research is introducing Geospatial Reasoning, a new framework that integrates generative AI with specialized foundation models to streamline complex geographical problem-solving. By combining large language models like Gemini with domain-specific data, the initiative seeks to make large-scale spatial analysis accessible to sectors like public health, urban development, and climate resilience. This research effort moves beyond traditional data silos, enabling agentic workflows that can interpret diverse data types—from satellite imagery to population dynamics—through natural language. ### Specialized Foundation Models for Human Activity * The Population Dynamics Foundation Model (PDFM) captures the complex interplay between human behaviors and their local environments. * A dedicated trajectory-based mobility foundation model has been developed to process and analyze movement patterns. * While initially tested in the US, experimental datasets are expanding to include the UK, Australia, Japan, Canada, and Malawi for selected partners. ### Remote Sensing and Vision Architectures * New models utilize advanced architectures including masked autoencoders, SigLIP, MaMMUT, and OWL-ViT, specifically adapted for the remote sensing domain. * Training involves high-resolution satellite and aerial imagery paired with text descriptions and bounding box annotations to enable precise object detection. * The models support zero-shot classification and retrieval, allowing users to locate specific features—such as "residential buildings with solar panels"—using flexible natural language queries. * Internal evaluations show state-of-the-art performance across multiple benchmarks, including image segmentation and post-disaster damage assessment. ### Agentic Workflows and Industry Collaboration * The Geospatial Reasoning framework utilizes LLMs like Gemini to manage complex datasets and orchestrate "agentic" workflows. * These workflows are grounded in geospatial data to ensure that the insights generated are both useful and contextually accurate. * Google is collaborating with inaugural industry partners, including Airbus, Maxar, Planet Labs, and WPP, to test these capabilities in real-world scenarios. Organizations interested in accelerating their geospatial analysis should consider applying for the trusted tester program to explore how these foundation models can be fine-tuned for specific proprietary data and use cases.

Computer Vision AI Generative AI Gemini+5