google·Aug 11, 2026·3 min readCurated summary

Advancing AMIE towards expert-level audio-visual clinical consultations

AMIE (Video) is Google’s real-time audiovisual medical AI system, designed to overcome the limitations of text-only clinical conversations. Built on Gemini and Project Astra, it observes visual and auditory cues, guides patients through virtual examinations, and performs diagnostic reasoning during live consultations. In a randomized study involving 300 simulated consultations, the system was evaluated against text-only AMIE and board-certified primary care physicians. ## Why Audio-Visual Consultation Matters - Traditional text-based systems lose important clinical information, including: - Gait and visible physical symptoms - Breathing patterns and signs of distress - Vocal and auditory cues - Patient responses during physical examination maneuvers - Requiring patients to describe symptoms in writing can reduce diagnostic accuracy, particularly for people with limited digital or health literacy. - Audiovisual interaction may also improve trust, communication, and access to medical expertise. ## AMIE’s Broader Development - Earlier versions of AMIE demonstrated expert-level performance in: - Text-based diagnostic dialogue - Differential diagnosis support - Disease treatment and longitudinal management - Specialist evaluations in oncology, cardiology, and ophthalmology - Reasoning over medical images and clinical documents - Google has also explored physician oversight and real-world clinical feasibility studies. ## Asynchronous Multi-Agent Architecture AMIE (Video) divides the consultation among three agents operating in parallel: - **Talker agent** - Maintains natural, low-latency spoken conversation. - Incorporates information and recommendations from the other agents. - **Planner agent** - Performs deeper clinical reasoning in the background. - Updates differential diagnoses and management plans. - Identifies missing information and reprioritizes clinical objectives. - **Perception agent** - Continuously analyzes audio and video. - Detects non-verbal findings such as visible distress, physical signs, and auditory abnormalities. - Interprets observations in the context of the conversation. This separation allows AMIE to reason deeply without creating long conversational pauses. Automated tests indicated that the agents contributed to improvements in history-taking, clinical reasoning, treatment recommendations, communication quality, and response latency. ## Automated Evaluation Framework - Google created a taxonomy of audiovisual clinical competencies based on medical literature. - The taxonomy covered: - Non-verbal visual cues - Auditory signals - Physical examination maneuvers - The evaluation suite included: - Single-turn tests targeting specific perception and reasoning abilities - Multi-turn simulated consultations assessing complete conversational performance - Simulations injected visual findings as textual descriptions, such as a patient holding handwriting samples up to the camera. - These tests helped identify capabilities and failure modes before human evaluation. ## Randomized Video Study - The study used a synchronous video consultation interface and an Objective Structured Clinical Examination format. - It included: - 100 clinical scenarios - Five body systems: cardiopulmonary, abdominal, HEENT, neurological/psychiatric, and musculoskeletal - 15 trained patient actors - 300 standardized consultations - Three study arms were compared: - **AMIE (Video):** Real-time audiovisual consultations - **AMIE (Text):** Text-only AMIE used to isolate the value of audiovisual capabilities - **PCP (Video):** Board-certified primary care physicians using the same video interface - An independent panel of 20 experienced primary care physicians assessed the consultations using established clinical rubrics. AMIE (Video) represents a move from text-based medical dialogue toward interactive, multimodal consultations. Its multi-agent design and audiovisual perception are intended to preserve conversational responsiveness while supporting richer clinical reasoning, though the reported findings come from simulated consultations and require further validation in real-world clinical care.

Read original(opens in new tab)

Multimodal AIGeminiTelehealthMulti Agent Architecture+3

meta·Jul 15, 2026·3 min readCurated summary

Exploring Hierarchical Interest Representation For Meta Ads Deep Funnel Optimization

Hierarchical Interest Representation is an upstream embedding layer for Meta Ads that connects users’ inferred interests with advertisers, products, and services. It combines engagement data, multimodal content, graph learning, and hierarchical abstractions to address sparse deep-funnel signals and rare or unseen entities. The resulting universal embeddings and “Bag-of-Meaning” interest tokens could support retrieval, personalization, supervision, and ranking across Meta’s advertising systems. ## Purpose and Role in Deep-Funnel Optimization - The system aims to identify people with genuine latent interest in an advertiser’s offerings. - It is intended to complement systems such as Meta’s Generative Ads Model (GEM), Andromeda, and the Adaptive Ranking Model. - It uses user behavior—including scrolling, engagement, and explicit “Interested/Not interested” feedback—to infer preferences. - The broader goal is to improve discovery-oriented ad experiences and downstream conversion performance. ## Technical Challenges ### Sparse Engagement and Large-Scale Graphs - Users, advertisers, products, services, and campaigns are modeled as graph nodes. - Interactions and activities form edges in a graph spanning millions of advertisers, millions of ads, and billions of users. - Deep-funnel feedback is relatively scarce, leaving many entities with limited direct evidence. ### Long-Range Relationships - Useful signals may come from indirectly connected users and entities rather than direct interactions. - Capturing these relationships at Meta’s scale requires memory-efficient sparse attention and high-performance graph-learning algorithms. ### Dynamic and Unseen Entities - The ads ecosystem changes rapidly, while individual entities may have little historical engagement. - Representations must generalize to rare and previously unseen businesses and products. ## Core Design Properties ### Dimension Reduction - The raw graph is projected into a configurable “super-graph.” - Learned latent interest primitives act as super-nodes. - Sparse user-ad relationships become denser connections at the interest level. - The primitive graph is more stable and stationary than the constantly changing ads vocabulary. ### Knowledge Enrichment - Advertiser and product representations incorporate text, images, video, metadata, and catalog attributes. - Vision and language models process this multimodal content. - Content helps the system understand what a product or business is, not merely how users interacted with it. - This enables better generalization to new or sparsely observed entities. ### Unified Relational Representation - Users, advertisers, products, and latent interest primitives are embedded in a shared metric space. - The system can estimate: - Relationships between interest primitives - Similarity between users, ads, and products - A user’s proximity to particular interests - Which interests an advertiser or product serves - Embedding operations support both primitive-to-primitive and cluster-to-cluster relationship modeling. ### Multiple Hierarchical Granularities - Coarse representations capture dense, stable, high-level interests. - Finer representations capture sparse and specific deep-funnel intent. - Cascading hierarchical layers allow the embeddings to serve different needs across retrieval, personalization, ranking, and supervision. ## Architecture and Training - The architecture combines: - An in-house transformer-based graph learner - Bias-aware attention - Self-supervised cross-view distillation - Sparse attention for long-range graph relationships - It combines real-world semantic knowledge with users’ temporal engagement histories. - The model learns multi-hierarchical interest representations across a large graph. - Training is performed end-to-end on real Meta Ads data involving billions of interactions. ## Outputs and Potential Applications - Universal embeddings for users and ads entities. - “Bag-of-Meaning” interest tokens representing latent interests at different granularities. - Potential uses include: - Ad retrieval - Personalization - Ranking - Specialized ranking architectures - Training supervision - Cross-entity similarity and discovery Hierarchical Interest Representation is best understood as shared infrastructure for Meta’s ads recommendation stack. By combining sparse behavioral evidence with multimodal world knowledge and hierarchical graph abstractions, it could make deep-funnel optimization more robust, especially for specialized, rare, or newly introduced products and advertisers.

Read original(opens in new tab)

Multimodal AILarge Language ModelsTransformerEmbeddings+4

aws·May 14, 2026·2 min readCurated summary

Amazon Bedrock introduces new advanced prompt optimization and migration tool | Amazon Web Services

Amazon Bedrock Advanced Prompt Optimization automatically rewrites prompts to improve performance or support migration between models. It compares original and optimized prompts across up to five Bedrock models using example inputs, reference answers, and configurable evaluation methods. The tool supports text and multimodal inputs, then reports quality scores, estimated costs, and latency. ## Prompt Optimization and Model Migration - Users can optimize prompts for their current model or compare a baseline model with up to four alternatives. - Bedrock runs prompts against supplied examples, evaluates the responses, and iteratively rewrites the prompt. - Results include: - Original and optimized prompt templates - Evaluation scores - Cost estimates - Latency information - The process can identify regressions in established use cases and improve weaker tasks. ## Multimodal and Dataset Support - Prompt templates and evaluation examples are supplied in JSONL format. - Each JSON object must appear on a single line. - Supported multimodal inputs include: - PNG and JPG images - PDF documents - Files can be uploaded directly or imported from Amazon S3. - Optimization results and evaluation data can also be written to an S3 location. ## Evaluation Methods Users can select one evaluation approach per prompt template: - **Lambda-based evaluation** - Uses custom Python scoring logic for metrics such as accuracy, F1, execution accuracy, or structured JSON matching. - The function compares model outputs with reference responses. - **LLM-as-a-judge** - Uses a custom rubric with named metrics, structured instructions, and rating scales. - Bedrock’s judge model evaluates responses and provides scores with reasoning. - Claude Sonnet 4.6 is the default judge, though other supported judge models can be selected. - **Natural-language steering criteria** - Describes desired qualities such as brand voice, formatting, or safety constraints. - A default LLM judge evaluates responses holistically against those criteria. ## Getting Started and Availability - Start from **Create prompt optimization** in the Amazon Bedrock console or use the `CreateAdvancedPromptOptimizationJob` API. - Optimization is available in selected AWS Regions across North America, Europe, Asia-Pacific, Canada, and South America. - Pricing is based on Bedrock inference tokens consumed during optimization, using standard per-token inference rates. The tool is most useful when teams need measurable prompt improvements or want to migrate applications to another model. Preparing representative evaluation examples and choosing an appropriate scoring method will be essential for producing reliable optimization results.

Read original(opens in new tab)

Multimodal AILarge Language ModelsAmazon BedrockAmazon S3+3

kakao·May 5, 2026·3 min readCurated summary

Bringing a Voice AI Model to Production: The Journey of Optimizing Kanana-O Serving

Kanana-O is a multimodal model that understands text, images, and audio, then responds with text and speech. Deploying it for real-time voice conversations required solving problems that do not arise during model training, including low first-response latency, concurrent users, streaming across multiple models, and uneven GPU memory demands. Kakao built the specialized Kanana-Omni Server, achieving 1.6× the throughput of vLLM-Omni at 64 concurrent users. ## Kanana-O’s Three-Stage Architecture - **Thinker** processes multimodal inputs and generates text. - **Talker** converts Thinker’s text embeddings into sequential speech tokens. - **VoiceBox** combines speech tokens into audible audio waveforms. - In production, these components must operate concurrently rather than sequentially to deliver audio within hundreds of milliseconds. ## Why a Specialized Serving Server Was Needed - Thinker passes hidden-state embeddings directly to Talker rather than ordinary token IDs. - These high-dimensional tensors must be transferred continuously, making serialization or CPU copies too expensive. - Talker produces speech tokens step by step, while VoiceBox waits for enough tokens to form larger audio chunks. - Talker also combines speaker embeddings, Thinker outputs, and its own accumulated audio embeddings, creating an input structure unlike standard autoregressive decoding. - These constraints made a custom server more suitable than general-purpose frameworks. ## Zero-Copy Data Transfer - The server preallocates shared-memory blocks during startup. - Thinker writes tensors into an available block, while Talker receives only metadata such as the block identifier and byte size. - This avoids repeated allocation, copying, and serialization. - For GPU tensors on the same node, CUDA IPC transfers data directly between GPU processes, avoiding Device→Host→Device movement. ## Cascaded Streaming Pipeline - Thinker, Talker, and VoiceBox run as overlapping asynchronous stages. - Thinker can send its first output chunk while Talker processes earlier chunks and VoiceBox synthesizes audio from still earlier ones. - Talker buffers speech tokens until VoiceBox has enough data to create an audio chunk. - This pipelining significantly reduces the time before the user hears the first response. ## Process Isolation and Fault Containment - Thinker and Talker each run their own vLLM engine in separate processes. - This avoids conflicts between CUDA contexts, model memory, KV caches, and schedulers. - Processes are started with `spawn` rather than `fork`, preventing inherited CUDA state from causing corruption. - If one component fails, such as Thinker running out of memory, the other components and the API server can continue operating and be restarted independently. ## Continuous Batching with vLLM - Manually batching requests is difficult because multimodal inputs and accumulated Talker embeddings vary in size. - The server submits requests rapidly and delegates batch construction to vLLM’s continuous-batching scheduler. - Each request runs as an independent asynchronous generation task. - vLLM combines requests internally during forward passes, while request IDs ensure each task receives only its own streamed output. - This improves GPU utilization without requiring custom synchronization and padding logic. ## Single FastAPI Worker and Asynchronous Execution - Multiple Uvicorn workers would load separate copies of the vLLM engines, multiplying GPU memory usage and model-loading costs. - Therefore, the server uses `workers=1`. - Since a blocking operation would otherwise stall every connected user, the entire request path—from the API endpoint through final audio generation—is designed around `async`/`await`. - Keeping the pipeline non-blocking allows one worker to accept and progress many concurrent requests. Kakao’s main recommendation is to design serving infrastructure around the model’s actual dataflow rather than forcing it into a generic framework. For complex multimodal pipelines, zero-copy transfers, asynchronous cascaded streaming, process isolation, and engine-level continuous batching can be more important than simply scaling API workers.

Read original(opens in new tab)

Multimodal AIVllmContinuous BatchingFastapi+3

kakao·Apr 21, 2026·3 min readCurated summary

Kanana Scala 1st Seminar On-site Sketch

Kakao’s first Kanana Scholar seminar brought together seven leading AI professors and Kakao researchers to discuss the company’s independent AI strategy. Kakao presented its from-scratch Kanana foundation models, emphasizing data efficiency, Korean-language capability, and multimodal processing. The discussion concluded that Kakao should focus less on generic benchmark scores and more on technology sovereignty, personalized agents, and practical execution in real services. ## Kanana Foundation Models - Kakao is developing its own foundation-model lineup to strengthen competitiveness and reduce dependence on overseas providers. - Kanana reportedly achieved strong performance using 11 trillion training tokens, compared with 23 trillion tokens for a similarly sized global-target model. - Kakao attributed this efficiency to the quality and refinement of its training data. - The company also demonstrated **Kanana-o**, an omni model capable of processing text, images, and audio in real time. - The model handled emotional speech and multi-speaker conversations naturally, receiving praise for its Korean fluency. ## Technology Sovereignty and Customization - Kakao argued that proprietary models protect it from external risks such as changing licensing policies and closed technologies. - Owning the technology enables Kakao to build efficient, customized models optimized for its services. - Participating professors agreed that control over Korean cultural context and local issues is essential for technological sovereignty. - They viewed an independent model as a strategic asset for long-term service stability. ## Digital World Models and Personalized Agents - Kakao aims to understand users’ behavioral context within KakaoTalk and provide highly personalized assistance. - On-device AI could protect private conversations while allowing agents to respond immediately to user needs. - The professors suggested expanding the idea of “physical AI” into a **digital world model** that predicts interactions and causal relationships across a platform. - This direction could create an area of AI differentiation uniquely suited to Kakao’s ecosystem. ## Evaluating Practical Agentic Intelligence - Kakao is prioritizing AI systems that can create multi-step plans, call necessary tools, and complete tasks independently. - It plans to use an internally developed orchestration benchmark to evaluate real-world problem-solving ability. - The professors cited Claude as an example of how users perceive intelligence through successful completion of complex requests, not merely high benchmark scores. - They recommended competing through practical execution in real service environments rather than focusing only on text-generation performance. ## Industry-Academic Cooperation - Kakao plans to explore GPU support for university research labs and undergraduate AI clubs. - Possible support could include credits, project-based resources, and other forms of infrastructure assistance. - The seminar marked the beginning of a broader collaboration aimed at advancing Korea’s AI ecosystem and developing future talent. Kakao’s recommended path is to combine proprietary, efficient models with privacy-preserving personalization and strong agentic execution. Success will depend on how effectively Kanana turns technical depth into useful intelligence that users can experience in everyday services.

Read original(opens in new tab)

Multimodal AIAIAI AgentsOrchestration+4

google·Feb 5, 2026·3 min readCurated summary

How AI tools can redefine universal design to increase accessibility

Google Research proposes Natively Adaptive Interfaces (NAI), a framework that uses multimodal and agentic AI to make interfaces adapt to individual users rather than forcing everyone into a fixed design. Developed through co-design with disability communities, NAI aims to reduce the accessibility gap by embedding assistive capabilities directly into products. Early prototypes suggest that personalized, context-aware interfaces can improve experiences for disabled users while also benefiting the broader population. ## Community-led co-design - Google follows the principle “Nothing About Us, Without Us,” involving people with disabilities as co-designers from the beginning. - Partnerships include RIT/NTID, The Arc of the United States, RNID, and Team Gleason. - These collaborations focus on real-world barriers and recognize the expertise of disability communities. - The approach also aims to create employment and economic opportunities for people who help shape the technology. ## Moving from reactive accessibility to adaptive interfaces - Google identifies an “accessibility gap” between the release of new features and the development of compatible assistive tools. - NAI addresses this by making accessibility native to the interface instead of adding it afterward. - Static navigation is replaced with dynamic, agent-driven modules that can interpret context and adjust the experience. ## Multi-system agents - An Orchestrator maintains shared context and delegates tasks to specialized sub-agents. - A Summarization Agent breaks down complex documents and assigns subtasks to expert agents. - A Settings Agent dynamically adjusts interface elements such as text size. - This structure lets users accomplish tasks without navigating complicated menus or searching for the right control. ## Multimodal interaction - Gemini-based prototypes combine voice, vision, and text rather than limiting accessibility to text-to-speech. - Live video can be converted into interactive audio descriptions. - Users can ask follow-up questions about specific visual details as events unfold. - Conversational interaction provides situational awareness and may reduce cognitive load. ## Proven prototypes - **StreetReaderAI** - Supports blind and low-vision users navigating physical spaces. - Combines an AI Describer that analyzes visual and geographic information with an AI Chat system for questions. - Maintains context so users can ask about previously encountered locations, such as the position of a bus stop. - **Multimodal Agent Video Player (MAVP)** - Makes audio description interactive rather than static. - Users can change the level of detail or ask questions during playback. - Uses an offline “dense index” of visual descriptions and retrieval-augmented generation (RAG) for fast responses. - **Grammar Laboratory** - Developed by RIT/NTID with Google.org support for American Sign Language and English learners. - Provides grammar instruction through ASL videos, English captions, spoken narration, and written transcripts. - Uses adaptive AI to customize lessons according to each student’s language preferences and interactions. ## The curb-cut effect - Accessibility features designed for people with significant constraints can benefit many other users. - Voice interfaces created for blind users may help sighted people who are multitasking. - AI synthesis and learning tools designed for people with learning disabilities can also support users who want information presented more clearly or flexibly. - NAI therefore treats accessibility as a source of better universal design, not as a specialized add-on. NAI’s central recommendation is to build accessibility into interfaces from the start, using multimodal AI, persistent context, and community-led design. The most effective systems will adapt to users while remaining accountable to the people whose needs they are intended to serve.

Read original(opens in new tab)

Multimodal AIAI AgentsGenerative AIAccessibility+4

dropbox·Jan 28, 2026·3 min readCurated summary

Engineering VP Josh Clemm on how we use knowledge graphs, MCP, and DSPy in Dash

Dropbox VP Josh Clemm argues that useful workplace AI requires a unified context engine capable of securely understanding and retrieving information across many SaaS applications. Dropbox Dash combines custom connectors, multimodal content processing, knowledge graphs, hybrid search, and personalized access-control-aware ranking to make that possible. Clemm favors indexed retrieval over purely federated approaches because preprocessing enables richer context, faster search, and company-wide access, though it requires substantial engineering and infrastructure. ## Building Dash’s Context Engine - Custom connectors crawl third-party applications while handling: - Rate limits - API differences - Application-specific permissions and ACLs - Incoming content is normalized, often into Markdown, and enriched with: - Titles and metadata - Extracted links - Embeddings - Other key information - Different media types require different processing: - Documents can be text-extracted and indexed. - Images may require CLIP or multimodal models. - PDFs combine text, figures, and other elements. - Audio is transcribed. - Videos may require scene-by-scene multimodal analysis when dialogue is insufficient. - Dash models relationships between content as a knowledge graph: - Meetings can connect to documents, participants, transcripts, and previous notes. - Cross-application relationships provide richer context for search and agents. - Data is stored in secure systems using: - BM25 lexical search - Dense-vector storage - Hybrid retrieval - Multiple ranking stages personalize results and enforce user-specific permissions. ## Indexed Retrieval Versus Federated Retrieval - Federated retrieval queries external systems at runtime. - Its advantages include: - Fast initial implementation - Minimal storage requirements - Relatively fresh data - Easy addition of MCP servers and connectors - Its drawbacks include: - Inconsistent API speed, quality, and ranking - Limited access to company-wide content - Expensive post-processing and reranking - Large token usage when agents reason over returned results - Indexed retrieval preprocesses content during ingestion. - Its advantages include: - Access to shared company connectors - Enriched datasets created offline - Faster queries - More opportunities for recall and ranking experiments - Its costs include: - Significant custom connector development - Freshness challenges - High hosting costs - Difficult storage and architecture choices involving vector search, BM25, hybrid retrieval, or graph RAG. ## Making MCP Practical at Scale - MCP can simplify tool integration, but tool definitions consume substantial context-window space. - Large tool descriptions and retrieval results contribute to context rot and reduce agent effectiveness. - Dash aims to limit context usage to roughly 100,000 tokens. - MCP-based agents can also be slow: simple queries may take up to 45 seconds, while direct index retrieval returns results within seconds. - Dropbox’s approach is to wrap its index in a consolidated “super tool,” reducing the need to expose many separate tools to the agent. ## Broader AI Engineering Practices - The talk also covers Dropbox’s use of: - LLMs as evaluators or judges - Prompt optimization with DSPy - Tool calling and MCP design - These techniques complement the underlying context engine rather than replacing the indexing, enrichment, graph modeling, and permission systems required for reliable workplace AI. A practical takeaway is that organizations building AI over proprietary data should treat retrieval as a full data-platform problem. Start with robust connectors and permissions, enrich content before retrieval, model relationships across sources, and use MCP selectively where indexed retrieval can provide faster and more controlled results.

Read original(opens in new tab)

Multimodal AIModel Context ProtocolRAGDspy+4

google·Jan 22, 2026·3 min readCurated summary

Small models, big results: Achieving superior intent extraction through decomposition

Small multimodal models can outperform much larger models at extracting user intent from UI interaction trajectories when the task is decomposed. Google’s approach first summarizes each screen and interaction, then derives an overall intent from those summaries. This enables accurate, faster, and more privacy-preserving on-device understanding without sending sensitive UI data to servers. ## Why On-Device Intent Understanding Matters - Understanding what users are doing across mobile and web interfaces can help agents anticipate useful next actions. - Large multimodal models perform well but often require server-side processing, introducing latency, cost, and privacy risks. - The goal is to make intent understanding practical with smaller models running directly on devices. ## Two-Stage Intent Extraction ### Screen and Interaction Summaries For each interaction, a small multimodal model examines a sliding window of three screens: the previous, current, and next screens. It generates information about: - Salient context on the current screen. - Actions the user just performed. - A speculation about what the user is trying to accomplish. This converts raw screenshots and actions into structured, manageable event summaries. ### Intent Extraction from Summaries A fine-tuned small model then processes the sequence of summaries and produces a single concise intent statement. The authors improve this stage through: - **Fine-tuning:** Training on examples of high-quality intent statements helps the model retain relevant details and discard noise. - **Label preparation:** Training intents are stripped of details absent from the summaries, reducing hallucinated information. - **Removing speculation:** Speculative fields help create richer individual summaries but are excluded from the second stage because they can confuse intent extraction. ## Evaluation with Atomic Facts - The authors use the Bi-Fact evaluation method to compare predicted intents with reference intents. - Each intent is split into indivisible “atomic facts,” such as “a one-way flight” or the separate origin and destination in a flight request. - The method measures: - **Recall:** How many reference facts were captured. - **Precision:** How many predicted facts are supported by the reference. - **F1:** The balance between precision and recall. - Tracking facts through both stages also reveals where details are lost or hallucinated. ## Results - The decomposed method outperformed chain-of-thought prompting and end-to-end fine-tuning. - Improvements held across both mobile and web interaction trajectories. - Results were consistent across Gemini and Qwen2 base models. - Gemini 1.5 Flash 8B achieved results comparable to Gemini 1.5 Pro while offering substantially lower cost and faster processing. - On mobile data, the small-model approach approached the performance of the larger Gemini Pro model. The study suggests that decomposing intent understanding into local summarization followed by sequence-level extraction is an effective path toward accurate, private, and efficient on-device assistants. As mobile hardware and small models improve, this technique could support a broad range of assistive features.

Read original(opens in new tab)

Multimodal AIGeminiNatural Language ProcessingFine Tuning+4

pinterest·Jan 13, 2026·3 min readCurated summary

PinLanding: Turn Billions of Products into Instant Shopping Collections with Multimodal AI

PinLanding is a production pipeline for turning billions of products into searchable shopping collections using multimodal AI. Rather than relying mainly on historical queries or manual curation, it derives structured product attributes from images and metadata, then aligns those attributes with real user search behavior. The system combines multimodal LLMs, embedding-based consolidation, a CLIP-style classifier, and distributed infrastructure to produce scalable, precise shopping feeds. ## Understanding Shopping Intent - Pinterest analyzes search history, autocomplete use, filters, and browsing paths to estimate shopping demand. - Existing systems handle high-volume queries such as “black cocktail dress” well, but provide weaker coverage for: - Long-tail queries - Conversational requests - Contextual intents such as “what to wear for an Italian summer vacation” - The analysis identifies: - Product areas with strong demand but poor collection coverage - Important attribute dimensions, including color, occasion, style, fit, price, and brand - The goal is to expand and improve collection coverage, not replace query understanding. ## Generating and Curating Shopping Topics - Each product is represented by an image plus metadata such as title, description, merchant tags, and price. - A vision-language model generates normalized key-value attributes rather than free-form descriptions. - Raw model output has high recall but produces: - Excessively specific attributes - Near-duplicates such as “boho,” “bohemian,” and “boho-chic” - Sparse attributes that apply to very few products - PinLanding builds a compact vocabulary through: - Frequency filtering to remove rarely useful attributes - Embedding-based clustering to merge semantically similar terms - Manual and LLM-assisted review - An LLM judge evaluates generated topics for semantic coherence, realistic shopping intent, and alignment with natural search phrasing. ## Scalable Attribute Assignment - Running the vision-language model over every product is too expensive and operationally fragile. - PinLanding trains a CLIP-inspired dual encoder: - One encoder embeds product images and text - Another embeds attribute phrases - Matching product-attribute pairs are trained as positives, while mismatches are negatives - A bidirectional contrastive loss aligns related products and attributes. - At inference, products and attributes are embedded once, and attributes are assigned when similarity exceeds a calibrated threshold. - This produces fewer distinct attributes while increasing the average number assigned to each product, creating a denser and more consistent attribute graph. ## Distributed Feed Construction - Ray handles large-scale batch inference across millions of products and topics. - The pipeline separates: - CPU-based image and metadata loading, tokenization, and serialization - GPU-based classifier inference - Streaming allows preprocessing and inference to overlap, while heterogeneous CPU and GPU clusters can scale independently. - The classifier pipeline reportedly completes in about 12 hours using eight NVIDIA A100 GPUs, at an estimated cost of roughly $500 per training run. - Feed construction uses approximate-nearest-neighbor techniques and strict attribute matching. - Topics are represented as attribute tuples, such as: - Category: dress - Color: yellow - Season: summer - Occasion: party - Apache Spark computes topic-product relevance using shared attributes and confidence weights, with partitioning and overlap filters reducing unnecessary candidate comparisons. The core recommendation is to combine user-behavior signals with content-first multimodal modeling. This approach can expand shopping coverage into conversational and long-tail intents while remaining practical through attribute consolidation, contrastive retrieval, and distributed inference.

Read original(opens in new tab)

Multimodal AIApache SparkContrastive LearningClip+4

google·Jan 12, 2026Original article

Next generation medical image interpretation with MedGemma 1.5 and medical speech to text with MedASR (opens in new tab)

Google Research has introduced MedGemma 1.5 4B and MedASR, expanding its suite of open medical AI models to support more complex clinical workflows. These updates significantly enhance the interpretation of high-dimensional imaging and medical speech-to-text, providing a compute-efficient foundation for healthcare developers to build upon. By maintaining an open-access model available on Hugging Face and Vertex AI, Google aims to accelerate the integration of multimodal AI into real-world medical applications. ### Multimodal Advancements in MedGemma 1.5 The latest update to the MedGemma 4B model focuses on high-dimensional and longitudinal data, moving beyond simple 2D image interpretation. * **3D Medical Imaging:** The model now supports volumetric representations from CT scans and MRIs, as well as whole-slide histopathology imaging. * **Longitudinal Review:** New capabilities allow for the review of chest X-ray time series, helping clinicians track disease progression over time. * **Anatomical Localization:** Developers can use the model to identify and localize specific anatomical features within chest X-rays. * **Document Understanding:** Enhanced support for extracting structured data from complex medical lab reports and documents. * **Edge Capability:** The 4B parameter size is specifically designed to be small enough to run offline while remaining accurate enough for core medical reasoning tasks. ### Medical Speech-to-Text with MedASR MedASR is a specialized automated speech recognition (ASR) model designed to bridge the gap between clinical dialogue and digital documentation. * **Clinical Dictation:** The model is specifically fine-tuned for medical terminology and the unique nuances of clinical dictation. * **Integrated Reasoning:** MedASR is designed to pair seamlessly with MedGemma, allowing transcribed text to be immediately processed for advanced medical reasoning or summarization. * **Accessibility:** Like other HAI-DEF models, it is free for research and commercial use and hosted on both Hugging Face and Google Cloud’s Vertex AI. ### Performance Benchmarks and Community Impact Google is incentivizing innovation through improved performance metrics and community-driven challenges. * **Accuracy Gains:** Internal benchmarks show MedGemma 1.5 improved disease-related CT classification by 3% and MRI classification by 14% compared to the previous version. * **MedGemma Impact Challenge:** A Kaggle-hosted hackathon with $100,000 in prizes has been launched to encourage developers to find creative applications for these multimodal tools. * **Model Collection:** The update complements existing tools like the MedSigLIP image encoder and the larger MedGemma 27B model, which remains the preferred choice for complex, text-heavy medical applications. Developers and researchers are encouraged to utilize MedGemma 1.5 for tasks requiring efficient, offline multimodal processing, while leveraging MedASR to automate clinical documentation. By participating in the MedGemma Impact Challenge, the community can help define the next generation of AI-assisted medical diagnostics and workflows.

Multimodal AIAIGenerative AISpeech To Text+5

kakao·Jan 4, 2026Original article

Kakao’s “ (opens in new tab)

Kakao's Kanana-v-4b-hybrid is a multimodal language model designed to transcend simple image-to-text conversion by integrating logical reasoning and self-verification directly into its response process. By employing a hybrid architecture that handles both intuitive dialogue and complex visual reasoning within a single model, it achieves high accuracy and reliability for sophisticated tasks. This approach allows the model to maintain consistency in user experience while excelling in Korean-specific contexts, as evidenced by its record-breaking 92.8 score on the KoNET evaluation. ### Integrated Hybrid Architecture * Consolidates intuitive tasks (like OCR and summarization) and logical tasks (complex reasoning) into a single model to reduce system complexity and maintenance costs. * Eliminates the need for external routing between specialized models, ensuring a consistent tone, response format, and safety policy throughout a single conversation session. * Utilizes a refined training recipe that balances data ratios and visual reasoning training to ensure that improvements in multimodal understanding benefit all types of user queries. ### Visual Reasoning and Self-Reflection * Follows a natural logic flow: synthesizing information from images and text, applying conditions, verifying candidates, and finally concluding the response. * Features a "Reflection" mechanism where the model actively monitors its own thought process to catch "small but fatal" errors, such as calculation mistakes or missed constraints. * Excels in high-stakes visual tasks like receipt auditing, table filtering, and mathematical problem-solving by double-checking intermediate results against original image data. ### Native Korean Logical Processing * Prioritizes "thinking in Korean" to accurately preserve the nuances of complex constraints, such as "except for X" or "only in cases of Y," which are often lost during internal translation. * Develops a native Korean Rationale process to prevent logical drift, ensuring that the internal reasoning steps remain perfectly aligned with the linguistic structure of the user's query. * Addresses the difficulty of processing information scattered throughout Korean-language documents or exam papers by synthesizing data without language-conversion overhead. Kanana-v-4b-hybrid marks a shift toward "verifiable AI" that provides evidence-based answers rather than just plausible text. For applications in education, finance, or complex document processing, this model offers a blueprint for building trust through transparent reasoning and self-correction.

Multimodal AIKananaChain Of ThoughtOcr+3

google·Dec 17, 2025Original article

Google Research 2025: Bolder breakthroughs, bigger impact (opens in new tab)

Google Research in 2025 has shifted toward an accelerated "Magic Cycle" that rapidly translates foundational breakthroughs into real-world applications across science, society, and consumer products. By prioritizing model efficiency, factuality, and agentic capabilities, the organization is moving beyond static text generation toward interactive, multi-modal systems that solve complex global challenges. This evolution is underpinned by a commitment to responsible AI development, ensuring that new technologies like quantum computing and generative UI are both safe and culturally inclusive. ## Enhancing Model Efficiency and Factuality * Google introduced new efficiency-focused techniques like block verification (an evolution of speculative decoding) and the LAVA scheduling algorithm, which optimizes resource allocation in large cloud data centers. * The Gemini 3 model achieved state-of-the-art results on factuality benchmarks, including SimpleQA Verified and the newly released FACTS benchmark suite, by emphasizing grounded world knowledge. * Research into Retrieval Augmented Generation (RAG) led to the development of the LLM Re-Ranker in Vertex AI, which helps models determine if they possess sufficient context to provide accurate answers. * The Gemma open model expanded to support over 140 languages, supported by the TUNA taxonomy and the Amplify initiative to improve socio-cultural intelligence and data representation. ## Interactive Experiences through Generative UI * A novel implementation of generative UI allows Gemini 3 to dynamically create visual interfaces, web pages, and tools in response to user prompts rather than providing static text. * This technology is powered by specialized models like "Gemini 3-interactive," which are trained to output structured code and design elements. * These capabilities have been integrated into AI Mode within Google Search, allowing for more immersive and customizable user journeys. ## Advanced Architectures and Agentic AI * Google is exploring hybrid model architectures, such as Jamba-style models that combine State Space Models (SSMs) with traditional attention mechanisms to handle long contexts more efficiently. * The development of agentic AI focuses on models that can reason, plan, and use tools, exemplified by Project Astra, a prototype for a universal AI agent. * Specialized models like Gemini 3-code have been optimized to act as autonomous collaborators for software developers, assisting in complex coding tasks and system design. ## AI for Science and Planetary Health * In biology, research teams utilized AI to map human heart and brain structures and employed RoseTTAFold-Diffusion to design new proteins for therapeutic use. * The NeuralGCM model has revolutionized Earth sciences by combining traditional physics with machine learning for faster, more accurate weather and climate forecasting. * Environmental initiatives include the FireSat satellite constellation for global wildfire detection and the expansion of AI-driven flood forecasting and contrail mitigation. ## Quantum Computing and Responsible AI * Google achieved significant milestones in quantum error correction, developing low-overhead codes that bring the industry closer to a reliable, large-scale quantum computer. * Security and safety remain central, with the expansion of SynthID—a watermarking tool for AI-generated text, audio, and video—to help users identify synthetic content. * The team continues to refine the Secure AI Framework (SAIF) to defend against emerging threats while promoting the safe deployment of generative media models like Veo and Imagen. To maximize the impact of these advancements, organizations should focus on integrating agentic workflows and RAG-based architectures to ensure their AI implementations are both factual and capable of performing multi-step tasks. Developers can leverage the Gemma open models to build culturally aware applications that scale across diverse global markets.

Multimodal AIAILarge Language ModelsGenerative AI+5

pinterest·Dec 8, 2025·4 min readCurated summary

How Pinterest Built a Real‑Time Radar for Violative Content using AI

Pinterest built an AI-assisted prevalence measurement system to estimate how often users actually see policy-violating content, rather than relying only on user reports. The system samples daily impressions, uses production risk scores to improve efficiency, labels content with a multimodal LLM, and applies statistical reweighting to preserve unbiased estimates. This enables daily, segmented monitoring with substantially lower cost and latency than human-only review. ## Why Prevalence Matters - User reports miss important harms because: - Some sensitive issues, such as self-harm, are under-reported. - Users seeking harmful content may not report it. - Rare policy categories provide too few reports for reliable trend detection. - Human review of reports is expensive and slow. - Prevalence measures exposure: the share of total views directed to violating content. - This helps Pinterest identify under-reported harms, evaluate interventions, and detect changes earlier. - Human-only prevalence studies were previously conducted only about every six months and required multiple reviewers plus adjudication. ## What Pinterest Measures - Daily prevalence is calculated as: - **Views of content violating a policy ÷ total views** - For example, 10 violating views in a sample of 100,000 produces an estimated prevalence of 0.01%. - Results include 95% confidence intervals to communicate statistical precision. - Metrics can be segmented by: - Policy area, such as Adult Content, Self-harm, or Graphic Violence - Sub-policy, such as nudity versus explicit sexual content - Surface, including Homefeed, Search, and Related Pins - Content age, geography, and user-age groups where relevant ## Risk-Aware, Unbiased Sampling - Pinterest samples from the daily user-impressions stream. - Production enforcement risk scores are used to prioritize likely high-risk and high-exposure content, but they are not treated as labels or eligibility rules. - Missing scores are replaced with the day’s median so that new content remains eligible. - Weighted reservoir sampling approximates probability-proportional-to-size sampling, considering impressions and risk scores. - Inverse-probability weighting removes the bias introduced by risk-based sampling, ensuring estimates represent impressions rather than model thresholds. - Pinterest uses Hansen–Hurwitz ratio estimators for sampling with replacement and Horvitz–Thompson ratio estimators for sampling without replacement. - Pure random sampling is also available for validation studies. ## LLM-Based Labeling - A multimodal LLM analyzes sampled content using both images and text. - Prompts are reviewed by policy subject-matter experts and can return structured label hierarchies such as `safe`, `not_safe`, and `unsure`. - Each decision records: - The label and brief rationale - Policy version - Prompt and model identifiers - Token usage and run cost - Human validation is performed on strategically selected samples to identify edge cases and AI blind spots. - The LLM is tested against human-reviewed gold sets before launch and periodically afterward to detect drift. - The workflow is reportedly 15 times faster and far cheaper than human-only labeling while maintaining comparable decision quality and statistical governance. ## Production System and Monitoring - Inputs include entity-by-day engagement data such as impressions, clicks, hides, and reports, alongside current production risk scores. - The system stores prevalence estimates, sampling weights, labels, diagnostics, and lineage for audits. - Dashboards display: - Daily prevalence and 95% confidence intervals - Confidence-interval width and effective sample size - Sample positive rate - Risk-score distributions - Prompt, model, taxonomy, and metric versions - Teams can pivot results by policy, sub-policy, and surface. - Validation samples and run-health information help monitor both statistical quality and operational reliability. Pinterest’s approach combines probability sampling, inverse-probability estimation, and continuously calibrated multimodal AI labeling to create a daily radar for harmful exposure. The practical recommendation is to use AI to scale measurement, but retain rigorous sampling, human validation, confidence intervals, and full model and policy lineage so that faster estimates remain trustworthy.

Read original(opens in new tab)

Multimodal AILarge Language ModelsMachine LearningTrust And Safety+3

pinterest·Dec 4, 2025·3 min readCurated summary

On the (re)-prioritization of open-source AI

Pinterest argues that AI competition is shifting beyond ever-larger proprietary models. Open-source models now deliver comparable quality at a fraction of the cost, while compact models fine-tuned for specific tasks can outperform general-purpose systems. The company’s strategy is to combine open-source models with Pinterest-specific data, internal systems, and deep product integration. ## Open-Source Models and Cost Efficiency - Pinterest reports achieving performance comparable to leading proprietary AI models at less than 10% of the cost. - The company is shifting more investment toward fine-tuned open-source models, especially for visual and multimodal applications. - As core LLM architectures become increasingly commoditized, competitive advantage is moving toward: - Domain-specific data - Personalization - Product integration - End-to-end system optimization ## Choosing What to Build, Buy, or Adapt Pinterest evaluates foundation-model strategy by modality: - **User modeling and recommendation** - These systems are tightly coupled to a product’s behavior and are generally built internally. - Pinterest uses long-term user-action sequences and a graph containing hundreds of billions of user, board, and content nodes. - Examples include PinFM for representation learning and PinRec for generative recommendations. - **Visual models** - Pinterest largely trains visual encoders and diffusion models in-house. - Its visual-search data and image-board collections provide the weakly supervised datasets needed for large-scale training. - Internal models benefit from Pinterest’s specialized visual domain. - **Text models** - Pinterest has historically relied more on open-source and proprietary third-party LLMs. - Progress in reasoning and language modeling depends heavily on enormous datasets and compute resources, making external models practical. ## Domain-Specific Data as the Differentiator - Open-source multimodal architectures are narrowing the capability gap with proprietary models. - Pinterest’s experience reflects an older machine-learning pattern: model architectures become broadly available, while value comes from specialized data and fine-tuning. - Its visual encoders, including UVE and PinCLIP, improved retrieval by training on Pinterest image and visual-search data rather than using generic embeddings. - Pinterest Canvas similarly adapts an internally trained diffusion model for image editing and enhancement, outperforming larger general-purpose visual-generation models in those use cases. ## Pinterest Assistant and Specialized Tools - Pinterest Assistant combines: - Multimodal retrieval systems - Recommendation services - Specialized generative models - A core multimodal LLM - Most recommendation and agentic capabilities are handled by Pinterest-native tools built on its user and visual foundation models. - The central LLM acts primarily as an intelligent router, handling query understanding, planning, and tool calling rather than performing every task itself. - This architecture allows Pinterest to improve the overall product by optimizing smaller, specialized components instead of relying solely on a larger general-purpose model. Pinterest’s recommendation is to use open-source models as adaptable building blocks, then differentiate through proprietary data, specialized models, and tight integration with the product. The most effective AI systems may therefore be smaller, cheaper, and more purpose-built than frontier general-purpose models.

Read original(opens in new tab)

Multimodal AILarge Language ModelsGenerative AIFine Tuning+4

naver·Dec 3, 2025Original article

Naver TV (opens in new tab)

Naver’s VLOps framework introduces an event-driven approach to MLOps, designed to overcome the rigidity of traditional pipeline-based systems like Kubeflow. By shifting from a monolithic pipeline structure to a system governed by autonomous sensors and typed messages, Naver has achieved a highly decoupled and scalable environment for multimodal AI development. This architecture allows for seamless functional expansion and cross-cloud compatibility, ultimately simplifying the transition from model training to large-scale evaluation and deployment. ### Event-Driven MLOps Architecture * Operations such as training, evaluation, and deployment are defined as "Typed Messages," which serve as the primary units of communication within the system. * An "Event Sensor" acts as the core logic hub, autonomously detecting these messages and triggering the corresponding tasks without requiring a predefined, end-to-end pipeline. * The system eliminates the need for complex version management of entire pipelines, as new features can be integrated simply by adding new message types. * This approach ensures loose coupling between evaluation and deployment systems, facilitating easier maintenance and infrastructure flexibility. ### Omni-Evaluator and Unified Benchmarking * The Omni-Evaluator serves as a centralized platform that integrates various evaluation engines and benchmarks into a single workflow. * It supports real-time monitoring of model performance, allowing researchers to track progress during the training and validation phases. * The system is designed specifically to handle the complexities of Multimodal LLMs, providing a standardized environment for diverse testing scenarios. * User-driven triggers are supported, enabling developers to initiate specific evaluation cycles manually when necessary. ### VLOps Dashboard and User Experience * The VLOps Dashboard acts as a central hub where users can manage the entire ML lifecycle without needing deep knowledge of the underlying orchestration logic. * Users can trigger complex pipelines simply by issuing a message, abstracting the technical difficulties of cloud infrastructure. * The dashboard provides a visual interface for monitoring events, message flows, and evaluation results, improving overall transparency for data scientists and researchers. For organizations managing large-scale multimodal models, moving toward an event-driven architecture is highly recommended. This model reduces the overhead of maintaining rigid pipelines and allows engineering teams to focus on model quality rather than infrastructure orchestration.

Multimodal AIOrchestrationMlopsLlm Evaluation+4