On Device Ai | Techlist.io

googleMay 27, 20263 min readCurated summary

Private analytics via zero-trust aggregation

Google presents a private analytics architecture combining one-shot cryptographic secure aggregation with trusted execution environments (TEEs). The design follows a zero-trust model: cryptography prevents individual data from being reconstructed, while TEE attestation verifies that approved, publicly auditable code is running. It aims to provide useful population-level insights without exposing raw user data, even if hardware protections are compromised. ## Why Private Analytics Matters - On-device AI protects user information, but developers still need to measure performance across millions of devices. - Aggregated analytics can reveal: - Model drift, such as difficulty handling new language or slang. - Hidden biases across regions, environments, or demographic conditions. - Real-world error rates and user reactions to model outputs. - Google already uses federated analytics in products including Pixel Recorder and Gboard. ## Limitations of Existing Protections - **TEEs** isolate sensitive computation from compromised operating systems and hypervisors. - Hardware attestation produces a cryptographic fingerprint proving which firmware and software are running. - However, TEEs remain vulnerable to newly discovered side-channel attacks. - Traditional cryptographic aggregation offers mathematical privacy guarantees, but many protocols require devices to stay online through several interactive rounds. - Extended multiround participation limits practical deployment at large scale. ## Combining Encryption and Isolation - The new system allows devices to submit data in a single message. - This removes the need for devices to remain connected for multiple protocol rounds. - Data is encrypted before leaving the device and is never reconstructed in server memory, including inside a TEE. - Unencrypted information is processed off-device only after it has been aggregated and anonymized. - TEE attestation provides verifiable evidence that the intended secure aggregation implementation is running correctly. - The cryptographic and hardware layers provide defense in depth: failure of one layer does not automatically expose individual data. ## One-Shot Lattice-Based Aggregation - The protocol uses lattice-based cryptography. - Ciphertexts can be combined so that their underlying data—and encryption keys—are aggregated together. - A resulting decryption key can reveal only the aggregate, not individual contributions. - Small client committees hold decryption hints and help unlock the aggregated result. - Differential privacy noise is added to further protect the aggregate. - Decryption authority is distributed across multiple parties, preventing any single party from accessing encrypted data. ## Application to Android SafetyCore - SafetyCore provides privacy-preserving, on-device safety features for Android 9 and later. - Private analytics can help measure which threats are detected and identify areas where detection needs improvement. - Google is working with the Android SafetyCore team to apply this system while preserving the confidentiality of individual users’ data. ## Practical Takeaway The approach pairs the scalability of one-shot cryptographic aggregation with the verifiability of TEEs. It is designed for private, large-scale analytics where neither individual data nor trust in a single hardware or software component is required.

Read original

On Device Ai Zero Trust Differential Privacy Federated Analytics+3

kakaoApr 21, 20263 min readCurated summary

Kanana Scala 1st Seminar On-site Sketch

Kakao’s first Kanana Scholar seminar brought together seven leading AI professors and Kakao researchers to discuss the company’s independent AI strategy. Kakao presented its from-scratch Kanana foundation models, emphasizing data efficiency, Korean-language capability, and multimodal processing. The discussion concluded that Kakao should focus less on generic benchmark scores and more on technology sovereignty, personalized agents, and practical execution in real services. ## Kanana Foundation Models - Kakao is developing its own foundation-model lineup to strengthen competitiveness and reduce dependence on overseas providers. - Kanana reportedly achieved strong performance using 11 trillion training tokens, compared with 23 trillion tokens for a similarly sized global-target model. - Kakao attributed this efficiency to the quality and refinement of its training data. - The company also demonstrated **Kanana-o**, an omni model capable of processing text, images, and audio in real time. - The model handled emotional speech and multi-speaker conversations naturally, receiving praise for its Korean fluency. ## Technology Sovereignty and Customization - Kakao argued that proprietary models protect it from external risks such as changing licensing policies and closed technologies. - Owning the technology enables Kakao to build efficient, customized models optimized for its services. - Participating professors agreed that control over Korean cultural context and local issues is essential for technological sovereignty. - They viewed an independent model as a strategic asset for long-term service stability. ## Digital World Models and Personalized Agents - Kakao aims to understand users’ behavioral context within KakaoTalk and provide highly personalized assistance. - On-device AI could protect private conversations while allowing agents to respond immediately to user needs. - The professors suggested expanding the idea of “physical AI” into a **digital world model** that predicts interactions and causal relationships across a platform. - This direction could create an area of AI differentiation uniquely suited to Kakao’s ecosystem. ## Evaluating Practical Agentic Intelligence - Kakao is prioritizing AI systems that can create multi-step plans, call necessary tools, and complete tasks independently. - It plans to use an internally developed orchestration benchmark to evaluate real-world problem-solving ability. - The professors cited Claude as an example of how users perceive intelligence through successful completion of complex requests, not merely high benchmark scores. - They recommended competing through practical execution in real service environments rather than focusing only on text-generation performance. ## Industry-Academic Cooperation - Kakao plans to explore GPU support for university research labs and undergraduate AI clubs. - Possible support could include credits, project-based resources, and other forms of infrastructure assistance. - The seminar marked the beginning of a broader collaboration aimed at advancing Korea’s AI ecosystem and developing future talent. Kakao’s recommended path is to combine proprietary, efficient models with privacy-preserving personalization and strong agentic execution. Success will depend on how effectively Kanana turns technical depth into useful intelligence that users can experience in everyday services.

Read original

On Device Ai AI AI Agents Multimodal AI+4

googleJan 22, 20263 min readCurated summary

Small models, big results: Achieving superior intent extraction through decomposition

Small multimodal models can outperform much larger models at extracting user intent from UI interaction trajectories when the task is decomposed. Google’s approach first summarizes each screen and interaction, then derives an overall intent from those summaries. This enables accurate, faster, and more privacy-preserving on-device understanding without sending sensitive UI data to servers. ## Why On-Device Intent Understanding Matters - Understanding what users are doing across mobile and web interfaces can help agents anticipate useful next actions. - Large multimodal models perform well but often require server-side processing, introducing latency, cost, and privacy risks. - The goal is to make intent understanding practical with smaller models running directly on devices. ## Two-Stage Intent Extraction ### Screen and Interaction Summaries For each interaction, a small multimodal model examines a sliding window of three screens: the previous, current, and next screens. It generates information about: - Salient context on the current screen. - Actions the user just performed. - A speculation about what the user is trying to accomplish. This converts raw screenshots and actions into structured, manageable event summaries. ### Intent Extraction from Summaries A fine-tuned small model then processes the sequence of summaries and produces a single concise intent statement. The authors improve this stage through: - **Fine-tuning:** Training on examples of high-quality intent statements helps the model retain relevant details and discard noise. - **Label preparation:** Training intents are stripped of details absent from the summaries, reducing hallucinated information. - **Removing speculation:** Speculative fields help create richer individual summaries but are excluded from the second stage because they can confuse intent extraction. ## Evaluation with Atomic Facts - The authors use the Bi-Fact evaluation method to compare predicted intents with reference intents. - Each intent is split into indivisible “atomic facts,” such as “a one-way flight” or the separate origin and destination in a flight request. - The method measures: - **Recall:** How many reference facts were captured. - **Precision:** How many predicted facts are supported by the reference. - **F1:** The balance between precision and recall. - Tracking facts through both stages also reveals where details are lost or hallucinated. ## Results - The decomposed method outperformed chain-of-thought prompting and end-to-end fine-tuning. - Improvements held across both mobile and web interaction trajectories. - Results were consistent across Gemini and Qwen2 base models. - Gemini 1.5 Flash 8B achieved results comparable to Gemini 1.5 Pro while offering substantially lower cost and faster processing. - On mobile data, the small-model approach approached the performance of the larger Gemini Pro model. The study suggests that decomposing intent understanding into local summarization followed by sequence-level extraction is an effective path toward accurate, private, and efficient on-device assistants. As mobile hardware and small models improve, this technique could support a broad range of assistive features.

Read original

On Device Ai Gemini Natural Language Processing Multimodal AI+4

googleSep 30, 2025Original article

Introducing interactive on-device segmentation in Snapseed (opens in new tab)

Google has introduced a new "Object Brush" feature in Snapseed that enables intuitive, real-time selective photo editing through a novel on-device segmentation technology. By leveraging a high-performance interactive AI model, users can isolate complex subjects with simple touch gestures in under 20 milliseconds, bridging the gap between professional-grade editing and mobile convenience. This breakthrough is achieved through a sophisticated teacher-student training architecture that prioritizes both pixel-perfect accuracy and low-latency performance on consumer hardware. ### High-Performance On-Device Inference * The system is powered by the Interactive Segmenter model, which is integrated directly into the Snapseed "Adjust" tool to facilitate immediate object-based modifications. * To ensure a fluid user experience, the model utilizes the MediaPipe framework and LiteRT’s GPU acceleration to process selections in less than 20ms. * The interface supports dynamic refinement, allowing users to provide real-time feedback by tracing lines or tapping to add or subtract specific areas of an image. ### Teacher-Student Model Distillation * The development team first created "Interactive Segmenter: Teacher," a large-scale model fine-tuned on 30,000 high-quality, pixel-perfect manual annotations across more than 350 object categories. * Because the Teacher model’s size and computational requirements are prohibitive for mobile use, researchers developed "Interactive Segmenter: Edge" through knowledge distillation. * This distillation process utilized a dataset of over 2 million weakly annotated images, allowing the smaller Edge model to inherit the generalization capabilities of the Teacher model while maintaining a footprint suitable for mobile devices. ### Training via Synthetic User Prompts * To make the model universally capable across all object types, the training process uses a class-agnostic approach based on the Big Transfer (BiT) strategy. * The model learns to interpret user intent through "prompt generation," which simulates real-world interactions such as random scribbles, taps, and lasso (box) selections. * During training, both the Teacher and Edge models receive identical prompts—such as red foreground scribbles and blue background scribbles—to ensure the student model learns to produce high-quality masks even from imprecise user input. This advancement significantly lowers the barrier to entry for complex photo manipulation by moving heavy-duty AI processing directly onto the mobile device. Users can expect a more responsive and precise editing experience that handles everything from fine-tuning a subject's lighting to isolating specific environmental elements like clouds or clothing.

On Device Ai AI Machine Learning Computer Vision+5

googleJul 23, 2025Original article

Synthetic and federated: Privacy-preserving domain adaptation with LLMs for mobile applications (opens in new tab)

Researchers at Google have developed a framework for improving both small and large language models (LMs) in mobile applications like Gboard by utilizing privacy-preserving synthetic data and federated learning. This approach combines differential privacy (DP) with large language model (LLM) generation to minimize data memorization risks while achieving significant gains in production metrics like next-word prediction and proofreading. The result is a robust pipeline that allows models to adapt to specific user domains without compromising individual privacy or requiring centralized data storage. ### Strengthening Privacy with DP-FL * Gboard has transitioned all production LMs trained on user data to a Federated Learning with Differential Privacy (DP-FL) framework, ensuring data remains on-device and is never memorized. * The deployment utilizes the **BLT-DP-FTRL** algorithm, which offers an optimized trade-off between privacy guarantees and model utility while being easier to deploy in production. * Engineers adopted the **SI-CIFG** model architecture to facilitate efficient on-device training, ensuring the hardware can handle local updates while maintaining compatibility with DP constraints. ### Synthetic Data Generation via Public LLMs * Powerful LLMs trained on public web data are prompted to synthesize high-quality text that mimics mobile user interactions without ever accessing actual private user data. * The process involves a two-step prompting strategy: first, filtering public datasets to identify topics common in mobile communication, and second, generating new, domain-specific text based on those patterns. * This synthetic data serves as a bridge for pre-training small LMs, which are then refined through private post-training on-device to capture the nuances of user behavior. ### Adapting LLMs for Mobile Proofreading * To support advanced features like Gboard's "Proofread," researchers developed a "Synthesize-then-Adapt" pipeline specifically for error correction. * LLMs generate synthetic "corrupted" text to simulate common mobile typing errors, providing the necessary training pairs (error/correction) that are difficult to find in public datasets. * Federated learning is then used to adapt these error-correction models to specific app domains (such as messaging or email) using on-device signals, ensuring the model understands the specific context of the user's typing. The success of these techniques in Gboard demonstrates that synthetic data can effectively replace or augment private data throughout the machine learning lifecycle. For developers working with sensitive user information, adopting a "synthetic-first" approach combined with federated learning provides a scalable path to model improvement that adheres to the core principles of data minimization and anonymization.

On Device Ai Large Language Models Differential Privacy Synthetic Data+3

googleMay 21, 2025Original article

Google Research at Google I/O 2025 (opens in new tab)

Google Research at I/O 2025 showcases the "research to reality" transition, highlighting how years of foundational breakthroughs are now being integrated into Gemini models and specialized products. By focusing on multimodal capabilities, pedagogy, and extreme model efficiency, Google aims to democratize access to advanced AI while ensuring it remains grounded and useful across global contexts. ## Specialized Healthcare Models: MedGemma and AMIE * **MedGemma:** This new open model, based on Gemma 3, is optimized for multimodal medical tasks such as radiology image analysis and clinical data summarization. It is available in 4B and 27B sizes, performing similarly to much larger models on the MedQA benchmark while remaining small enough for efficient local fine-tuning. * **AMIE (Articulate Medical Intelligence Explorer):** A research AI agent designed for diagnostic medical reasoning. Its latest multimodal version can now interpret and reason about visual medical information, such as skin lesions or medical imaging, to assist clinicians in diagnostic accuracy. ## Educational Optimization through LearnLM * **Gemini 2.5 Pro Integration:** The LearnLM family of models, developed with educational experts, is now integrated into Gemini 2.5 Pro. This fine-tuning enhances STEM reasoning, multimodal understanding, and pedagogical feedback. * **Interactive Learning Tools:** A new research-optimized quiz experience allows students to generate custom assessments from their own notes, providing specific feedback on right and wrong answers rather than just providing solutions. * **Global Assessment Pilots:** Through partnerships like the one with Kayma, Google is testing the automatic assessment of short and long-form content in regions like Ghana to scale quality educational tools. ## Multilingual Expansion and On-Device Gemma Models * **Gemma 3 and 3n:** Research breakthroughs have expanded Gemma 3’s support to over 140 languages. The introduction of **Gemma 3n** targets extreme efficiency, capable of running on devices with as little as 2GB of RAM while maintaining low latency and low energy consumption. * **ECLeKTic Benchmark:** To assist the developer community, Google introduced this novel benchmark specifically for evaluating how well large language models transfer knowledge across different languages. ## Model Efficiency and Factuality in Search * **Inference Techniques:** Google Research continues to set industry standards for model speed and accessibility through technical innovations like **speculative decoding** and **cascades**, which reduce the computational cost of generating high-quality responses. * **Grounded Outputs:** Significant focus remains on factual consistency, ensuring that the AI models powering features like AI Overviews in Search provide reliable and grounded information to users. As Google continues to shrink the gap between laboratory breakthroughs and consumer products, the emphasis remains on making high-performance AI accessible on low-cost hardware and across diverse linguistic landscapes. Developers and researchers can now leverage these specialized tools via platforms like HuggingFace and Vertex AI to build more targeted, efficient applications.

On Device Ai AI Generative AI Gemini+5