lineMar 6, 2026

On-Device Image Model (opens in new tab)

The post describes building an on-device image understanding system for messaging apps, with semantic image search as the first focus. Its central strategy was knowledge distillation: a multilingual student text encoder learned to reproduce the embedding space of a strong, English-only teacher model. This preserved most English retrieval quality while enabling Japanese, Traditional Chinese, Thai, and Korean search, achieving an average Recall@5 above 78%. ## Why Messaging Apps Need On-Device Image Understanding - Images are often treated simply as “a photo,” unlike text messages, which can support search, summaries, and notification previews. - Image understanding could improve: - Notifications: “Sent one photo” → “Sent a photo of a dog” - Search: queries such as “dog,” “puppy,” or “a cat inside a box” - Recommendations: automatic image classification and organization - Shared image-text embeddings allow semantically equivalent phrases—such as “dog,” “puppy,” and “개”—to retrieve the same images. ## Why the Model Had to Run On-Device - **Latency:** Network round trips make notifications and search less predictable and responsive. - **Privacy:** Sending photos, captions, or embeddings to a server increases privacy risks. - **Offline support:** The feature should work in subways, airplanes, roaming environments, and unstable networks. - **Mobile constraints:** The model needed to run on both Android and iOS with limited memory and compute resources. - The project targeted a model under **200 MB**, response times within a few hundred milliseconds including cold start, and LiteRT compatibility. ## Project Goals and Evaluation - The image search system needed to: - Retrieve images by semantic meaning rather than keyword matching. - Support English, Japanese, Traditional Chinese, Thai, and Korean. - A separate captioning system was designed to generate short, natural descriptions of roughly eight words or fewer. - Search quality was measured using: - Image-to-Text Recall@5 - Text-to-Image Recall@5 - Caption quality was evaluated with CIDEr, CLIPScore, and an LLM-based acceptance ratio designed to detect repetition, typos, and grammatical problems. ## Why Translation Was Not Enough The initial approach translated each query into English before using an English-only image-text model: ```text Query → Language detection → Translation → English text encoder → Embedding → Search ``` This approach introduced several problems: - **Quality loss:** Informal terms or short queries could be mistranslated. For example, “멍멍이” might be interpreted as “barking” instead of “dog.” - **Additional latency:** Translation adds a fixed cost before text encoding. - **Inconsistent results:** Translation quality varies by language pair and wording. - **Operational complexity:** Each additional language requires more models, updates, and failure handling. Training a multilingual image-text model from scratch would require substantial data and compute. Instead, the project retained the proven English image embedding space and expanded only the text encoder. ## Knowledge Distillation for Multilingual Search - The original English text encoder served as the frozen **teacher**. - A copied text encoder served as the trainable **student**. - English text was passed to the teacher, while corresponding multilingual text was passed to the student. - The student was trained to match the teacher’s embeddings using mean squared error (MSE). ```text teacher_embedding = teacher(English text) student_embedding = student(Multilingual text) loss = MSE(teacher_embedding, student_embedding) ``` The image encoder remained frozen so that the established image-text embedding space would not be disrupted. Important implementation considerations included: - Ensuring the tokenizer handled multilingual characters correctly. - Defining consistent case-insensitivity rules. - Balancing training samples across languages. - Matching training-time preprocessing and tokenization with mobile inference behavior. ## Retrieval Results - English performance declined slightly: - Image-to-Text Recall@5: **79.58% → 76.56%** - Text-to-Image Recall@5: **75.89% → 74.47%** - Multilingual performance improved from below **10% average Recall@5** to above **78%**, roughly a sevenfold improvement. - Japanese achieved **81.94%**, exceeding the original English model in the reported evaluation. - Traditional Chinese, Thai, and Korean also reached practically usable retrieval quality. The trade-off—slightly lower English performance in exchange for four additional languages—provided substantially greater overall product value. ## Converting the Model to LiteRT - LiteRT was selected because it officially supports both Android and iOS and provides mobile-oriented operators, quantization, and optimization tools. - Core ML was rejected because it is iOS-specific and introduced conversion and long-term cross-platform maintenance concerns. - Conversion required addressing unsupported PyTorch operators. - For example, LiteRT did not support `erf`, so the model’s implementation had to replace it with a compatible pseudo-`erf` operation. The resulting approach demonstrates that knowledge distillation can efficiently extend an existing English image-text model to multiple languages while preserving its on-device deployment advantages.

embedding-modelpytorchknowledge-distillationlitert+4

daangnFeb 27, 2026

How will long-term user modeling (opens in new tab)

Long-term user modeling captures persistent interests, cross-vertical behavior, and signals beyond what short-term recommendation logs can reveal. 당근 built a Transformer-based user encoder that learns from tens of billions of actions across its local marketplace, jobs, real estate, and other services, then exposes the resulting embedding as a shared feature for ranking, retrieval, and advertising models. The approach improved scalability and reuse, but introduced freshness and representation-transfer limitations. ## Why Long-Term User Modeling Matters - Recent actions reveal immediate intent, but miss recurring interests such as seasonal shopping or repeated moving-related searches. - Long-term, cross-vertical activity can connect behaviors such as: - Searching for real estate - Looking for furniture and appliances - Reading neighborhood moving advice - Longer histories can reduce selection bias caused by training only on items previously exposed by recommendation models. - Simply adding more history is insufficient because ranking systems are latency-sensitive and long sequences increase computation and infrastructure complexity. ## Shared User Embeddings as a Common Feature - A separate user encoder processes long-term history offline. - Home-feed ranking, candidate generation, and advertising models consume the resulting embedding as a shared user feature. - Benefits: - Downstream models avoid directly processing massive histories. - The encoder can scale independently in model size, data, and compute. - One embedding can be reused across multiple recommendation surfaces. - Limitations: - A downstream model receives only a fixed vector, so it cannot fully exploit the encoder’s richer representations. - Batch inference means recent actions are not reflected immediately. - Possible future improvements include more frequent or real-time updates, fine-tuning, and distillation. ## Contrastive User Modeling - The encoder uses a two-tower architecture: - A causal Transformer converts the user’s action sequence into a user embedding. - An MLP converts item features into item embeddings. - InfoNCE loss trains the user embedding to predict the next interacted item. - In-batch negatives provide alternative items for contrastive learning. - Training uses clicks and conversion actions across all major verticals and surfaces. - The dataset contains tens of billions of actions—around 150 times more than the existing home-feed candidate model’s training data. ## Item ID Embeddings vs. Content Embeddings ### Problems with Item ID Embeddings - New items have no learned ID embedding, creating a cold-item problem. - Hundreds of millions of item IDs require enormous embedding tables. - In the ID-based model, embedding tables accounted for over 99% of parameters, leaving little GPU capacity for the Transformer. - Hashing and embedding-sharding techniques were considered but did not provide a sufficient solution. ### Content Embeddings - The system switched to LLM-generated embeddings based on post metadata. - This enables: - Representations for newly created items - Much larger Transformer models, with Transformer parameters becoming roughly 1,000 times larger than in the ID-based setup - Large-scale lookup required two memory-efficient techniques: - `memmap` loads only needed embedding segments from disk and benefits from shared OS page caches during distributed training. - `bbhash` maps item IDs to embedding locations using roughly three bits per key, reducing mapping memory by about 97% compared with Python dictionaries. - Together, these methods made training with hundreds of gigabytes of item embeddings practical. ## Region-Constrained Batch Sampling - Standard in-batch negatives assume that other items in the batch were visible but not selected. - This assumption fails in a local service: users generally cannot view items outside their geographic area. - More than 86% of transactions occur within five kilometers, yet random batches mixed users and items nationwide. - Consequently, about 98% of random in-batch negatives were “impossible negatives”—items users could never have seen. - These negatives teach geographic unavailability rather than user preference, weakening the contrastive signal. ### RCBS Solution - Region-Constrained Batch Sampling (RCBS) groups users from the same region into a batch. - This reduced impossible negatives from 98% to 30%. - The remaining impossible negatives mainly came from differences in viewing radius or users’ historical activity in other regions. - Feasible negatives are harder because they represent items users could have viewed but rejected, forcing the model to distinguish genuine preferences among similar local items. ### Why Sampling Was Better Than Masking - Masking impossible negatives would remove most of the batch, drastically reducing effective batch size. - Hard-negative mining would require checking feasibility separately for each user and could be expensive and complex. - RCBS naturally produces more feasible and difficult negatives without changing the loss function or adding specialized mining. ## Applying the Embeddings - For home-feed and advertising ranking, the embedding is projected and concatenated with existing features. - The long-term encoder supplies persistent preference signals, while existing ranking models continue handling short-term and real-time signals. - For retrieval models, the best-performing approach used the user embedding alone to generate candidates rather than merely adding it as another feature. - The separate candidate source appeared to improve recommendation diversity. ## Embedding Refresh and Serving - Offline tests showed little difference between frozen embeddings and 12- or 24-hour refreshes. - Online A/B tests favored periodic updates, with shorter intervals performing better. - A 24-hour refresh cycle was selected as the best cost-performance trade-off. - GPU inference runs through a Beam pipeline on GCP Dataflow. - Only users who acted during the refresh window are reprocessed, avoiding unnecessary inference for inactive users. - Near-real-time inference remains a major future engineering challenge. The overall recommendation is to treat long-term user modeling as a separate, reusable representation system rather than forcing every downstream model to process extensive histories directly. For geographically constrained services, the training data pipeline—especially negative sampling—must reflect actual item visibility, making region-aware batching as important as the model architecture itself.

embedding-modeltransformercontrastive-learningrecommendation-system+4