How will long-term user modeling (opens in new tab)
Long-term user modeling captures persistent interests, cross-vertical behavior, and signals beyond what short-term recommendation logs can reveal. 당근 built a Transformer-based user encoder that learns from tens of billions of actions across its local marketplace, jobs, real estate, and other services, then exposes the resulting embedding as a shared feature for ranking, retrieval, and advertising models. The approach improved scalability and reuse, but introduced freshness and representation-transfer limitations. ## Why Long-Term User Modeling Matters - Recent actions reveal immediate intent, but miss recurring interests such as seasonal shopping or repeated moving-related searches. - Long-term, cross-vertical activity can connect behaviors such as: - Searching for real estate - Looking for furniture and appliances - Reading neighborhood moving advice - Longer histories can reduce selection bias caused by training only on items previously exposed by recommendation models. - Simply adding more history is insufficient because ranking systems are latency-sensitive and long sequences increase computation and infrastructure complexity. ## Shared User Embeddings as a Common Feature - A separate user encoder processes long-term history offline. - Home-feed ranking, candidate generation, and advertising models consume the resulting embedding as a shared user feature. - Benefits: - Downstream models avoid directly processing massive histories. - The encoder can scale independently in model size, data, and compute. - One embedding can be reused across multiple recommendation surfaces. - Limitations: - A downstream model receives only a fixed vector, so it cannot fully exploit the encoder’s richer representations. - Batch inference means recent actions are not reflected immediately. - Possible future improvements include more frequent or real-time updates, fine-tuning, and distillation. ## Contrastive User Modeling - The encoder uses a two-tower architecture: - A causal Transformer converts the user’s action sequence into a user embedding. - An MLP converts item features into item embeddings. - InfoNCE loss trains the user embedding to predict the next interacted item. - In-batch negatives provide alternative items for contrastive learning. - Training uses clicks and conversion actions across all major verticals and surfaces. - The dataset contains tens of billions of actions—around 150 times more than the existing home-feed candidate model’s training data. ## Item ID Embeddings vs. Content Embeddings ### Problems with Item ID Embeddings - New items have no learned ID embedding, creating a cold-item problem. - Hundreds of millions of item IDs require enormous embedding tables. - In the ID-based model, embedding tables accounted for over 99% of parameters, leaving little GPU capacity for the Transformer. - Hashing and embedding-sharding techniques were considered but did not provide a sufficient solution. ### Content Embeddings - The system switched to LLM-generated embeddings based on post metadata. - This enables: - Representations for newly created items - Much larger Transformer models, with Transformer parameters becoming roughly 1,000 times larger than in the ID-based setup - Large-scale lookup required two memory-efficient techniques: - `memmap` loads only needed embedding segments from disk and benefits from shared OS page caches during distributed training. - `bbhash` maps item IDs to embedding locations using roughly three bits per key, reducing mapping memory by about 97% compared with Python dictionaries. - Together, these methods made training with hundreds of gigabytes of item embeddings practical. ## Region-Constrained Batch Sampling - Standard in-batch negatives assume that other items in the batch were visible but not selected. - This assumption fails in a local service: users generally cannot view items outside their geographic area. - More than 86% of transactions occur within five kilometers, yet random batches mixed users and items nationwide. - Consequently, about 98% of random in-batch negatives were “impossible negatives”—items users could never have seen. - These negatives teach geographic unavailability rather than user preference, weakening the contrastive signal. ### RCBS Solution - Region-Constrained Batch Sampling (RCBS) groups users from the same region into a batch. - This reduced impossible negatives from 98% to 30%. - The remaining impossible negatives mainly came from differences in viewing radius or users’ historical activity in other regions. - Feasible negatives are harder because they represent items users could have viewed but rejected, forcing the model to distinguish genuine preferences among similar local items. ### Why Sampling Was Better Than Masking - Masking impossible negatives would remove most of the batch, drastically reducing effective batch size. - Hard-negative mining would require checking feasibility separately for each user and could be expensive and complex. - RCBS naturally produces more feasible and difficult negatives without changing the loss function or adding specialized mining. ## Applying the Embeddings - For home-feed and advertising ranking, the embedding is projected and concatenated with existing features. - The long-term encoder supplies persistent preference signals, while existing ranking models continue handling short-term and real-time signals. - For retrieval models, the best-performing approach used the user embedding alone to generate candidates rather than merely adding it as another feature. - The separate candidate source appeared to improve recommendation diversity. ## Embedding Refresh and Serving - Offline tests showed little difference between frozen embeddings and 12- or 24-hour refreshes. - Online A/B tests favored periodic updates, with shorter intervals performing better. - A 24-hour refresh cycle was selected as the best cost-performance trade-off. - GPU inference runs through a Beam pipeline on GCP Dataflow. - Only users who acted during the refresh window are reprocessed, avoiding unnecessary inference for inactive users. - Near-real-time inference remains a major future engineering challenge. The overall recommendation is to treat long-term user modeling as a separate, reusable representation system rather than forcing every downstream model to process extensive histories directly. For geographically constrained services, the training data pipeline—especially negative sampling—must reflect actual item visibility, making region-aware batching as important as the model architecture itself.