Solving the Cold-Start Problem in Search Reranking Through Embedding Stabilization: A LINE Part Time Jobs Case Study (opens in new tab)
LY Corporation improved LINE Part Time Jobs’ real-time search reranking by stabilizing user and item embeddings produced by a two-tower recommendation model. The approach addressed both cold-start degradation and daily embedding-space drift without changing the underlying model or training pipeline. Offline and online evaluations showed substantial gains, including a 4.7% overall KPI increase and 6.5% revenue growth. ## Search Reranking at LINE Part Time Jobs - Search consists of: - Retrieval, which finds listings matching a query. - Reranking, which orders the retrieved candidates. - The previous system ranked listings by cosine similarity between precomputed user-to-item two-tower embeddings. - This approach was computationally simple and captured broad user preferences, but: - It ignored query-specific information, such as the distance from a selected station. - Its embeddings combined behavior from multiple services and recommendation modules, not just search activity. - The team therefore introduced a dedicated real-time reranking model. ## Challenges with the Dedicated Reranking Model ### Cold Start - Most job listings are replaced at the beginning of each month. - New listings initially lack sufficient interaction data. - As a result, reranking quality dropped until enough training data accumulated. ### Embedding-Space Drift - Two-tower models were regularly retrained from random initialization. - Each training run produced a different embedding space. - Using embeddings as downstream features caused a mismatch between training-time and inference-time data, reducing model performance. ## Stabilizing the Embedding Space - Each day’s embeddings are aligned with the previous day’s stabilized embeddings. - The first day’s embeddings are used without stabilization. - This preserves continuity across retraining cycles and allows embeddings generated on different days to remain comparable. - Downstream models and embedding generation no longer need perfectly synchronized update schedules. ### Low-Rank SVD - User and item embeddings are converted into a more standardized low-dimensional representation. - Instead of decomposing the enormous user-item score matrix directly, transformation matrices are derived from the embedding matrices. - This makes the procedure practical for large-scale data. ### Orthogonal Procrustes Alignment - The transformed embeddings are aligned to the previous day’s stabilized space. - The orthogonal transformation only rotates or reflects the space. - Distances and inner-product relationships are therefore largely preserved, maintaining the two-tower model’s scoring behavior. ## Scalable Implementation - The algorithm was implemented with Apache Spark to handle LINE Part Time Jobs’ large datasets. - For low-rank SVD: - The original QR decomposition was optimized using Cholesky decomposition. - The Gram matrix \(G=A^\top A\) is decomposed to obtain the same upper-triangular matrix \(R\) as QR decomposition. - For Procrustes alignment: - The large matrix multiplication \(M=B^\top A\) is distributed across Spark. - The resulting \(e \times e\) matrix is small enough for SVD on a single node using NumPy. ## Evaluation Results ### Embedding Stability - Before stabilization, embeddings from randomly selected days had correlations close to zero. - After stabilization: - Similarity remained around 0.88 after one week. - Similarity remained around 0.87 after one month. - This reduced performance loss caused by embedding drift. ### Offline Evaluation - Unstabilized embeddings reduced nDCG by approximately 1–5% when training and inference used different days. - Stabilized embeddings improved: - Conversion nDCG by about 9.0%. - Click nDCG by about 4.5%. ### Online A/B Test - Search-page KPIs alone did not show statistically significant improvement. - Across the entire service: - KPIs increased by 4.7%. - Revenue increased by 6.5%. - The results suggest that the embeddings captured long-term user preferences that influenced later actions across the service, not only behavior on the search page. - The added embedding features also helped mitigate the initial cold-start problem. ## Practical Benefits and Future Work - The solution required no changes to the two-tower model itself. - Stabilization was added as post-processing, minimizing changes to existing pipelines and reducing deployment risk. - LY Corporation plans to test the method as the service expands its sources of job listings and to reuse the approach across other services through its internal machine-learning platform. Overall, sequential low-rank SVD and orthogonal Procrustes alignment provide a relatively simple way to make frequently retrained embeddings reliable downstream features while improving real-time reranking and business outcomes.
