Knowledge-Geometry Decoupling: Refreshable Pretrained Transfer for Streaming Recommendation
Zixuan Wang, Yuhong Chen, Yuxuan Zhu, Guidong Lei, Zhiluohan Guo, Yu Zhao, Kun Wang, Bangyang Hong, Kangle Wu, Yabo Ni, Anxiang Zeng, Cong Fu, Hui Li
cs.IR
2026-08-04
Industrial recommenders pretrain then transfer, but behavior drifts. KGD learns cleaner knowledge with behavioral multi-token prediction and decouples read-write ownership so the encoder refreshes daily without harming task adaptation: +4-12% on eight benchmarks, +1.75% GMV per user live at Shopee.
Industrial recommenders use pretrain-then-transfer: pretrain on massive behavior sequences, then transfer to the target task. The trouble is behavioral drift on live streams, which raises two questions: what should you learn from behavior sequences, and how do you transfer that knowledge while the pretrained model is continually refreshed?
Conventional next-token prediction treats adjacency as dependency. A user sequence is several unrelated interest sessions concatenated, so it models spurious cross-session jumps (bicycle gear followed by pants) as real dependencies, producing dirty knowledge. On the transfer side, pretrained knowledge and task-specific geometry push shared parameters in conflicting directions, with near-zero gradient cosine similarity. Existing fixes either fine-tune and overwrite pretrained knowledge, or freeze and forbid adaptation.
KGD (Knowledge-Geometry Decoupling) splits "what to learn" and "how to transfer" into independent mechanisms.
For what to learn, BMTP (Behavioral Multi-Token Prediction) replaces next-token prediction: instead of predicting every adjacent next item, it keeps only future items that are collaboratively or semantically related to the current one. Collaborative relevance comes from an item co-occurrence graph; semantic relevance from text-embedding cosine similarity; only items above a threshold survive.
For how to transfer, KGD uses decoupled read-write ownership. A refreshable encoder owns behavioral knowledge and is refreshed daily. A task learner reads encoder states through read-only cross-attention (stop-gradient on keys and values, so task gradients cannot reach the encoder) and writes its own geometry through ACR (Anchored Calibration Residual). ACR constrains the task residual to the orthogonal complement of the pretrained embedding, so the task adds discriminative directions without overwriting pretrained ones. Because the task geometry is an orthogonal residual rather than absolute coordinates, the encoder can refresh daily without invalidating downstream adaptation.
On eight public benchmarks (Amazon-2023), KGD beats strong pretrain-transfer baselines by 4-12%. Arts adds +6.5% N@50, and BMTP alone adds +33.7% N@50 on Games versus next-token prediction. Over a 90-day production stream, KGD holds its edge while frozen transfer and buffer replay degrade.
Live A/B on Shopee Homepage Search (10% traffic per bucket, tens of millions of users): GMV per user +1.75%, ad revenue +1.53%, CTR +0.95%, CVR +0.72%, all significant at p<0.01. A one-week reversal cut GMV 1.21% and revenue 1.50%, confirming causality. It is deployed in production with 120ms serving latency on A30.
For recsys practitioners, this is a workable path to continually refreshing a pretrained model without breaking downstream adaptation. The core idea, physically separating read-write ownership and anchoring task geometry as an orthogonal residual, is not specific to recommendation; any setting that refreshes a pretrained model online without harming downstream could borrow it. The reversal test is a plus: the authors did the causal homework, not just offline metrics.
The authors scope it to classic deep-learning recommenders, not LLM-based ones whose knowledge is distilled from text. Streaming evaluation differs from incremental learning that seeks invariant representations. Aggressive sampling on public benchmarks distorts the real distribution, making it hard to separate the collaborative and semantic filters' contributions. The live A/B covers only Homepage Search, and the 90-day stream reports relative trends, not absolute AUC, so do not expect the same percentages on another business.