Self-Evolving Memory for Generative Recommendation
Xinyu Lin, Zhuosong Jiang, Zixiao Suo, Siqin Wang, Hanqing Zeng, Hanchao Yu, Yinglong Xia, Jiang Zhang, Aashu Singh, Fei Liu, Wenjie Wang, Fuli Feng, Yang Song, Qifan Wang, Tat-Seng Chua
cs.IR, cs.AI
2026-09-14
NUS and Meta show shared-parameter updates let dominant behaviors crowd out rare ones; LION's sparse KV memory lifts Games Recall@10 29.8% over TIGER.
Generative recommenders emit the next item's identifier as a token sequence. User-history encoding and item generation share one Transformer. Preferences keep moving, so the model has to track streaming interactions. Continual fine-tuning, distillation, and replay all still write those updates into the same shared weights.
The structural failure is evolution conflict. Gradients from common behaviors of active users are large and frequent; gradients from rare behaviors of inactive users are small; the two can point opposite ways. Updates follow the majority, and the tail is slowly ignored. On Toys, TIGER split by activity makes the point: from Period 0 to Period 1, active users with common patterns improve, inactive users drop, and the aggregate falls with them.
LION states three design rules: isolated memorization so different behaviors take different parameter paths; reinforced evolution so rare patterns get stronger supervision; scalable application so parameters do not grow with the user count.
The implementation is a sparse Key-Value memory inserted in a TIGER (T5) backbone. N trainable KV pairs. The user vector is taken from the full interaction history, dotted with every key, and the Top-K values are added back into the hidden state with min-max weights. Layers around the memory still see only a recent window, which keeps compute in check; routing sees the full history so a short recency window cannot flatten the user.
Training uses two losses. The recommendation loss is the usual next-identifier likelihood. A consolidation loss predicts the next item from the memory-augmented representation itself, so rare preferences have to be written into the memory, not only into the final head. The sum is Lrec + λ Lcon. When a new period arrives, backbone and memory are fine-tuned together and then served for the next period.
On the theory side, conflict is confined to the K overlapping activated slots; a smaller overlap ratio ρ means less conflict than a fully shared baseline. Cross-pattern interference is folded into effective gradient noise, which is the claimed reason rare patterns converge faster. Parameter count is 2Nd, independent of |U|, and cheaper than per-user embeddings once |U| > 2N.
Experiments use Amazon Games, CDs, and Toys, split into five chronological periods of equal interaction count, with users bucketed into five activity quintiles from the previous period. Every method sits on TIGER. Baselines include plain continual fine-tuning, Replay, SAIL-PIW, PISA, and generative-specific RecICL, LSAT, and PESO.
Sample-weighted averages over four eval periods and five user groups:
| Method | Games R@10 | CDs R@10 | Toys R@10 |
| TIGER | 0.0558 | 0.0331 | 0.0666 |
| RecICL | 0.0686 | 0.0352 | 0.0530 |
| SAIL-PIW | 0.0572 | 0.0392 | 0.0689 |
| LION | 0.0724 | 0.0449 | 0.0769 |
| vs TIGER | +29.8% | +35.5% | +15.6% |
LION wins most cells, not all of them. On Games NDCG@10, RecICL scores 0.0470 against LION's 0.0454, so ranking quality still goes to in-context examples. Recall@10 gains on inactive users (G3–G5) are larger than on active users in every period, which matches the claim that isolation is mainly for the tail.
The ablation is blunt. Consolidation without sparse memory drops Games Recall@10 from 0.0558 to 0.0532: the auxiliary loss hits the shared representation and makes the conflict worse. Memory alone does nothing (0.0558). Memory plus full-history routing reaches 0.0586. All three pieces reach 0.0724, with consolidation as the main lift. After λ ≥ 0.3 both Games and CDs stay strong; Games peaks at λ=1.0, CDs at 0.5.
Gradient cosine backs the isolation story: TIGER's active/inactive gradients are often negative, LION's memory-layer gradients stay positive across four periods. The same pattern holds for popular versus long-tail items. Sweeping the memory from 4,096 keys to 65,536 and K from 8 to 32 moves Recall@10 by at most ±0.4%. Putting the layer at position 0 collapses Games to 0.0261; from layer 1 onward every seat is usable, with layer 5 as the default.
If a generative recommender is going to be updated weekly or daily, continual fine-tuning on a shared backbone will systematically under-serve inactive users. LION is a patch whose size does not grow with the user base: sparse memory as the isolation layer, consolidation as the tail booster. For teams already running a TIGER-style ID generator, this is an architectural add-on, not a new paradigm.
It is an incremental gain. RecICL can still beat LION on Games NDCG, so in-context demonstrations are not obsolete. Absolute Recall@10 sits in 0.04–0.08, far from a production ranking list. The value is that the conflict is measured and partly unpacked, not that a new SOTA curve appeared.
The backbone is T5 TIGER only; whether LLM recommenders or HSTU-scale models behave the same is listed as future work. Routing uses a fixed Top-K, so complex and simple users spend the same memory budget. λ for consolidation is chosen per dataset; there is no universal default.
Three Amazon review categories with equal-count period cuts are not production traffic. Baselines were rewritten onto TIGER, which may flatten methods that were not generative to begin with. Key specialization is reported as an observed trend, not a guarantee; ρ → 0 depends on data and optimization. There is no explicit user ID, and stacking with a light user adapter is untested.