Seven days of continual training cut Day-1 HR@50 from 39.09 to 0.19 on KuaiRecV2

KuaiRecV2: Benchmarking Large-Scale Continual Learning for Diversified and Multi-task Recommendation

Chenxu Li, Shuchang Liu, Hantao Shu, Wei Yuan, Zhe Xu, Guoqing Hu, Yan Wang, Biao Yang, Xiang Li, Cheng Ling, Fan Yang, Yongqi Liu, Lantao Hu, Han Li, Kaiqiao Zhan, Kun Gai, An Zhang, Xiang Wang

NeurIPS 2026

cs.IR

2026-11-12

Kuaishou and USTC's KuaiRecV2 has 484M interactions over 30 days. Without replay, Day-1 HR@50 falls from 39.09 to 0.19 in seven days; 5% replay holds 3.20.

What problem this solves

Generative recommenders turn each video into discrete tokens, generate those tokens, and map them back to a video. Hit rate on a static split can look strong. It comes apart on a short-video feed when the offline set shuffles time, because the model then trains on interactions that had not happened yet and is scored on the past.

On Kuaishou the catalog is not a fixed list. New videos enter every day, and a video drops out once it is seven days old. The model has to keep early preferences while that pool moves. Each video is several tokens deep, so a bias in one token multiplies through the rest, and the accuracy-diversity tradeoff is harder than a list-level rerank. Business categories also do not line up with the semantic ID. Click, long view, like, and hate need to move together, but a generative model fits the next item and does not expose one head per behavior.

Method

KuaiRecV2 comes from Kuaishou and the University of Science and Technology of China, in the NeurIPS 2026 evaluations and datasets track. Users are sampled uniformly and kept only if they have at least 100 interactions in the 30-day window. The full month holds 124,055 users, 24,898,648 videos, and 484,045,145 interactions, at 99.9843% sparsity and about 3,902 interactions per user. The first two days form Small, about 2.45 million videos and 38.96 million interactions. One further day is held out for the leaderboard.

Labels cover short view, complete view, like, comment, follow, forward, search, profile enter, and hate. Three item features are frozen, and the models that built them never trained on these logs: a 64-d multimodal vector, an 8-level semantic ID from RQ-VAE, and tags from a multimodal LLM. A semantic ID is a stack of discrete codes. The generator emits that stack and maps it back to a video. User profiles are removed.

Training rolls forward by day. Day 1 produces θ1. Each later day starts from yesterday's parameters and trains only on the new day. Histories are spliced across midnight. On day t the pool keeps videos created from day t-6 through day t. Validation is that user's last interaction of the day. A checkpoint from day k is scored again on day 1, which is the forgetting probe.

NDCG and HR@K measure click retrieval. HR@K is the fraction of cases where the held-out next item sits in the top K. Diversity adds item coverage, the Gini coefficient, and token-level path diversity, unique tokens per layer, and entropy. Every behavior gets its own ranking scores, but one model must produce the list for all of them. A gain counts only when every metric improves. Baselines are SASRec, GRU4Rec (listed as GRURec), TIGER, LLaDA-Rec, and MiniOneRec at 0.5B. The last three generate semantic IDs. LLaDA-Rec decodes them in parallel with discrete diffusion. Training compares next-item SFT with RL on a weighted sum of behavior rewards. The reward is a training signal, not the reported metric.

Results

A shared semantic ID does not absorb a pool change. MiniOneRec's loss jumps at each day boundary. The first 20 steps of day 1 start at a loss of 1.236, the day then settles, and the next boundary jumps again.

Day-1 HR@50 after further training is below. The paper does not scale these entries by 10^{-3}. Do not set them next to the percentages in the static table.

MethodDay 1Day 2Day 3Day 5Day 7
MiniOneRec39.0917.896.6250.410.19
EWC LoRA38.6722.937.180.600.20
5% replay39.0925.2410.265.453.20

Bare continual training falls from 39.09 to 0.19, about 1/206 of the day-1 score. By day 4 it is already 1.03, while replay still has 5.59. EWC LoRA puts an importance penalty on the low-rank adapter and only helps through day 5. On day 7 it is 0.20. Replaying 5% of past instances holds 3.20, about 17 times the bare model and under a tenth of day 1.

On Small, the static table is the mean of seeds 42, 2026, and 999. Every entry is scaled by 10^{-3}.

MethodSizeNDCG@10HR@10HR@50
SASRecnot reported0.971.966.61
GRURecnot reported1.202.7110.99
LLaDA-Rec5.3M3.106.4717.31
TIGER-Tiny4.9M4.018.0120.52
TIGER-Small30M7.3715.5841.07
MiniOneRec0.5B12.0123.5962.63

MiniOneRec's HR@50 is 6.26%, about 9.5 times SASRec at 0.66%. LLaDA-Rec at 5.3M already beats GRU4Rec. TIGER-Tiny's HR@50 has a standard deviation of 3.27, so the gap to LLaDA-Rec at 17.31 does not clear seed noise. MiniOneRec at 62.63 versus TIGER-Small at 41.07 does.

Gini is computed only on items that appear at least once, and it is still high. Gini@50, times 10^{-2}, is 92.41 for LLaDA-Rec, 95.87 for TIGER-Tiny, 93.10 for TIGER-Small, and 90.69 for MiniOneRec. Coverage@50, times 10^{-4}, flips the ranking: MiniOneRec, the most accurate model, covers 18.80, and LLaDA-Rec covers 98.86.

On day 21, for clicks, MiniOneRec's top 50 lists use only 12.47 distinct tokens at layer 1 of the 8-level code. Path diversity is 12.47 at layer 1 and 42.84 at layer 3, then stalls between 45.93 and 48.69 from layer 4 on. Later tokens add almost no new combinations.

Days 1 through 19 are click SFT only. From day 20, alternating SFT and RL beats continued SFT. RL alone rises on day 20 and drops once the pool changes on day 21. Multi-behavior RL does not survive the shift unless SFT stays in the loop. The figure does not list daily deltas. On the day-21 SFT checkpoint, times 10^{-3}, HR@50 is 47.60 for click, 51.50 for long view, and 50.99 for like. Hate is 0 at every cutoff. Search reaches an HR@50 of 55.77 with an NDCG@1 of 0.

Why it matters

The usable piece is the protocol: a 7-day sliding pool, daily updates, diversity at both token and item level, and one model scored on every behavior. A hit-rate-only gain does not count. KuaiRec in 2022 had 1,411 users, 3,327 videos, and about 4 million interactions, and it evaluated a fully observed offline set. Interaction count here is about 120 times larger, and the video count grows from a few thousand to about 24.9 million. FORGE is larger still, at 14 billion interactions, but it scores semantic-ID quality rather than continual recommendation.

Semantic IDs do not remove the loss spike when the pool changes. Five percent replay beats EWC and still sits at 3.20 on day 7. The 0.5B model with the best hit rate also has the worst coverage. Reporting HR alone picks the narrower list.

Limitations

The appendix states three limits. User features are dropped for privacy, the public log is simpler than production, and offline evaluation still does not replace an online test. There is no cross-domain split, so transfer is untested.

Table 3 never says whether HR@50 is scaled by 10^{-3}, so 39.09 cannot be set next to the 6.26% in Table 4. The forgetting table stops at day 7, the loss plot is detailed for five days, and the RL claim has no daily number table. The n=100 filter drops light users. About 3,902 interactions per user is not the platform average. Gini ignores videos that were never recommended. Hate stays at 0, which shows that a click generator is not retrieving against negative feedback. The checklist says no significance test was run. Semantic IDs are frozen, so a better tokenizer would not show up. The pool jumps once per calendar day, while production changes continuously, and nothing measures how that choice moves the forgetting curve.

Terms

Source

What people are saying

Related papers

All paper explainers