Teacher Retains Full Tokens, Student Merges Efficiently: TM20K for E-Commerce Sequence Modeling in Ad Recommendation
Xinchun Li, Duoru Zheng, Wenlin Zhao, Ziyi Zhou, Jingxuan Tan, Huizhi Yang, Linlan Chen, Dongjian Wang, Dongyue Wang, Xiaosong Li, Hongyue Mao, Yaocheng Tan
cs.IR
2026-08-07
ByteDance TM20K brings full attention plus three token-merge rules and one-time teacher distillation to 20K-length ad sequences, gaining +1.036% ADSS online at only +5.6% latency.
Recommenders guess the next item a user will click by modeling their historical behavior sequence. Longer sequences capture both long-term interest and immediate intent, so the industry keeps stretching them. The cost is steep: the authors measure that extending a sequence from 5K to 20K raises training time 3.5x, adds 49G of GPU memory, and inflates serving latency 6.3x, which is unworkable online.
Existing cheap routes both have flaws. Search- and cluster-based compression shortens the sequence but discards fine-grained behavioral information. Lightweight target attention looks only at history relevant to the candidate item and misses the intrinsic dependencies among behavior tokens. TM20K aims to keep both effectiveness and efficiency, getting sequences to 20K and still shipping.
The recipe combines full attention, token merging, and one-time teacher distillation.
Deployed in ByteDance's e-commerce ad recommender, with sequences extended to 20K. Online A/B:
| Setup | ADSS | ADVV | Serving latency |
| TM20K student (with distillation) | +1.036% | +0.780% | +5.6% |
| TM20K student (no distillation) | +0.881% | +0.515% | +5.6% |
Offline uses an industrial CVR prediction dataset with billions of samples. The 5K baseline reaches AUC 0.8212; the 20K full-token teacher reaches 0.8233 but throughput collapses to 11K. The distilled student hits AUC 0.8230, nearly matching the teacher, at throughput 83K (versus the 88K baseline, down only 5.7%), with mean sequence length compressed from 20K to 1.8K (P90 of 2.6K). Baselines compared: STCA, LONGER, MTFM, HyFormer (DIN and TWIN excluded for poor performance on ultra-long sequences).
Long-sequence modeling is the most direct effectiveness-versus-cost tradeoff in industrial recommendation. TM20K's deployable answer: use full attention where it helps, but push the compute burden offline through token merging and one-time teacher distillation, serving only the compressed student online. For ad and recommendation engineers, the three token-merge rules each come from a concrete observation about attention distribution and can be borrowed independently.
The authors concede two points: the token-merge rules are rule-based and need re-tuning when ported to other scenarios, and sparse attention is not adopted because it requires highly customized GPU kernels with heavy engineering cost. All results also come from ByteDance's internal e-commerce ad data and online experiments, with no public-dataset reproduction, so outside readers can only take the internal numbers at face value.