Rethinking Item Tokenization in Generative Recommenders: From Fixed Atoms to Semantic Subwords
Xinrui Miao, Mingjia Yin, Jiaqing Zhang, Wei Guo, Yong Liu, Yuyang Ye, Hao Wang, Enhong Chen
cs.IR
2026-08-24
SST compresses history into semantic subwords and keeps fixed SID targets. TIGER-KM Beauty HR@5 rises 0.0372 to 0.0423; variable targets drop merged items to 0.0002 HR@10.
Generative recommenders encode items as fixed-length Semantic IDs such as ⟨a⟩⟨b⟩⟨c⟩⟨d⟩ and decode the next item token by token. That grammar is convenient for beam search. The same atom sequence is then reused for every item in user history. Flattening turns n items into n×L tokens, and the encoder spends attention reassembling atoms that already belong together before it can model item-level transitions such as sports gear to wearables. The paper calls this Intra-item Attention Overload.
Variable-length SID methods such as SARQ and VSID change code length by item discriminability or popularity. They do not target this history-side attention waste. SST’s claim is narrower: compress history, keep the target grammar fixed.
SST has two stages, both encoder-only.
Item-level Subword Tokenization (IST) learns merges inside an item. Adjacent atom pairs are scored by frequency and item support, then merged under BPE, WordPiece, or CondEntropy, a rule that wants mutual predictability and low local conditional entropy. A merged span becomes one semantic subword; uncovered atoms stay. Pairs never cross item boundaries. History becomes shorter and variable-length. Targets remain the original L-slot SID.
Behavior-induced Co-occurrence Augmentation (BCA) then spends the freed capacity. The first P=2 SID tokens of each item are a coarse prefix. Frequent prefix transitions in a sliding window are mined, and the intervening history spans are replayed with the right-hand item as the target. Compression removes low-order reassembly; replay points the model at inter-item moves.
Backbones are TIGER-VAE, TIGER-KM, and LETTER, each with 4 codebooks of size 256. Datasets are 5-core Amazon Beauty, Instruments, and Yelp, leave-one-out, full-catalog ranking, beam 20. Controls include fixed SIDs and history-only ports of SARQ and VSID that keep fixed targets.
SST beats the strongest non-SST baseline on all three backbones and datasets. The largest cell is TIGER-KM on Beauty: HR@5 from 0.0372 to 0.0423 (+10.4%), NDCG@5 from 0.0238 to 0.0281 (+13.3%). TIGER-VAE on Beauty is +7.6% HR@5. LETTER is strongest in absolute terms and gains less: +5.1% HR@5 on Beauty, only +1.4% HR@10. IST and BCA each usually beat fixed SIDs; together they win most columns, but they are not additive. On LETTER / Beauty, IST alone reaches 0.0444 HR@5 against 0.0431 for full SST.
| Backbone / data | Fixed HR@5 | SST HR@5 | vs best baseline |
| TIGER-KM / Beauty | 0.0372 | 0.0423 | +10.4% |
| TIGER-VAE / Beauty | 0.0357 | 0.0384 | +7.6% |
| LETTER / Beauty | 0.0400 | 0.0431 | +5.1% |
| LETTER / Yelp | 0.0266 | 0.0287 | +7.9% |
Attention probes match the story. On TIGER-VAE, SST lowers Intra-Item Attention Budget and Atom Reassembly Load; leftover same-item attention moves onto subword tokens. Coverage buckets show larger NDCG gains when more history tokens are subwords. Efficiency is roughly a wash: Beauty mean history length 28.61→26.73, wall time 0.718h→0.659h, extra BCA samples offset by shorter sequences.
Asymmetry is load-bearing. On Yelp with TIGER-KM, variable history plus fixed targets scores 0.0412 HR@10; variable targets drop to 0.0329. Split by whether the target contains a merged subword: the merged bucket falls from about 0.04 HR@10 to 0.0002, while unmerged items still score 0.0530. Subwords are long-tail decoder symbols and lose the early beam to frequent atoms.
Item tokenization is usually treated as a tokenizer detail. SST offers a practical split: keep a fixed decoding grammar on the target, and spend subwords on history. That is cheaper than converting every SID to variable length, and it avoids the long-tail target-token failure. The gains are incremental. The best cell is about 10% relative HR@5, and LETTER moves less. Swapping the tokenizer is not a new generation of model.
Backbones are small T5-style models with hidden size 128, trained up to 200 epochs with validation-loss model selection. There is no industrial-scale model and no online A/B. SARQ and VSID are history-only ports, a test of “is shortening enough,” not a full reproduction of those papers. Subword count and BCA top-k are searched on validation loss; too many merges add noisy subwords. On LETTER, IST alone sometimes beats IST+BCA; the paper does not isolate how the two modules interact.