What Makes a Good Semantic ID for Generative Recommendation? A Reproducibility Study
Yufei Chen, Junchen Fu, Jujia Zhao, Yukun Zhao, Zhaochun Ren
cs.IR
2026-09-21
Reproducing 11 semantic ID designs under one framework: no universal winner, codebook utilization nearly uncorrelated with quality, backbone and code-length scaling both non-monotonic.
Generative recommenders represent each item as a short sequence of discrete tokens (a semantic ID, SID) and generate the next item autoregressively, TIGER-style. Since TIGER appeared in 2023, SID constructions have fragmented: residual-quantized RQ-VAE, optimized product quantization, multi-level balanced K-means, tokenizers learned end to end with the recommender. Every paper reports on its own datasets and backbones, so how much these design choices actually contribute has been unanswerable. A team from Shandong University, the University of Glasgow and Leiden University put 11 designs through one shared framework and released the code (SID-Repro).
Four datasets: Amazon Video Games, Microlens-50K/100K, Yelp, all under the same global temporal 70:13:17 split with warm-only evaluation. The broad comparison runs each method with its official hyperparameters; the controlled analyses fix the TIGER framework and swap only the tokenizer.
Four research questions:
Broad comparison, NDCG@10 (five rows):
| Method | Video Games | Microlens | Yelp |
| RPG (OPQ) | 0.0301 | 0.0173 | 0.0207 |
| TIGER (RQ-VAE) | 0.0272 | 0.0089 | 0.0172 |
| LETTER-TIGER | 0.0257 | 0.0121 | 0.0232 |
| SEATER | 0.0184 | 0.0189 | 0.0201 |
| SASRec (no SID) | 0.0152 | 0.0156 | 0.0177 |
No method wins on all three datasets. The uncomfortable row is SASRec: a conventional baseline using no semantic IDs beats TIGER, DiffGRM and OneRec on Microlens and Yelp.
Utilization: RQ-Kmeans (OneRec's construction) has a perfectly balanced first-level codebook, normalized entropy 1.00 with all 256 codes in use; TIGER sits at 0.90 with 161 to 166 live codes. Yet first-level entropy correlates with NDCG@10 at Pearson -0.02 and Spearman -0.08, which is no correlation at all. Adding a diversity regularizer to LETTER lifts entropy by about 0.11 and moves NDCG@10 by at most 0.002. Utilization is a diagnostic, not an objective.
Scaling fails in both directions. Growing the backbone from T5-small to large takes the OPQ route from 0.0293 down to 0.0239. On code length, RQ designs peak at L=3 (0.0272), RQ-Kmeans falls from 0.0276 at L=3 to 0.0207 at L=12, while OPQ prefers L=6 (0.0321). Longer autoregressive sequences leave more room for decoding errors.
On semantic preservation the two families split the field: OPQ best recovers neighbor sets, highest Jaccard@20 on all four datasets; RQ-Kmeans best preserves neighbor ordering, highest RBO@20 on all four. The paper calls this complementary, and no design dominates.
Numbers in this subfield have not been comparable across papers for three years, since datasets, backbones and tuning budgets all differ. Three takeaways are directly usable. Do not pick a tokenizer for codebook balance; the data says that intuition is wrong. Treat any single-dataset SID result as needing revalidation on your own data, because rankings flip. Keep SASRec in the baseline suite, since generative methods hold no universal advantage. For teams deploying generative recommendation, one round of controlled comparison on your own traffic beats chasing the next paper.
The authors list their own: RQ2 to RQ4 fix the TIGER framework, so conclusions may not transfer to other backbones or decoding rules; the broad comparison uses official hyperparameters without per-dataset tuning, so small gaps deserve caution; evaluation is warm-only with cold start untouched; compute limited backbone scaling to T5-large. One more from the setup: all four datasets are mid-scale consumer data, and behavior at much larger catalog sizes is unverified.