Centroid SID-token init lifts cold-item Recall@5 by 60% in LLM-based generative recommendation

Preserving Item Semantics for Free: Rethinking Token Initialization in LLM-Based Generative Recommendation

Donald Loveland, Liam Collins, Bhuvesh Kumar, Danai Koutra, Neil Shah

cs.IR, cs.LG

2026-08-08

Initializing semantic-ID tokens from semantic centroids instead of random vectors lifts pure-SFT Recall@5 by 6.5% on average and cold-item Recall@5 by up to 60%, at zero overhead.

What problem this solves

LLM-based generative recommendation represents items as semantic IDs (SIDs) added to the LLM vocabulary as special tokens. Ideally these SID tokens carry semantic priors into the model and help generalization. But standard vocabulary expansion initializes the new tokens as random Gaussian vectors, discarding the continuous semantic geometry the SIDs carried and forcing the LLM to relearn item relationships from interaction data. The authors first demonstrate two failures: training from this random initialization organizes SID embeddings around item popularity rather than semantics, and even expensive continual pretraining (CPT) fails to reliably recover the original semantic geometry. This undermines the whole selling point of bringing LLM pretraining semantics into recommendation.

Method

The fix is almost free: initialize each SID token embedding directly from its corresponding centroid in the semantic embedding space. For semantic code k at level l, e = Phi(c{l,k} - cbar) + mubase, where c is the centroid, c-bar the mean across centroids, mu-base the mean of the original LLM token embeddings, and Phi a projection to model dimension. Two design choices matter:

Mean shift: recenter the centroids on mu-base so the new tokens land inside the pretrained vocabulary distribution, preserving the training stability of random initialization.
Truncation for dimension matching: modern text embeddings are Matryoshka (MRL, with leading dimensions carrying most information), so instead of a learned projection that could distort geometry, the method simply truncates to the first dmodel dimensions, with zero parameters and zero overhead.

SID construction uses residual k-means (not RQ-VAE), which already operates in native semantic space, so centroids are available for free. Algorithm 1 is about six lines: load centroids, mean-shift, and write them into the SID rows of the embedding matrix, with no extra training or inference cost.

Results

Across five datasets and both 0.6B and 4B models. Several groups of numbers:

On semantic purity, random initialization reaches only 0.24-0.37 on 10-nearest-neighbor purity P10 (random baseline 0.10); centroid initialization often doubles it. Spectral analysis quantifies the cause: under random init the geometry is small-scale, dominated by popularity (R32(Tpop) >= 0.80) and captures almost no semantics (R32(Tsem) <= 0.22); centroid init reverses this, with Tpop dropping from 0.88 to 0.00 on Beauty and Tsem rising to 0.64.

Under pure SFT (no CPT), Recall@5 rises 6.5% on average across five datasets and convergence is 27% faster; Beauty gains 15.9% with 40% fewer steps, Toys 7.4% with 33% fewer steps. Cold items (training-target frequency <=1) benefit most: under pure SFT, Recall@5 rises 37% on average, MovieLens +60%, Steam +50%, Beauty +23%; even with one CPT epoch the cross-dataset lead holds at 14%.

Where CPT is used, centroid initialization peaks at 1-4 epochs versus 6-10 for random, cutting CPT epochs by up to 80%. The 4B model is consistent: Toys pure-SFT +9.8% with 43% faster convergence.

Why it matters

For anyone building LLM generative recommenders, this is a textbook case of checking the defaults first. A random initialization taken for granted is systematically steering the model to learn around popularity and leave cold items behind, and the fix is a few lines, zero overhead, parameter-free. Cold-item +37% (up to +60%) and 80% fewer CPT epochs are direct wins for the two costs that hurt most in recommendation: cold start and long CPT training. The paper also hands over a usable diagnostic: spectral analysis of whether embeddings are popularity- or semantic-driven can flag a bad initialization early.

Limitations

The authors list several. Beauty is an exception: under CPT+SFT, random initialization reaches marginally higher absolute performance after 10x more CPT epochs, so some datasets benefit more from heavy CPT relearning. The relative advantage of centroid init shrinks as CPT epochs grow, with smaller gains past 4-6 epochs (CPT eventually helps even randomly-initialized models learn cold items). Evaluation covers only CPT and SFT, not later post-training such as RL. Centroid init only dampens popularity encoding, not eliminates it (some residual shows in Table 2). Readers should also note the gains are measured under their residual k-means SID construction; whether they transfer to other SID schemes needs verification.

Terms

Semantic ID (SID): an item encoded as a sequence of discrete semantic tokens added to the LLM vocabulary; generative recommenders generate it to recommend.
Continual pretraining (CPT): further pretraining of the vocabulary-expanded model before SFT so it learns the new tokens; usually expensive.
Centroid init: initialize each token from the cluster center of its semantic code in embedding space, replacing random vectors.
Matryoshka representation (MRL): nested embeddings where the leading dimensions already work as a shorter vector; most information sits up front.
Cold item: an item with very few (frequency <=1) training occurrences; the hardest to recommend and the strongest test of generalization.
Residual k-means: layer-by-layer residual clustering to build semantic IDs, operating in native semantic space so centroids come for free.