Robust Fusion of Semantic and Behavioural Signals for LLM Reranking in Personalised Search
Aleksandr V. Petrov, Nathan Stein, Erik Lybecker, Emma Schüldt, Daniel Lazarovski, Hugues Bouchard, Mounia Lalmas
cs.IR
2026-09-22
Spotify injects QSS into a 0.6B reranker for +13.3% NDCG@7; dual-sample dropout keeps the gain and is 4.0% better with QSS removed. Live success +2%.
Personalised search has to honour the query and the user. Production rankers have long leaned on behavioural statistics such as click-through and success rates. LLM cross-encoders make it easy to dump those numbers into the prompt as text tokens. The failure mode is shortcut learning: the history is so predictive that the model stops learning semantic and user-context patterns, then collapses on the long tail and cold start where the statistic is missing.
Spotify pins the problem on one production feature. Query Slice Stats (QSS) is the historical search-success rate for a query–item–country slice. It is strong on frequent pairs and sparse or absent for rare queries and new catalogue items. The question is not whether stats can enter a prompt. It is whether the reranker then forgets how to rank without them.
The reranker is a 0.6B open-source language model with a Semantic-ID-extended vocabulary. It scores music, podcast, and audiobook candidates from the production retrieval stack in a pointwise setup. The prompt carries three groups of evidence: query and candidate text (title, creator, type, Semantic ID); user context (market, recent listens); and bucketed popularity plus QSS. The continuous success rate is cut into five ordinal tiers from verylow to veryhigh; missing values are omitted. The model emits Y or N at the decision position, and the log-probability of Y is the ranking score.
Training is standard SFT with loss only on the label token. Dual-sample feature-dropout turns every QSS-present example into a pair: x+ keeps all features, x− deletes only QSS, same label. The mixture is α·L(x+)+(1−α)·L(x−). α=1 is naive always-on injection; α=0 never sees QSS in training. Random dropout is a supporting ablation whose expected exposure can match 1−α; the paper foregrounds the deterministic pair because the two views are matched. Serving is unchanged: one forward pass, same prompt interface. The extra compute is training-only.
Logs cover two weeks of production search. Train: 1M sessions, 452k queries, 46.1M candidates. Eval: 10k sessions. A session has 514 retrieved candidates and 1.3 positives on average. 34.7% of eval queries are unseen in training; 97.8% are long-tail (frequency ≤500). Metrics are NDCG@7 and Recall@7, the cutoff of the unscrolled results pane.
| Method | NDCG@7 all features | NDCG@7 QSS removed |
| BM25 / monoT5 | 0.105 / 0.104 | n/a |
| QSS only | 0.494 | n/a |
| α=0 (no QSS in training) | 0.549 | 0.550 |
| α=0.5 (dual-sample) | 0.622 | 0.547 |
| α=1.0 (always inject) | 0.618 | 0.526 |
α=0.5 is +13.3% NDCG@7 over α=0 with all features. Naive injection is +12.6% NDCG@7 and +9.5% Recall@7, but falls to 0.526 with QSS removed, significantly below the 0.550 of a model that never saw QSS. Dual-sample stays at 0.547, indistinguishable from 0.550 (p=0.15) and 4.0% above naive injection. Recall@7 for α=0.5 is 0.756 with features and 0.684 without, versus 0.686 for the no-QSS baseline. QSS alone already scores 0.494, far above the text anchors, so the shortcut is real; α=0.5 still beats it by 0.128.
On frequency slices, dual-sample gains about +0.061 NDCG@7 on cold-start queries versus +0.055 for always-on, and the edge holds through the tail up to frequency 500, reversing only on the short head (>500). Versus always-on, long-tail queries are +0.44% (95% CI 0.12–0.75%); the short-head point estimate is −0.55% and not significant.
Online A/B/C: dual-sample +2.05% search success, always-on +2.33%. The 1σ intervals overlap, so the aggregate test does not separate them. On the cold-start item slice dual-sample is +0.75% versus always-on, with zero near the 1σ edge. Direction matches offline; the experiment was not powered for fine slices.
A strong behavioural token changes what the model learns, not just what it sees. Production LLM rerankers should be scored with the feature present and with it stripped, then sliced by head, tail, and cold start. Dual-sample moves the robustness cost into training. Serving stays one model, no fallback ranker.
α=0.5 already captures most of the QSS lift. Always-on stats are competitive on the head and thin out semantics when stats vanish. The live 2% success lift and the offline 13.3% are different units, and the aggregate test cannot pick a training recipe. The diagnostic to watch is QSS-removed evaluation plus cold-start slices.
Full QSS removal is a stress test, not every partial-missing, stale, or noisy production case. User context is not isolated from the rest of the prompt, and there is no learned fusion baseline. The online slices are underpowered. The study is one statistic, one reranker, one platform.