The Disconnect Between Better Descriptive Reasoning Trace Quality and Recommendation Effectiveness
Gustavo Penha, Juan Elenter, Claudia Hauff, Hugues Bouchard, Paul Bennett, Mounia Lalmas
cs.IR
2026-08-24
On Qwen3-1.7B, titles raise groundedness 1.58 to 4.07 while Games Recall@10 falls 20%. A denser LLM-judge reward recovers part of that drop; better traces alone do not.
Generative recommendation decodes the next item as tokens. Production stacks already use residual-quantized Semantic IDs on 10M-item catalogs. The next wave prepends a natural-language rationale; SIDReasoner and OneRec-Think are in that line. SIDs are not in the pretrained vocabulary, so they need alignment: SIDReasoner uses eight auxiliary tasks, and other work reports up to a 25% accuracy drop when grounding is weak.
The tempting story is that titles are words the LLM already knows, so the alignment tax is lower and reasoning should help more. This paper splits representation (SID vs title) from whether a trace is written at all, and asks a sharper question: if the traces read better, does leave-one-out next-item ranking improve?
One Qwen3-1.7B backbone, SIDReasoner’s Amazon Reviews 2018 splits, and the same GPT teacher traces. Three domains: Video Games (9,053 users / 3,858 items), Office, Industrial; 5-core; chronological 8:1:1; history cap 10. SIDs are 3-level RQ-VAE codes with 256 entries, 768 special tokens. Titles are truncated to 32 tokens and mapped back with BM25.
The factorial is SID / Title × no reasoning / reasoning. Teacher text is identical except for how items are named. Three stages: LoRA direct next-item prediction; full SFT on 〈think〉 traces mixed 50/50 with direct examples and a 20× loss weight on item tokens; GRPO over 16 samples with a prefix-match reward (SID levels or whitespace title tokens) and gradients only on item tokens.
The headline metric is Δ versus the matched no-reasoning baseline. Evaluation is full-catalog and item-level (ASIN). SID collisions expand to consecutive ranks so quantization ties cannot inflate hits. An appendix checks the title arm: reasoning does not lower the rate at which generations are faithful catalog titles (98.0% on both Office arms), so the drop is not a BM25 string mismatch.
Under SFT and prefix-match GRPO, adding reasoning yields negative or zero Δ on every domain and both representations. Titles drop more, the opposite of the alignment-tax prediction. Video Games Title Recall@10: 0.0783 without reasoning, 0.0627 after SFT (−20%), 0.0614 after GRPO. SID in the same domain goes 0.0728→0.0689, not significant. Office Title is a significant −11% Recall@10; SID is essentially flat.
Paying the full alignment tax does not fix ranking. SID-A copies SIDReasoner’s eight-task Stage 1. The no-reasoning model is stronger on Office / Industrial (R@10 0.1609 / 0.1416 vs 0.1595 / 0.1361 for the lean SID). Adding reasoning then costs 17–23% R@10, worse than lean SID. Trace quality does rise. Gemini 2.5 Pro scores six dimensions: lean SID 2.22 aggregate, SID-A 3.51, Title 3.33; groundedness moves 1.58→4.07. Logical coherence and recommendation justification stay near the floor. SID-A has the best traces and the worst ranking.
Prefix-match is sparse: 70–96% of prompts yield 16 zeros and no advantage. On Games/Title, GRPO shrinks mean trace length from 69.6 words to 1.1 as 〈think〉 blocks empty out. A GPT-4o-mini mix of 0.5 accuracy + 0.25 trace + 0.25 relevance lets SID beat the no-reasoning baseline on Office / Industrial (0.1646 vs 0.1595, +3.2%; 0.1401 vs 0.1361, +2.9%) and recovers about 41% of the Games gap. Title only ties on Industrial and never recovers Office. An E5 cosine substitute nearly matches the judge on Office / Industrial, so the working ingredient is denser reward, not judge semantics or gradients through reasoning tokens.
On production-scale playlist continuation (on the order of 10⁶ playlists and 10⁸ tracks), reasoning SFT is HR@30 0.6343 vs 0.6337 without reasoning, with slightly worse NDCG. Rank-aggregating the two candidate lists reaches 0.6515 HR@30. Reasoning swaps tracks in; it does not systematically raise the total.
| Setting (R@10) | Games | Office | Industrial |
| SID no reasoning | 0.0728 | 0.1595 | 0.1361 |
| SID reasoning SFT | 0.0689 | 0.1589 | 0.1324 |
| Title no reasoning | 0.0783 | 0.1587 | 0.1253 |
| Title reasoning SFT | 0.0627 | 0.1416 | 0.1240 |
| SID GRPO+Judge | 0.0705 | 0.1646 | 0.1401 |
Clearer CoT and better next-item ranking are different objects in this setup. Titles and eight-task alignment both improve readable traces and do not carry leave-one-out Recall with them. Denser rewards can recover ranking while trace scores barely move. If the product needs an explanation, optimize the explanation. If it needs next-item hits, inspect whether the reward is zero 90% of the time before wrapping the decoder in 〈think〉.
This is a small-catalog, small-backbone factorial plus one production-scale side study. It cuts the default arrow from “better traces” to “better recommendations.” It does not say reasoning is useless.
Only Qwen3-1.7B; larger models may flip the sign. Stage 2 distills GPT teacher traces rather than growing the model’s own reasoning, so teacher-free lines such as OneRec-Think may differ. Evaluation is offline single-ground-truth, so relevant-but-not-that-item predictions score zero and can diverge from LLM-judged relevance. The title arm uses BM25 against SID trie decoding; the appendix argues resolution is not the main effect, but the decoding stacks are still unequal. GRPO+Judge calls GPT-4o-mini in the training loop, with cost and judge bias. The playlist study never reruns the dense reward.