SpeakGR: Teaching Generative Retrievers Semantic IDs Without Forgetting Language, Cutting Drift 80%+
_reachsumit · x · 2026-09-29
A new arXiv paper (Fu et al., with Hamed Haddadi) introduces SpeakGR, a dual-objective framework tackling a known pain point of generative retrieval: retrieval-only fine-tuning over-specializes pretrained LLMs to semantic ID (SID) prediction and distorts their natural-language generation.
- It combines supervised SID learning with speak-preserving regularization: an on-policy distillation objective aligning the current model with a frozen copy on student-generated prefixes via forward KL;
- Adaptive SpeakGR dynamically tunes the preservation strength based on observed language drift;
- Versus SFT-only, WikiText-2 forward KL drops 81.3-93.8% on MS MARCO and 81.2-85.2% on Natural Questions while retaining effective retrieval across three LLMs; the adaptive variant improves retrieval further in most settings.
More from Research
- Wavestone reverse-engineers 11 coding agents, maps the same 7-part harness inside all of them — blaizedsouza · 2026-10-24
- Looped Transformers heat up: 10 new papers on recurrent loop architectures — TheTuringPost · 2026-10-24
- FormalizeSampling: Sampling and diffusion convergence theorems machine-checked in Lean 4 — TianbaoX · 2026-10-24
- Alison Gopnik: What Kids Are 'Bad' At May Be Features, Not Flaws — AlisonGopnik · 2026-10-24
- Explicit Diophantine equation provably equivalent to Con(ZFC) constructed, Lean proof in progress — IgorCarron · 2026-10-24
- Study Estimates LLMs Generate $123-$417 of Economic Value per Megajoule — dan_s_becker · 2026-10-24