Retrieval Grounding Latent Reasoning for Dense Retrieval
Gang Zhou, Xiongxi Yu, Hu Tian, Yang Wei, Lu Pan, Ke Zeng, Shibiao Xu, Xiaolong Zheng
cs.AI
2026-08-14
Meituan's RGLT grounds each step of a dense retriever's latent reasoning trajectory in the actual ranking gain its explicit-CoT counterpart produces, beating the strongest prior latent-reasoning baseline by 14.4% nDCG@10 on BRIGHT.
Retrieval tasks that require reasoning to judge relevance, like determining whether a piece of code actually solves a given math problem, can't rely on semantic similarity alone. Query and target document may share almost no surface vocabulary, with relevance only becoming visible after multiple reasoning steps. One prior approach generates an explicit chain-of-thought and encodes the expanded text for retrieval, but that introduces autoregressive decoding latency and makes retrieval quality sensitive to the exact wording of the generated rationale. A newer line moves reasoning into hidden space, using a sequence of silent tokens that never get decoded to text, cutting the generation cost. But supervision there typically either targets only the final retrieval objective or just requires latent states to resemble their explicit-CoT counterparts. The gap: resembling a state and actually producing a retrieval improvement are different things. A model can learn a latent trajectory that looks like reasoning while none of its individual steps improve ranking at all.
RGLT appends a fixed sequence of K silent tokens to the encoder input, letting their hidden states evolve layer by layer into a non-autoregressive latent reasoning trajectory, then uses the final token's hidden state directly as the query embedding, no extra pooling layer needed.
A dedicated anchor token first compresses the query and retrieval instruction into a unified context vector, which then modulates the silent tokens' key/value representations at every layer through a gated low-rank residual, keeping the reasoning trajectory anchored to the retrieval intent throughout rather than drifting toward generic semantic understanding.
Training combines three objectives:
| Benchmark | Metric | RGLT | Best prior latent baseline (LaSER) | Gain |
| BRIGHT (12 domains, 1,384 queries) | nDCG@10 | 34.20 | 29.90 | +14.4% |
| BRIGHT | Recall@10 | 39.97 | 34.27 | +16.6% |
| FollowIR | Avg. Score | 13.70 | 12.50 | -- |
| BrowseComp-Plus | Recall@1000 | 70.83 | 66.90 | -- |
RGLT beats LaSER on 9 of BRIGHT's 12 domains, with clear leads on AoPS, psychology, robotics, and both TheoremQA subsets; it trails LaSER only on biology, earth science, and StackOverflow, by narrow margins. The improvement is broad rather than driven by a single domain.
Ablations confirm retrieval-effect credit transfer is what matters. Replacing it with conventional state alignment drops nDCG@10 from 34.20 to 30.64; removing stage-level retrieval-effect matching entirely drops it to 31.26; a naive terminal-state-only variant with no stage supervision only reaches 30.87. Both design choices, stage-wise supervision and grounding in ranking effect rather than state similarity, contribute independently.
Retrieval quality also accumulates progressively across the reasoning trajectory: nDCG@10 at the four stage-terminal states climbs 32.25 (ST4) to 32.48 (ST8) to 33.56 (ST12) to 34.20 (ST16), starting from a base-query-state score of 26.39, showing retrieval capability builds up along the chain rather than jumping only at the final step.
On efficiency, RGLT adds just 12% query-side latency (24.0ms vs. 21.5ms for a base retriever with no reasoning). Explicit CoT generation before retrieval (Rewrite-then-Retrieve) is 186x slower (4000ms); LaSER, the other latent-reasoning baseline, still carries a 1.51x penalty from exposing all K silent positions at once.
For teams building retrieval-augmented generation or any retrieval setup that needs multi-step reasoning to judge relevance (legal, code, math, complex QA), this offers a way to fold reasoning into the query embedding in a single forward pass, no autoregressive text generation at inference, with almost no latency cost. The more transferable lesson is methodological: when replacing explicit CoT with latent-space reasoning, matching intermediate-state similarity isn't enough, supervision needs to be tied directly to whether each step produces a measurable downstream gain. That principle likely generalizes to other latent-reasoning applications beyond retrieval.
Experiments run on a single backbone, Qwen3-8B; the paper doesn't report whether the same gains hold at smaller or larger scales. Performance peaks at K=16 silent tokens and drops at K=24, with the paper offering only a one-line explanation (an excessively large K may complicate optimization) rather than analyzing why this non-monotonic turn happens, a sensitivity that often means the optimal K won't transfer directly to other data distributions. Training depends on an external LLM to generate the explicit CoT that supervision is built from, and the paper doesn't discuss the cost of that pipeline or how sensitive results are to CoT quality. Some ablation variants, like the gated retrieval-credit fusion variant, are independently trained architectural variants rather than strict one-factor ablations, a caveat the paper flags itself, which weakens how much causal weight the ablation table can bear.