Google's Retrieve-for-Train Moves Search Thinking to Training Time for Sub-second Responses

Google Research released Retrieve-for-Train (R4T), accepted at ICML 2026 (paper dated September 15). It shifts "online thinking" in AI search into offline training: instead of costly autoregressive thinking budgets at inference time, RL trains a fan-out language model to produce expert query-expansion slates, rewarded on set-level quality (diversity and coverage), and the learned behavior is distilled into a small 53.9M-parameter diffusion model that generates multiple retrieval expansion directions in a single forward pass.

Confirmed

• Facts consistently reported by @aigclink, @dejanseo, @imjustnewatai, @gaganghotra and @steveplunkett: complex search responses drop from nearly 50 seconds to sub-second-to-second level, 12-20x faster than autoregressive approaches, with the paper claiming up to 20x latency reduction.
• The paper was released by the Google team with an article date of September 15.
• @dejanseo added the motivation: naive LLM fan-out queries suffer quality issues, which R4T constrains via set-level rewards for diversity and coverage.
• @imjustnewatai confirmed the 53.9M-parameter diffusion model that produces multiple retrieval expansion directions in parallel, speeding up query expansion 12-20x.

Why it matters

• The method exemplifies a "think at training time, instant at inference" paradigm: one-off RL training plus distillation buys order-of-magnitude latency reductions at inference, directly relevant to cost and responsiveness of large-scale online search systems.
• Replacing heavy autoregressive inference with a lightweight diffusion model also offers a transferable recipe for other retrieval and agent scenarios requiring multi-way generation or expansion.

2026-09-16 ~ 2026-09-16 · 6 related posts

• [source] Google's 53.9M-param diffusion model speeds search query expansion 12-20x — imjustnewatai · 2026-09-16
• Google's Retrieve-for-Train replaces heavy autoregressive inference with a lightweight RL-trained diffusion model — gaganghotra_ · 2026-09-16
• Google's Retrieve-for-Train cuts complex AI search from ~50s to sub-second — aigclink · 2026-09-16
• [source] Google's R4T: RL-trained diffusion retriever cuts fan-out query latency up to 20x — dejanseo · 2026-09-16
• Google's Retrieve-for-Train uses diffusion models to make complex AI search 20x faster — steveplunkett · 2026-09-16

1 near-duplicate retellings: aigclink