OPDSearch+: distill a frozen teacher then refine with RL to beat RL-only training
_reachsumit · x · 2026-08-26
arXiv paper OPDSearch+ proposes the first distillation paradigm for search-augmented reasoning that requires no teacher fine-tuning. Motivation: collecting multi-turn search trajectories for SFT is prohibitively expensive, while on-policy distillation with an off-the-shelf teacher is unstable and caps the student at the teacher's ceiling. Two stages: the student interacts with a live search engine and is distilled via per-position forward KL from a frozen instruct model, transferring reasoning decomposition and evidence-integration skills; then RL refines the distilled student. Key insight: the teacher reshapes the student's policy distribution so subsequent RL converges to solutions RL alone cannot reach, especially on multi-hop questions.
More from Research
- How LLMs Self-Correct Mid-Generation: The Role of Reasoning RL and Instructions — dejanseo · 2026-08-26
- U. de Chile Students Publish Book on Maturana and Varela's Relevance in AI — PolarBearby · 2026-08-26
- Face Anything: 4D Face Reconstruction from Any Image Sequence (ECCV 2026) — rsasaki0109 · 2026-08-26
- Gemini 3.7 Flash helps revive interactive CMA-ES explainer site — doodlestein · 2026-08-26
- From PDE Numerical Solvers to Neural Emulators and Back: PhD Thesis — chaumian · 2026-08-26
- Combining PSGD-Kron and KL-Shampoo yields an optimizer without eig/inverse/solve — YouJiacheng · 2026-08-26