OPDSearch+: On-Policy Distillation with RL Refinement for Search-Augmented Reasoning
Qinglin Ye, Zhiyuan Gu, Jingjie Xia, Yiheng Zhang, Kaiyan Zhao, Shunchao Zheng, Yuhang Mu, Wenchao Du, Yiming Wang
cs.AI
2026-08-25
A frozen 14B instruct teacher distills a 3B student on live search traces, then GRPO; mean EM 0.4402 across seven QA sets, +13.1% on HotpotQA.
Search-augmented reasoning is still hard for small models. One line of work, Search-R1 and its descendants, uses outcome RL so the model learns when to search and what to query. Another line distills a larger teacher's trajectories. Sparse EM rewards make multi-hop decomposition hard for a 3B model to discover on its own. Distillation has its own tax: multi-turn search traces depend on live retriever replies, so offline SFT data barely transfers across retrievers or corpus versions. Training a task-specific teacher is expensive; using an off-the-shelf instruct model as teacher tends to cap the student and destabilize training.
OPDSearch+ treats the frozen instruct teacher as a distribution reshaper, not a performance ceiling. The claim is that later RL can then reach a solution that RL from the same base cannot.
The student is Qwen2.5-3B (base). The teacher is frozen Qwen2.5-14B-Instruct. Retrieval matches Search-R1: E5-base-v2 over the December 2018 Wikipedia dump, top-3 passages. Training data is NQ plus HotpotQA, about 170k pairs.
Stage one: the student rolls out against a live search engine, interleaving thoughts, queries, retrieved passages, and a final answer. Eight trajectories per question, at most four search turns. The teacher assigns token-level log-probabilities only on student-generated tokens; retrieved passages are state-masked out of the loss. The distillation objective is per-position forward KL, implemented as clipped importance-weighted cross-entropy: tokens the teacher likes more than the student get amplified gradients, with the importance ratio clipped to [1e-6, 10]. A KL penalty against the initial checkpoint keeps the student from drifting too far.
Forward KL is a deliberate choice. Reverse KL inflates gradients on tokens the student already samples often, which collapses entropy. Forward KL puts large weights on rarely sampled tokens and bounds the gradient second moment. Under matched hyperparameters, JSD and log-ratio variants hit NaN within 20 steps, clipped reverse KL degrades around step 60, and an entropy-adaptive reverse-KL variant stays stable but plateaus at 0.470 validation EM versus 0.493 for forward KL, which keeps improving past 400 steps.
Stage two drops the teacher. RL starts from the OPD checkpoint at step 150 and runs GRPO with reward 0.9 EM + 0.1 format. Training uses veRL on 4× H200 at 1e-6. Total cost is about 48 GPU-hours, comparable to Search-R1.
Mean Exact Match across seven QA benchmarks is 0.4402, above the previous best 3B result, GiGPO-Instruct at 0.421.
| Method | Mean EM | HotpotQA | 2Wiki |
| OPDSearch+ (14B then RL) | 0.4402 | 0.4580 | 0.4264 |
| AutoRefine-Base | 0.405 | 0.405 | 0.393 |
| GiGPO-Instruct | 0.421 | 0.369 | 0.370 |
| Reproduced pure RL | 0.3524 | 0.3467 | 0.3142 |
| Frozen 14B teacher, no training | 0.3985 | 0.3916 | 0.3513 |
| OPD only, no RL | 0.3655 | 0.3602 | 0.3281 |
HotpotQA is 13.1% above AutoRefine-Base, 2Wiki 8.5%. The gap is multi-hop: pure RL already reaches 0.514 single-hop average, but only 0.347 on HotpotQA and 0.314 on 2Wiki; OPD then RL lifts those to 0.458 and 0.426. The 3B student also beats the frozen 14B teacher (0.3985). Offline SFT then RL lands at 0.4185. Joint OPD+RL is 0.3660: the two objectives interfere.
OPD raises policy entropy entering RL from 0.82 to 1.35 and cuts mean search turns from 3.5 to 2.7. Bamboogle is the miss: GiGPO scores 0.641 versus 0.456 here, on a 125-example set.
For anyone training a small search agent, the usable recipe is sequential: on-policy forward-KL from a frozen instruct model, then RL. No task-specific teacher fine-tune, no hoard of multi-turn SFT traces. The multi-hop lift suggests sub-query decomposition transfers faster by distillation than by sparse EM exploration. Cost stays in the same band as pure RL.
This is a better initialization, not a new agent protocol. The interaction loop and GRPO stage are still Search-R1.
Stage-two RL still collapses; the run stops when collapse is observed. Distillation improves the starting point, it does not remove outcome-RL instability. The Bamboogle gap versus GiGPO is large and the set is tiny, so it should not be read as a multi-hop sweep. Evaluation is locked to 2018 Wikipedia plus E5, not the open web. Reverse-KL variants may be salvageable with objective-specific tuning; that was not exhausted. Teacher scale is only 7B and 14B, with the student fixed at 3B.