RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning
Yan Yu, Zhengxi Lu, Yizhou Liu, Yichen Pan, Aozhe Wang, Qipeng Chen, Hua Yang, Wenqi Zhang, Weiming Lu, Qianglong Chen, Yongliang Shen
cs.CL, cs.AI
2026-09-18
RetireOPD trains a skill teacher with rewards, jointly trains a skill-free student, then drops the teacher when alignment stalls. ALFWorld rises 14.1–18.8 points over GRPO.
Multi-turn agent RL scores a whole trajectory with one scalar, so intermediate tokens go unsupervised. On-policy distillation tries to fill that hole: a teacher that sees privileged task skills scores the student's own prefixes, and the student is supposed to internalize the skills so inference needs no extra context.
Two assumptions fail on ALFWorld and WebShop. A skill-conditioned branch is not automatically a better teacher. If it is never trained to use the skills, a 7B model prompted with them still sits at 23.4% success. Adding distillation to GRPO removes the slow start of pure RL and the low ceiling of pure distillation, but the teacher-student gap narrows then widens. Once the student has the behavior, reward gradients fight imitation. Fixed annealing or a preset two-stage switch withdraws too early on some runs and too late on others; the stall appears anywhere from step 50 to 90.
Teacher and student share architecture and initialization. The teacher sees skills from SkillRL's SkillBank, retrieved by keyword match. Three stages:
The teacher has to earn its privilege in the environment. After transfer stalls, it leaves so RL can keep going.
On Qwen2.5-1.5B/3B/7B, RetireOPD reaches 89.8%, 93.8%, and 95.3% ALFWorld success against GRPO's 72.8%, 75.0%, and 81.2% (+17.0 / +18.8 / +14.1). WebShop accuracy is 75.8%, 77.3%, and 84.4% against 56.8%, 63.3%, and 72.6% (+19.0 / +14.0 / +11.8). In every setting the student also beats its Skill-GRPO teacher (81.2% / 79.7% / 90.6% on ALFWorld).
| Setting | ALFWorld | vs GRPO |
| Qwen2.5-3B RetireOPD | 93.8% | 75.0% |
| same model, GRPO+OPD never retired | 82.8% | 75.0% |
| same model, OPSD with untrained teacher | 28.1% | 75.0% |
On the 3B run, retirement hits around step 60 at 76.6% success; GRPO alone then climbs to 93.8%. Keeping the joint objective plateaus. Switching to distillation-only shrinks the gap and does not raise task score. Linear annealing rises early and then stalls. Dropping either retirement signal yields 89.0%; never retiring yields 82.8%. For γ in 0.80–0.96 and δ in −0.10–0.04, the retirement step stays within ±5 of the default.
For anyone distilling privileged agent skills, the teacher should be reward-trained first, and the off-ramp should read training signals rather than a calendar. The student infers without skill context and still passes the skilled teacher, which means the skills landed in the weights and later RL found actions the teacher does not take. This is a training-schedule improvement, not a new environment or backbone.
The study stays on ALFWorld and WebShop with Qwen2.5 1.5B–7B. Skills come from a fixed SkillBank with keyword retrieval; a messier catalog may not produce a readable retirement point. γ and δ are insensitive but still hyperparameters. Reverse KL is a sampled-token estimate, not a full-vocabulary sum. There is no dedicated limitations section; the bounds follow from the setup: no GUI, no code agents, no larger models, no cross-task transfer.