SAF-OPD: Stable Advantage Fusion for On-Policy Distillation
Yifan Ding, Xincheng Wei, Yoshua Y. Li, Ziheng Li, Yuquan Lu, Siyu Zhang, Dongsheng Ma, Rongxiang Weng, Xunliang Cai, Yun Chen
cs.LG, cs.AI
2026-07-31
Fusing GRPO with on-policy distillation under a fixed weight collapses policy entropy. SAF's four-stage advantage controller beats fixed fusion by 0.51–2.70% across six settings.
RLVR (reinforcement learning with verifiable rewards), as in GRPO, turns a single "is this answer right" judgment into one reward broadcast to every token. The signal is sparse but can push past a teacher's ceiling. On-policy distillation (OPD) does the opposite, having a stronger teacher score each token: dense, but capped at teacher quality and hostile to exploration. The two are complementary, so adding their advantages together is natural. Adding them with a fixed coefficient triggers entropy collapse: the model goes overconfident early and stops learning. This paper pinpoints two concrete miscalibrations behind that collapse.
SAF (Stable Advantage Fusion) touches only the OPD branch; the RLVR advantage is added unchanged. Before OPD's advantage is summed in, it passes through four stages:
These two address the magnitude mismatch. The authors measure OPD advantages spiking to 20.36 while the largest co-occurring GRPO advantage is only 2.47, and all 180 of the largest OPD tokens they inspected exceed their sequence's GRPO magnitude. Under fixed fusion those spikes drown out the GRPO signal; compressing to ±0.1 puts the two on the same scale.
These two address the temporal mismatch. Full-strength OPD throughout training keeps dragging the student toward the teacher, erasing the exploration RLVR is meant to provide, and annealing lets OPD help early and step aside late. Each stage is independently switchable and adds negligible overhead.
Seven benchmarks: math (AIME-24, AIME-25, HMMT25 February and November) and code (HumanEval+, MBPP+, LiveCodeBench). Students are Qwen3-1.7B/4B/8B; teacher is Qwen3-30B-A3B-Instruct-2507. Baselines are the base model, GRPO-only, OPD-only, and fixed-coefficient GRPO+OPD.
| Setting | Gain over fixed fusion |
| Qwen3-8B | math +0.97%, code +1.67% |
| Qwen3-4B | math +1.51%, code +2.70% |
| Qwen3-1.7B | math +1.85%, code +0.51% |
All six model-domain settings are positive, 0.51–2.70%, with the six-setting average aggregate reaching 49.46%. On the training curves, fixed fusion drives policy entropy from about 0.35 down to 0.30 and parks it there, while SAF holds it at 0.35–0.38. The configuration with the lowest KL, the one glued tightest to the teacher, is also the least accurate, evidence that over-imitation hurts. On Qwen3-4B, SAF is best on six of seven benchmarks. The authors are upfront about the exception: on MBPP+, SAF trails GRPO-only and OPD-only by no more than 0.53 points yet still beats fixed fusion on average.
Mixing OPD with RLVR is common when tuning small models, and most people pick a fixed weight and watch training go unstable or the model go "dead." The contribution recasts "why is it unstable" into two actionable mechanisms, magnitude and timing, and hands over a fix that is switchable per stage and nearly free. The gains are single-digit points, an incremental improvement rather than a paradigm shift, but as a stabilizer that drops onto an existing GRPO+distillation pipeline it is high-value and easy to try.
The authors explicitly do not claim their results hold beyond 1.7B–8B, with a different teacher, or under different hyperparameters. The gain pattern couples scale with domain (code gains rise with scale, math does not), so there is no clean "bigger is better." Only GRPO was tested as the RLVR instance; PPO and RLOO are untouched. The smallest gain on code is at 1.7B, just +0.51%, so whether the controller earns its keep on small models is open. The update-geometry analysis (stable rank, subspace overlap) is used only to observe behavior, not translated into design guidance, and is left to future work.