DASH: Divergence-Adaptive Supervision Horizons for On-Policy Self-Distillation of Reasoning Models
ZhiYan Hou, Xinyu Tang, Hongyan An, Jianjin Zhang, Weizhen Wang, Yunyun Han, Gengsheng Li, Xiangzhao Hao, Haiyun Guo, Wenbin Hu, Jinqiao Wang, Yafeng Deng
cs.AI
2026-08-07
DASH turns each local signal's gap from the sequence mean into a propagation gate for backward aggregation, beating matched OPSD on all nine benchmark-scale cells at under 1% overhead.
Reinforcement learning with verifiable rewards (RLVR) trains reasoning models on signals like answer correctness, but those signals are sparse and sequence-level, so the contribution of each token to the final outcome stays unclear. On-policy self-distillation (OPSD) softens this: the student generates its own rollout, then the same model conditioned on a reference answer acts as a privileged teacher and gives dense token-level distributional supervision at each student-visited prefix.
Vanilla OPSD, however, assigns every local divergence (the teacher-student KL at a position) the same coefficient 1/T, regardless of where it sits or what mismatch history precedes it. In autoregressive generation the same divergence magnitude can follow very different discrepancy trajectories, and the paper shows the local value barely predicts the mean future divergence. Two forks with matched local divergence diverge sharply afterward.
DASH turns the gap between each local signal and the sequence mean into a propagation gate, then uses the gates to control a backward multi-step aggregation.
The gates and the mean are detached, so gradients flow only through local signals. DASH reuses the teacher and student distributions OPSD already computes, adds no forward pass, and the extra scalar backward scan costs under 1% of step time.
Three math benchmarks (AIME 2024, AIME 2025, HMMT Feb 2025) and three Qwen3 scales, Avg@12.
| Model | OPSD | DASH | Gain |
| Qwen3-1.7B | 41.87 | 45.07 | +3.20 |
| Qwen3-4B | 63.60 | 65.00 | +1.40 |
| Qwen3-8B | 64.80 | 66.40 | +1.60 |
DASH is highest in all nine benchmark-scale cells and beats its matched OPSD rerun on every benchmark at every scale, ahead of the next-best PW-OPSD by 1.64 (1.7B) and 1.17 (8B).
The ablations isolate the source. A fixed lambda (best 0.1) lifts the mean from 41.87 to 43.63, and DASH to 45.07, so part of the gain is just multi-step aggregation and part is the adaptation. Reversing the gate sign (Inverse-gap) lands at 42.10, essentially OPSD, so direction matters: small divergences open, large ones close. A 2x2 factorial separating the relative coefficient profile from its average scale shows dynamic allocation adds 2.40/2.50 points while uniform scaling adds only 0.70/0.80, so the win is the relative allocation, not stronger supervision. Forward KL (45.07) beats symmetric JSD (38.23) and reverse KL (41.47).
For anyone post-training reasoning models, this is a near-free OPSD tweak: no extra forward pass, no new parameters, under 1% overhead, turning uniform token weights into context-aware ones. The gains are modest (1.4 to 3.2 points) but consistent. The more lasting contribution may be the observation itself: the standard OPSD objective is invariant to different temporal arrangements of the same divergence values, a structural blind spot that probably has more to exploit downstream.
There is no significance test against the single-seed baselines (EOPD, AVSD, PW-OPSD are single runs; Base, SFT, GRPO are external). The authors are explicit that "highest score" means highest among the displayed results, not statistically superior. The gains are small and validated only on math reasoning; code and other long-horizon tasks are untested. Checkpoint selection is best-within-200-step on the evaluation benchmarks, not held-out validation, which the paper admits. The fixed-horizon gradient decomposition is offered only as structural motivation; DASH does not recover the trajectory score-function term it identifies, and the authors state plainly that they perform no future-to-past credit assignment.