When Teachers Mislead: Spurious-Signal-Aware On-Policy Distillation
Yinuo Jiang, Yongjie Ye, Zhou Tao, Xiang Zhuang, Qiang Zhang, Huajun Chen, Tiankai Li
cs.AI
2026-08-04
Distillation teachers can follow template priors over the input. SA-OPD adds a no-prompt forward pass, drops high-gradient tokens whose signal barely changes, and lifts VLM scores 3.5 on average.
On-policy distillation (OPD) has the student roll out its own trajectories while the teacher gives dense token-level supervision, which is more efficient than sparse sequence-level rewards. Recent selective OPD goes further, prioritizing tokens that are confident, informative, or learnable.
These methods share a blind spot: a teacher's judgment at a token may not be driven by the current input at all, but propped up by language priors, formatting conventions, or stereotyped reasoning templates. The authors call such optimization-relevant but weakly input-grounded supervision a spurious signal. It produces large gradients with almost no task direction, a pure nuisance gradient that also drifts the parameters.
The core of SA-OPD is an input-groundedness proxy: a way to tell whether a token's distillation signal actually depends on the input.
The ideal measure is the conditional mutual information between input and signal, but it has no closed form. The proxy is direct: for each token, recompute the teacher-student divergence once more with the prompt removed, keeping the student's generated prefix. If the signal truly depends on the input, removing the prompt should change it noticeably; if it barely changes, the judgment is template-driven and would say the same for any input. The gap between the two passes is the groundedness; a small gap means a spurious signal.
Groundedness alone is not enough. The authors define a spurious signal as two conditions together: low groundedness (signal does not depend on input) and large divergence magnitude (high optimization impact). Filtering on divergence alone would kill useful, grounded high-divergence tokens; filtering on groundedness alone would over-remove low-impact noise. Intersecting the two axes hits only the most harmful tokens, the large-gradient ones that are not grounded.
A dynamic bound on removed gradient mass keeps the filter from deleting too much at once.
The main runs use Qwen3-4B-Instruct to Qwen3-1.7B (LLM) and Qwen3.5-35B-A3B to Qwen3.5-2B (VLM).
On VLM, visual-understanding average rises from Vanilla OPD's 50.5 to SA-OPD's 54.0 (+3.5), and visual reasoning from 60.4 to 63.5 (+3.1), best on all six subtasks. The largest single-task gain is CountQA, 26.4 to 33.6 (+7.2).
| Setting | Vis. understanding | Vis. reasoning |
| Vanilla OPD | 50.5 | 60.4 |
| TIP (best rival) | 52.3 | 61.6 |
| SA-OPD | 54.0 | 63.5 |
LLM math gains are more modest, average 28.5 to 30.4 (+1.9), with Math500 rising 66.4 to 69.6. The authors' read: on VLM tasks spurious signals are persistent structural contamination, so filtering pays off; on LLM math they behave more like transient noise, so the gain is smaller.
Ablations confirm both axes are needed. Divergence-only scores 69.8, groundedness-only 69.0, SA-OPD 72.2 (Geo3K). Overhead is modest: 2.64% to 7.53% more wall-clock per step than Vanilla OPD, with no extra trainable parameters, just one more teacher and student forward over the prefix.
For anyone running distillation, the takeaway is actionable: a teacher's confidence is not the same as being grounded in the input. Adding a no-prompt forward pass and dropping tokens whose signal survives it is a light, plug-in change, and it pays off most where supervision is dirtiest, namely VLM. It also says signal selection in distillation should watch one more dimension beyond whether a token is learnable: is this judgment actually about the current input.
The paper has no dedicated Limitations section; the bounds are scattered. The proxy is an approximation: it measures whether the signal is stable without the prompt, not true mutual information. A fixed selection ratio can be brittle across tasks; the authors mitigate with a dynamic bound, but its beta is tuned seriously only on the counting task. Gains are sensitive to beta, with the CountQA score swinging over a roughly 3-point band. LLM is tested only on math reasoning, with no code, open-ended generation, agent, or multilingual evaluation. About 60% of filtered tokens are content words, so useful signal is dropped on every step by design. No test of non-Qwen families, of teacher and student from different families, or of RLHF settings.