SPOT: Sparse Probing and Outcome Calibration for On-Policy Distillation
Zikun Qu, Min Zhang, Mingze Kong, Zhiwei Shang, Yikun Ban, Shuang Qiu, Zhongxiang Dai
cs.LG, cs.AI
2026-08-05
SPOT probes only positions where the teacher is uncertain, its mass is concentrated, and the student is mismatched, then weights candidates by real verifier outcomes. It posts the best Pass@8 across 0.6B-4B students, beating EOPD by about 3 points.
Distilling a large model's reasoning into a smaller one has three main routes. Supervised fine-tuning and off-policy distillation train on teacher-generated trajectories, but the contexts the student sees differ from those its own predictions induce, causing exposure bias. On-policy reinforcement learning optimizes the student's own rollouts, but reasoning RL usually gives only sequence-level or terminal verifier rewards, which is coarse. On-policy distillation (OPD) takes the best of both: train on student-generated prefixes while keeping dense token-level teacher feedback.
OPD usually minimizes reverse KL, which is mode-seeking: it piles probability on the teacher's single favored continuation and under-covers other equally plausible ones. Prior work EOPD tries to fix this with a teacher-entropy trigger, adding a top-k forward-KL term at high-entropy positions to preserve coverage. But high entropy alone says nothing about three things: whether the teacher's uncertainty is concentrated in a few tokens or spread over a long tail; whether the student already represents those candidates well; and whether the teacher's local next-token probability predicts downstream success at all. SPOT closes those three gaps.
SPOT splits the problem into two coupled decisions, where to probe and what to distill, and runs an acquisition-exploration-exploitation loop. Acquisition scores each position by multiplying three terms: normalized teacher entropy (the teacher is genuinely uncertain here), the mass the teacher's top-ks candidates capture (the uncertainty is concentrated, not in a long tail), and the student-teacher gap (the student does not yet cover or rank those candidates). Any low factor drags the position's priority down, so the limited probing budget M is spent only where it pays.
Exploration: at each chosen position, each of the teacher's top-kp candidate tokens is appended to the prefix, the student continues, a verifier scores it, and a candidate-conditioned "student-executable value" is estimated. Only positions with at least one positive-reward candidate are kept. Exploitation: those verified values produce a closed-form target within a KL trust region around the teacher distribution, tilting probability toward candidates with better downstream outcomes while staying anchored to the teacher. The training loss is standard OPD plus a branch loss at those positions. When no candidate gets a positive reward, the branch term vanishes and the method reverts to standard OPD.
The key move is treating the teacher's local distribution as a proposal prior rather than as proof of downstream success, then calibrating it with the student's actual continuation outcomes.
The teacher is Qwen3-8B with thinking disabled; students are Qwen3-0.6B/1.7B (trained on MATH) and Qwen3-4B (on DAPO-Math-14k), evaluated on six math reasoning benchmarks. SPOT posts the best macro Pass@8 at all three student scales and the best or second-best Avg@8.
| Student | Method | Avg@8 | Pass@8 |
| Qwen3-4B | OPD | 35.66 | 49.57 |
| Qwen3-4B | EOPD | 35.45 | 51.81 |
| Qwen3-4B | SPOT | 36.13 | 54.30 |
Against standard OPD, SPOT improves Avg@8 by 0.47 to 1.48 points and Pass@8 by 4.55 to 5.28; against EOPD, 0.29 to 0.68 on Avg@8 and 2.49 to 3.19 on Pass@8. Pass@8 climbs much more than Avg@8, meaning SPOT widens coverage of multiple solution paths rather than diluting single-attempt accuracy. Ablations show all three acquisition factors are necessary, and removing verifier calibration costs 7.38 Pass@8 points.
For anyone doing distillation, this is a clean and restrained improvement. Rather than adding forward KL at every high-entropy position, first judge whether the teacher's uncertainty is concentrated in a few candidates and whether the student already handles them, spend the expensive probing only where it pays, and set the target from real outcomes rather than teacher probabilities. It is especially useful for majority-voting inference, where Pass@8 is the metric that actually matters.
Experiments cover only math reasoning (all six benchmarks are math); code or other reasoning tasks are untested, so generalization is uncertain. Students range only from 0.6B to 4B and the teacher is a single Qwen3-8B, so whether the gain holds for larger students or stronger teachers is unclear. The Avg@8 gain is modest (under 1.5 points); the story rests mainly on coverage (Pass@8). Probing adds M x kp x Np rollouts plus verifier calls; the budget is controlled but the paper gives no compute-cost breakdown against baselines.