UIUC paper: distilling with just 0.1% of tokens can beat full on-policy distillation
jiank_uiuc · x · 2026-09-23
Researchers present 1% of Tokens Can Be Enough: On Gradient Estimation in On-Policy Distillation, introducing the Information-Efficiency Ratio (IER) to measure gradient-estimation reliability in on-policy distillation.
Key insight: the expected distillation gradient averages over all possible next tokens, but in practice it's estimated from a single sampled token — so updates stay noisy even when teacher guidance is useful. IER is derived from a signal-to-noise decomposition in information geometry (higher IER = lower relative gradient estimation error) and can be approximated with a small candidate set to rank tokens.
Results:
- Distilling Qwen3-1.7B from JustRL-Qwen3-4B (thinking off), 0.1% of tokens beats full OPD: AIME25 14.4→19.4, AIME26 12.5→16.3, HMMT25 8.7→11.7, HMMT26 13.4→14.4
- On HealthBench, roughly one token per trajectory matches full OPD
- Gains hold with thinking on or off, strongest at 0.1–1% budgets; more supervised tokens sometimes helps little or hurts
- IER combines with existing usefulness selectors via IER-OR/IER-AND, keeping the sampled reverse-KL objective unchanged.
More from Research
- minpts-check: one-run predictor tells if your HDBSCAN clusters survive min_samples changes — leland_mcinnes · 2026-11-12
- Architecture study sparks monitoring concerns over a multi-hop planning spike — ChowdhuryNeil · 2026-11-12
- SGLang Megakernel Pushes Muse Spark to 1,109 tok/s, 2.1x Over Baseline — hsu_byron · 2026-11-12
- ArcGrav: a cross-model Agent Control Language to cut token overhead — ArchitecturalGravity · 2026-11-12
- Open-source dual-arm robot solves tangram puzzles with 90 demos, ~60% success — DanGrover · 2026-11-12
- FutureHouse's deslopper trains Kosmos to write indistinguishably from human experts — proteinrosh · 2026-11-12