Paper: Supervising Just 0.1%-1% of Tokens Can Match Full On-Policy Distillation
jiank_uiuc · x · 2026-09-23
The paper "1% of Tokens Can Be Enough: On Gradient Estimation in On-Policy Distillation" shows that useful teacher guidance still yields noisy updates. The authors introduce the Information-Efficiency Ratio (IER) to measure gradient-estimation reliability and combine it with usefulness scores, matching or beating full OPD with only 0.1%-1% of tokens in their experiments. Paper and code are available.
More from Research
- Architecture study sparks monitoring concerns over a multi-hop planning spike — ChowdhuryNeil · 2026-11-12
- SGLang Megakernel Pushes Muse Spark to 1,109 tok/s, 2.1x Over Baseline — hsu_byron · 2026-11-12
- ArcGrav: a cross-model Agent Control Language to cut token overhead — ArchitecturalGravity · 2026-11-12
- Open-source dual-arm robot solves tangram puzzles with 90 demos, ~60% success — DanGrover · 2026-11-12
- FutureHouse's deslopper trains Kosmos to write indistinguishably from human experts — proteinrosh · 2026-11-12
- Fleuret amplifies debate: AI can't slash wet-lab experiments in unmapped biology — francoisfleuret · 2026-11-12