New paper: supervising just 1% of tokens can match full on-policy distillation, 0.1% sometimes suffices
jiank_uiuc · x · 2026-09-23
A new MBZUAI–Ant Group paper, "1% of Tokens Can Be Enough: On Gradient Estimation in On-Policy Distillation," studies why useful teacher guidance in sparse on-policy distillation (OPD) can still yield noisy updates when gradients are estimated from a single sampled token.
Key points:
- Introduces the Information-Efficiency Ratio (IER), derived from a signal-to-noise decomposition, to measure gradient-estimation reliability at a given prefix
- IER can be combined with existing usefulness scores for token selection while keeping the sampled reverse-KL objective
- On math and medical reasoning tasks, 0.1%–1% token budgets match or beat full OPD without selection
- Gains hold in both thinking-on and thinking-off modes, strongest at small budgets
Paper and code are public.
More from Models
- Datalab fast-mode OCR beats Opus 5.5 and GPT-6.1 Sol at 1/4 to 1/68 the price — VikParuchuri · 2026-11-12
- Claude's world knowledge impresses: model guesses user identity from one prompt — adonis_singh · 2026-11-12
- Gemini 4 Carbon opens to all at $2/$10, price doubling to $4/$20 on Dec 1 — socialwithaayan · 2026-11-12
- Ten wild Gemini 4 Carbon demos in 24 hours: CUDA meadows, V8 engines, mech stomps — socialwithaayan · 2026-11-12
- How to cut Fable 5.2 token costs by 90%: Arc routing plus a lean CLAUDE.md — EXM7777 · 2026-11-12
- Grok Bot 1.0 launches with support for joining Meet, Zoom and Teams calls — mark_k · 2026-11-12