Sparse distillation paper: supervising just 0.1%-1% of tokens can match or beat full OPD
jiank_uiuc · x · 2026-09-23
A new arXiv paper, 1% of Tokens Can Be Enough: On Gradient Estimation in On-Policy Distillation, tackles a hidden inefficiency in sparse on-policy distillation (OPD): even useful teacher guidance produces noisy updates when the gradient is estimated from a single sampled next token.
Method
- Working in information geometry, the authors derive an information-efficiency ratio (IER) from a signal-to-noise decomposition under an optimal scalar baseline — higher IER means lower relative gradient estimation error.
- IER can be approximated with a small candidate set and combined with existing usefulness selectors via IER-OR / IER-AND, keeping the sampled reverse-KL objective unchanged.
Results
- On math reasoning with a 0.1% token budget (JustRL-Qwen3-4B → Qwen3-1.7B, thinking off, Bayes@32), IER-based selection beats full OPD: AIME25 14.4→19.4, AIME26 12.5→16.3, HMMT25 8.7→11.7, HMMT26 13.4→14.4.
- On HealthBench, roughly one supervised token per trajectory matches full OPD.
- Gains hold in both thinking-on and thinking-off modes, and larger token budgets sometimes add little or hurt — which tokens you supervise matters more than how many.
Paper and code are available.
More from Models
- antirez backs low-bit DeepSeek 4.1 Flash over Qwen 27B dense on Apple M5 Macs — antirez · 2026-11-12
- Gemini re-renders images as text when vision is removed, dropping only 6% WSR — zacharynado · 2026-11-12
- Bare-harness coding benchmark: Fable 5.2 best overall, Gemini 4 Carbon best value — Semantic_meaning · 2026-11-12
- Google's Physical-Engineering's Last Exam: all 14 frontier models score below 50% — dl_weekly · 2026-11-12
- Reddit's 2026 LLM hall of fame: Opus 5.5 still unbeaten after 9 rival releases — G0dZylla · 2026-11-12
- Commit leak reveals unreleased Gemini 4 variants: Argon, Hydrogen and Helix (unconfirmed) — BlackHC · 2026-11-11