Ant Group and MBZUAI Paper: Supervising Just 1% of Tokens Matches Full On-Policy Distillation
jiank_uiuc · x · 2026-09-22
A new paper from Ant Group and MBZUAI researchers shows that sparse on-policy distillation can match full teacher supervision with only 0.1%–1% of tokens supervised.
Problem and method
- In sparse OPD, teacher guidance allocated to a few tokens of student-generated trajectories can yield noisy updates, since each gradient is estimated from a single sampled next token.
- The authors analyze this at a fixed prefix in information geometry and propose an information-efficiency ratio (IER) based on a signal-to-noise decomposition, characterizing relative gradient-estimation error under an optimal scalar baseline.
- A candidate-set approximation enables token selection by IER, optionally combined with existing usefulness scores, while keeping the sampled reverse-KL objective.
Results
- On math and medical reasoning tasks, adding IER improves existing selectors across settings.
- Sparse configurations at 0.1%–1% token budgets match or exceed full OPD without token selection.
Code: github.com/BruceSheng1202/IER-OPD
More from Research
- PIR lie-detector reads LLM internal states to expose hidden knowledge at 0.70-0.87 accuracy — Hiskias Dingeto · 2026-09-22
- TAPe+ML: sub-100K-parameter vision system hits 84.7 mAP50 on COCO detection — Comexp · 2026-09-22
- RoboDawn: Tsinghua & Tencent Hunyuan drive robots with frozen VLM, 73.6% one-shot success — _akhaliq · 2026-09-22
- PragmaBot: robots learn online from real-world failures without retraining, RA-L/IROS 2026 — ChongZzZhang · 2026-09-22
- Iterative sensitivity probing: how bitsandbytes2 finds each layer's compression limit — Tim_Dettmers · 2026-09-22
- Dynamic REAP swaps 4 experts every 256 tokens, keeping only 25% in GPU memory — Tim_Dettmers · 2026-09-22