Sparse On-Policy Distillation: Supervising Just 1% of Tokens Can Match Full OPD

A late-September arXiv paper from MBZUAI, Ant Group and other institutions, "1% of Tokens Can Be Enough: On Gradient Estimation in On-Policy Distillation" (arXiv:2609.24432), examines gradient estimation in sparse on-policy distillation (OPD). One of the authors, jiankuiuc, introduced the work in a series of posts: sparse OPD applies teacher supervision to only a small fraction of tokens in the student's generated trajectories, and experiments show that supervising just 1% of tokens matches full OPD — in some cases 0.1% suffices and can even surpass full supervision.

Confirmed

• The paper was jointly released by MBZUAI, Ant Group and other partners, with UIUC-affiliated authors (including jiankuiuc) involved and promoting it in successive social media posts themselves.
• Core finding: teacher guidance is useful, but gradient estimation based on single-token sampling is highly noisy, which is one of the bottlenecks for distillation efficiency.
• From an information-geometry perspective, the authors break down the noise source: the expected gradient of distillation requires averaging over possible next tokens, whereas in practice it is typically estimated with just one sampled token, leading to high-variance updates.
• The paper proposes metrics such as the Information Efficiency Ratio (IER) to characterize the relationship between the number of supervised tokens and performance.

Why it matters

• If distillation quality can be preserved by supervising only 0.1%–1% of tokens, teacher forward/scoring overhead during training can be sharply reduced, substantially cutting distillation costs.
• The result offers a gradient-estimation-level theoretical explanation for why "more supervised tokens aren't necessarily better," with direct implications for designing on-policy distillation pipelines for large-scale models.

2026-09-22 ~ 2026-09-23 · 7 related posts

• Ant Group and MBZUAI Paper: Supervising Just 1% of Tokens Matches Full On-Policy Distillation — jiank_uiuc · 2026-09-22
• [source] Sparse distillation paper: supervising just 0.1%-1% of tokens can match or beat full OPD — jiank_uiuc · 2026-09-23
• [source] Why distillation gradients are noisy: single-token sampling breaks the expected-teacher estimate — jiank_uiuc · 2026-09-23
• [source] 1% of Tokens Can Be Enough: MBZUAI and Ant Group Improve On-Policy Distillation — jiank_uiuc · 2026-09-23

3 near-duplicate retellings: jiank_uiuc · jiank_uiuc · jiank_uiuc