New Paper: Supervising Just 1% of Tokens Can Match Full On-Policy Distillation

MBZUAI, Ant Group and collaborating institutions released an arXiv paper (arXiv:2609.24432), "1% of Tokens Can Be Enough: On Gradient Estimation in On-Policy Distillation," studying gradient estimation in sparse on-policy distillation (OPD). Co-author jiankuiuc introduced the work in a series of posts: sparse OPD applies teacher supervision to only a small fraction of tokens in student-generated trajectories, and experiments show that supervising just 1% of tokens matches full OPD, with 0.1% sometimes sufficient and occasionally even surpassing full supervision.

Confirmed

The paper is a collaboration involving MBZUAI and Ant Group, with UIUC-affiliated authors (including jiankuiuc) promoting it directly on social media.
Core finding: teacher guidance is useful, but gradient estimation based on single-token sampling is very noisy, a key bottleneck for distillation efficiency.
The authors decompose the noise source via an information-geometric view: the expected gradient requires averaging over possible next tokens, while practice uses a single sampled token, causing high-variance updates.
The paper proposes metrics such as the Information Efficiency Ratio (IER) to characterize the relationship between the number of supervised tokens and performance.

Why it matters

If 0.1%–1% of tokens suffice, teacher forward/scoring overhead during training drops dramatically, cutting distillation cost.
The results give a gradient-estimation-level theoretical account of why more supervised tokens are not always better, directly informing OPD pipeline design for large models.

2026-09-22 ~ 2026-09-23 · 7 related posts

Primary sources

Sparse distillation paper: supervising just 0.1%-1% of tokens can match or beat full OPD — jiank_uiuc · 2026-09-23 00:25 (UTC+08:00)
UIUC paper: distilling with just 0.1% of tokens can beat full on-policy distillation — jiank_uiuc · 2026-09-23 00:56 (UTC+08:00)
1% of Tokens Can Be Enough: MBZUAI and Ant Group Improve On-Policy Distillation — jiank_uiuc · 2026-09-23 00:56 (UTC+08:00)

Ant Group and MBZUAI Paper: Supervising Just 1% of Tokens Matches Full On-Policy Distillation — jiank_uiuc · 2026-09-22
[source] Sparse distillation paper: supervising just 0.1%-1% of tokens can match or beat full OPD — jiank_uiuc · 2026-09-23
Why distillation gradients are noisy: single-token sampling breaks the expected-teacher estimate — jiank_uiuc · 2026-09-23
[source] 1% of Tokens Can Be Enough: MBZUAI and Ant Group Improve On-Policy Distillation — jiank_uiuc · 2026-09-23

3 near-duplicate retellings: jiank_uiuc · jiank_uiuc · jiank_uiuc