EOS Token Mismatch Causes Length Inflation in Online Distillation

A paper by UNC, BYU, and Microsoft shows that EOS token mismatch between teacher and student models causes output length inflation in on-policy distillation—one Qwen3 rollout added 7,098 redundant tokens after answering correctly—and proposes fixes.

2026-09-18 ~ 2026-09-20 · 3 related posts

• EOS mismatch drives length inflation in on-policy distillation, HF study shows fix — Yuxiao Yang · 2026-09-18
• New research traces distillation length inflation to student-teacher EOS token mismatch — tw_killian · 2026-09-20
• EOS token mismatch inflates outputs in on-policy distillation: Qwen3 rollout added 7,098 redundant tokens after solving in 1,094 — tw_killian · 2026-09-20