When EOS Tokens Disagree: Understanding Length Inflation in On-Policy Distillation
Yuxiao Yang, Tianrun Yu, Shangzhe Li, Kaixiang Zhao, Xuchao Zhang, Chetan Bansal, Huaxiu Yao, Taylor W. Killian, Weitong Zhang
cs.LG
2026-09-17
Student and teacher can stop on different EOS tokens. Sampled-token OPD drives native stop from ~0.8 to near 0; summing those EOS tokens as one stop action contains the inflation.
On-policy distillation (OPD) trains a student on its own sampled rollouts, with a token-level signal from a frozen teacher. Several recent papers report that student answers grow longer through training until they hit the generation budget, often with repetition after a correct answer is already on the page. Typical explanations point to reverse KL, entropy collapse, or teacher supervision that gets worse as the trajectory gets longer.
This note from UNC Chapel Hill, Brigham Young University, and Microsoft isolates an earlier, cheaper cause in a common pairing: a base student distilled from a post-trained teacher. The two models can put "stop here" on different EOS surfaces. Qwen3-Base declares <|endoftext|>; the instruct checkpoint also treats <|imend|> as a conversational stop. Llama 3.2 adds chat-end tokens such as <|eotid|>. Gemma 3 PT and IT already declare the same set, <eos> and <endofturn>, yet still prefer different members of that set. The student almost never samples the teacher's favorite token, and the teacher assigns low probability to the student's native one. Sampled-token OPD then repeatedly punishes the stop action the student actually uses.
The main pair distills Qwen3-4B (non-thinking) into Qwen3-1.7B-Base on DAPO-Math-17K, with the TTRL template that asks for a \boxed{} answer. Training caps prompts at 1,024 tokens and responses at 7,168; checkpoint eval allows 8,192 generated tokens. The objective is sampled-token OPD: the advantage on a generated token is the teacher-minus-student log-probability gap. Runs last 200 steps, 64 trajectories per step, AdamW at \(1\times10^{-6}\). Downstream metric is Avg@16 over AIME24, AIME25, and AMC23.
Four patches:
Cross-family runs default to Fix 3, which does not require naming a canonical surface token. K2-Horizon-7B checkpoints from pretraining, midtraining, SFT, and final post-training show how the preference itself moves through training.
On Qwen3, vanilla OPD drives the student's native <|endoftext|> probability from about 0.8 early in training to near zero. The teacher-preferred <|imend|> sits around \(10^{-11}\) on the student, so it is almost never sampled. Qualitative rollouts already contain the right answer, then spend 86.6%, 90.7%, and 94.4% of remaining tokens on padding. Fix 1 tracks vanilla length and clipping. Fixes 2, 3, and 4 pull both curves back toward the teacher reference and stop the native-EOS collapse.
Gemma 3 is the cleaner control: the declared stop set matches, the PT student still prefers <eos>, and the IT teacher prefers <endofturn>. After semantic aggregation the student stops reliably without switching surface tokens. Llama 3.2 also loses the runaway growth, with the largest leftover gap versus its teacher; Qwen3 recovers fastest.
On K2-Horizon, midtraining already puts substantial mass on the teacher's token, and SFT is closer still. Vanilla OPD from those two stages does not show the base-pair collapse, and Fix 3 is largely inert. Pretrain-to-Final is different. The pretrained student can already sample the teacher EOS, so vanilla OPD can transfer the surface form, then late in the 400-step run both stop probabilities fall again and length rehits the budget. Fix 3 slows that second wave. It does not remove it.
Prompt templates change observed length and scores. The stock DAPO grader is brittle to Qwen3 LaTeX layouts: the Qwen3-4B teacher scores 6.28% Avg@16 under the original parser and 24.34% after a format-compatible parse. Length inflation is therefore not the same thing as a matching drop in reasoning. The paper does not tabulate Avg@16 for each fix; the primary evidence is length, clipping, and stop-probability curves.
Anyone running OPD on Qwen, Llama, or Gemma base-to-instruct pairs should check where stop probability actually sits before rewriting the objective. Editing the eos list in generationconfig does almost nothing. Semantic aggregation is a small change, works across the three families, and the implementation is public.
This is an implementation bug with a training-objective signature, not a new distillation theorem. Prior accounts that blame reverse KL or rollout degradation still apply when teacher and student already agree on termination. The practical order of operations is: align stop semantics first, then diagnose algorithm-level pathologies.
The authors mark three bounds. Termination mismatch does not explain all length dynamics; the late K2-Horizon re-inflation survives semantic alignment. Experiments are single-turn math. In multi-turn or agent settings, end-of-turn, tool handoff, and document end are different events and should not be collapsed into one stop. Template effects are measured only on Qwen3, with three templates and two graders.
Most headline curves are plots, not tables, so there is no reported "clipping fell to X%" or "Avg@16 rose by Y" for the fixes. The leftover Llama gap is not causally decomposed. Full-vocabulary reverse KL was skipped on cost; the argument is that even a full softmax still weights updates by student probability, so a \(10^{-11}\) token remains hard to recover.