Mitigating the Length-Scaling Tax with Online Distillation
Xu Wan, Wenyue Xu, Shengjie Zhao, Mingyang Sun
cs.LG, cs.AI
2026-09-30
Cuts RL's length-scaling tax on solved queries from 19.0% to -3.7% (math) and 31.4% to 13.7% (agentic) by routing them to EMA self-distillation, with accuracy held or improved.
Since RLVR (RL with verifiable rewards) became the default post-training recipe, growing response length has usually been read as a sign of improving reasoning. On queries the model already solves, that growth buys nothing. The paper quantifies the cost as the length-scaling tax (LST): freeze a set of queries that some checkpoint solves consistently, then measure how much longer later checkpoints answer those same queries, relative to the shortest RL checkpoint that still meets the accuracy bar.
The mechanism sits in the objective. Under GRPO, a rollout group where every response is correct gets near-zero advantages, so easy prompts stop producing any gradient that preserves concise solutions. But all prompts share one parameter set, and updates from hard prompts still shift token probabilities on easy prefixes, so length only drifts upward. Common training choices compound it: dynamic sampling discards all-correct groups, and difficulty-aware curricula concentrate training on hard prompts.
The measurements confirm the tax exists. Qwen3-4B-Base trained with RLVR on DAPO-Math-17K: no matter which checkpoint defines the easy set, accuracy on it stays flat while length climbs. Hard-prompts-only training produces the highest tax; an 8k rollout budget from the start amplifies it early, while switching to 8k later actually helps. Across 5,824 responses, the added tokens are mostly reflection words and repeated bigrams, not useful reasoning.
LSD (Length Self-Distillation) adds two pieces to a standard RL trainer.
Three distillation objectives: SG-FKL minimizes forward KL over the teacher's top-32 tokens plus stop tokens, with gradients through the student logits; SG-RKL minimizes reverse KL on the same support; PG-RKL treats the teacher-minus-rollout-policy log-probability gap as a detached advantage in a PPO-style clip. Route losses are weighted by their sequence counts. The design logic: what easy groups lack is exactly a token-level preservation signal, and OPD supplies dense supervision on the student's own prefixes; hard groups still need exploration, so they stay under RLVR.
Single-turn math (Qwen3-4B-Base; AMC23 + AIME25/26, 143 prompts, 32 samples each):
| Method | Avg Pass@1 | Easy len | Hard len | LST1000 |
| RL | 42.90 | 997 | 2329 | +19.0% |
| CRISP | 40.56 | 972 | 2364 | +10.6% |
| Fixed SG-FKL (π0 teacher) | 30.44 | 691 | 1323 | -10.8% |
| LSD (SG-FKL) | 44.20 | 859 | 2511 | -3.7% |
| LSD (SG-RKL) | 42.80 | 757 | 2395 | -10.9% |
| LSD (PG-RKL) | 43.56 | 802 | 2393 | +1.4% |
SG-FKL is the best overall: +1.3 pp average Pass@1 over RL, about 14% shorter easy-query responses, and the tax turns negative. On paired evaluation over identical frozen query sets, all three variants beat RL on both length and accuracy: at anchor b=1000 (23 queries), RL sits at 98.64% accuracy and 1,016 mean tokens, SG-FKL at 99.73% and 822. Hard queries are not dragged along: mean hard-query length rises from 2,329 under RL to 2,511 under SG-FKL, and after step 500 only 10.6-12.2% of training tokens flow through the easy route. A random-routing control with the same teacher and objective drops Pass@1 to 22.08%, so the routing itself carries the gain.
Multi-turn agentic tasks (Qwen3-8B-Base, CutTheBill training, BrowseComp-Plus, 830 test queries):
| Method | Pass@1 | Avg turns | LST50 |
| RL | 29.50 | 5.41 | +31.4% |
| RL + Length Penalty | 22.58 | 4.66 | +1.2% |
| LSD (SG-FKL) | 30.72 | 5.24 | +13.7% |
| LSD (SG-RKL) | 29.60 | 4.93 | +9.2% |
| LSD (PG-RKL) | 31.65 | 5.66 | +16.1% |
RL + Length Penalty (stronger penalties on easier queries) pushes the tax down to 1.2% but costs 6.9 pp of Pass@1; LSD cuts the tax by about half while moving Pass@1 above RL. Ablations: EMA half-life H = 4 is best (41.85% single-turn Pass@1); loosening the routing threshold from 1.0 to 0.85/0.75 drops Pass@1 to 40.91%/39.31%, which supports preserving only fully solved groups and leaving partially solved ones to RL.
Response length is inference cost. A post-trained model that systematically lengthens on queries it already solves pays extra tokens on every easy request, and LST gives a difficulty-sliced measurement that average-length statistics miss.
The method needs no external teacher and no extra rollouts: the routing signal comes free from rollouts the trainer already generates, so it bolts onto an existing GRPO setup. Compared with length-penalty reward shaping it avoids the accuracy hit, and compared with always-on distillation like CRISP it leaves exploration on hard prompts intact. This is an incremental improvement to the RLVR pipeline rather than a new paradigm, but a cheap and low-risk one.
Stated by the authors: validation stops at Qwen3-4B and 8B, larger models are future work, and code is released only after acceptance.
Open questions from the read: the paired-evaluation easy sets are small (10, 22, and 23 queries at the three anchors), so single-query noise can move numbers, and the paper's own example aime256 falls from 31/32 correct under RL to 28/32 under SG-RKL. The LST reference, the shortest accuracy-qualified RL checkpoint, is a somewhat arbitrary choice, and the authors acknowledge that at τ < 1 the metric compares length under an accuracy threshold, not waste at identical accuracy. In the threshold ablation, distillation token share and the effective coefficient change together, and the confound is not isolated. On multi-turn tasks, PG-RKL shows the smallest tax reduction and more turns than RL, so a lower LST does not directly mean cheaper interactions. The agentic setup is tied to CutTheBill and a separate refinement agent; transfer to other agent stacks is untested.