Short-to-Long Recovery for Pruned Models
ByteDance · hf · 2026-07-16
This paper investigates why structured-pruned LLMs suddenly collapse during real-world generation tasks.
The authors identify two key phenomena:
- Post-pruning, greedy pass@1 nearly vanishes, but pass@k under repeated sampling recovers significantly, indicating that many useful outputs aren't lost but "downgraded."
- Recovery failures often concentrate on suffix repetition, where the model continuously repeats redundant tails during long generations.
To address this, they propose ShortOPD, a short-to-long on-policy distillation schedule:
- Using a teacher model to identify repetitive suffixes
- Treating the remaining valid prefix as the "effective length" for rollouts
- Allocating the training budget to length intervals the current model can actually utilize
Experiments span math, code, and open-ended generation. Results show ShortOPD boosts the compressed model's scores to roughly 9x its unrecovered state and outperforms standard recovery methods by 1.6–4.4x. Concurrently, it achieves performance comparable to a fixed 8192 token rollout using only a quarter of the training time and 71% fewer rollout tokens.
More from Research
- Soft Clamp cuts tool-call overuse in multi-teacher distillation, from 13.7% to 9.0% — antgroup · 2026-07-21
- ShotPlan adds learnable planning tokens for cinematic multi-shot video generation — Tele-AI · 2026-07-21
- A silicon photonic reservoir chip compensates fiber distortion in real time at 28 Gbps — bravo_abad · 2026-07-21
- A developer maps out six design rules for CLIs that humans and AI agents can both use — yujiezha · 2026-07-21
- GPT 5.6 vs. Claude Fable tested in Dyad AI for Physical AI model tuning — ChrisRackauckas · 2026-07-21
- Sampling multiple solutions and voting may be a strong label-free path to better reasoning — iatitov · 2026-07-21