Perturbing inputs improves LM pretraining distillation, new paper shows
DimitrisPapail · x · 2026-11-17
New paper "Perturbing Inputs Improves Pretraining Distillation" (with Yihe Dong, Dimitris Papail, Shivam Garg) changes one thing in pretraining distillation: for the first part of training, both teacher and student read perturbed text (e.g. token-level noise) and the student matches the teacher's next-token predictions on it, then training switches back to clean text.
- Beats standard distillation at the same step/token budget, and works in data-constrained, multi-epoch settings
- Figure illustrates four design choices: fixed teacher, independent noise, consistent augmentation, and consistent augmentation plus mixup-style interpolation — essentially porting mixup from image distillation to LMs
- Original distillation author giffmana amplified it, noting they should have finished on pure examples back then too
Related event: Perturbing Inputs Boosts Pretraining Distillation, New Paper Shows(3 posts)→
More from Models
- Google's Rumored 'Neon' Model Tied to Upsell Code, Likely a Pro-Tier Offering — testingcatalog · 2026-11-17
- User deliberately maxes out Anthropic quota to make it lose money this week — ryunuck · 2026-11-17
- Inkling 2 Small hits 81.7% on ARC-AGI-2, beating DeepSeek V4.1 Flash but trailing on ARC-AGI-3 — teortaxesTex · 2026-11-17
- Inkling-2-Small completes 911 enterprise tasks/hour on one B300 node, 43% cheaper than DeepSeek — ryanshrout · 2026-11-17
- Every used Claude Fable 5.2 to mine hundreds of meetings into a product philosophy doc — every · 2026-11-17
- Quantized Opus 5.5 says 'open' instead of 'run' — quantization quirks again — ryunuck · 2026-11-17