Perturbed Inputs Improve Pretraining Distillation, New Paper Finds Gains Even in Multi-Epoch Settings
DimitrisPapail · x · 2026-11-17
A new paper, Perturbing Inputs Improves Pretraining Distillation, makes one change to standard distillation: for the first part of training, both a small student LM and its larger teacher read perturbed versions of the training text (e.g. noise added to token inputs), and the student matches the teacher's next-token predictions on that noisy text before switching back to clean data.
- With the same steps and token budget, this beats standard distillation.
- It also works well in data-constrained, multi-epoch settings.
- Intuition from commenters: autoregressive models make mistakes while generating, and noising teaches them to self-correct during pretraining, making their own generations less out-of-distribution (a similar issue RL addresses).
Related event: Perturbing Inputs Boosts Pretraining Distillation, New Paper Shows(3 posts)→
More from Research
- Probing activations restores diversity in distilled image and video generators — _akhaliq · 2026-11-17
- Perturbing inputs improves LM pretraining distillation, new paper shows — DimitrisPapail · 2026-11-17
- ReDiverse restores output diversity in few-step distilled diffusion models — anand_bhattad · 2026-11-17
- Inkling 2 Small hits 81.7% on ARC-AGI-2, beating DeepSeek V4.1 Flash but trailing on ARC-AGI-3 — teortaxesTex · 2026-11-17
- AI-formalized math spans Galois's 1832 last letter, fixing errors in a dozen published papers — stevenstrogatz · 2026-11-17
- FLUX 3 trains on images, video and sound: BFL on visual intelligence — Latent Space · 2026-11-17