New Paper: Adding Input Noise During Pretraining Distillation Beats Standard Distillation
DimitrisPapail · x · 2026-11-16
A new paper, Perturbing Inputs Improves Pretraining Distillation by Jack Cai with Yihe Dong, Dimitris Papail, and Shivam Garg, tweaks one thing in pretraining distillation: for the first part of training, both the small student LM and the larger teacher read a perturbed version of the text (e.g., noise added to token inputs), with the student matching the teacher's predictions on that input, before switching back to clean text.
- With the same step and token budget, this beats standard distillation
- Also works well in data-constrained, multi-epoch settings
- Co-author Papail quipped that "a bit of noise during distillation magically makes everything better"
More from Research
- AI/ML for Fluids cohort 2 opens: 60+ live hours over 3 months for $500 — ricardovinuesa · 2026-11-16
- 1,000 agents write 5,000 rule-based PyBot versions in pure Python, beating a 360M LLM — LoubnaBenAllal1 · 2026-11-16
- "The conference-paper game is 100% dead": researcher urges GitHub-first research output — DimitrisPapail · 2026-11-16
- DeepMind researcher on doing research in the AI era: minimize friction to excellent work — DimitrisPapail · 2026-11-16
- Google AI Co-Scientist progress presented by Michael Brenner at MLCB 2026 — anshulkundaje · 2026-11-16
- Auto Weight-Space Programming: Qwen3.5-4B learns to apply logit lens to its own neurons at 99.8% — udmrzn · 2026-11-16