Perturbing Inputs Beats Standard Distillation in Princeton/Microsoft Pretraining Study
udmrzn · x · 2026-11-18
Princeton and Microsoft Research find that knowledge distillation improves when you deliberately corrupt the inputs. Early in pretraining, train the student to match the teacher on perturbed inputs (additive token noise, blended documents, or blended neighboring tokens), then spend the rest of the token budget on ordinary distillation.
- Best method cuts validation loss by 0.025 nats at 135M and 0.020 nats at 0.5B (three-seed mean); at 135M that's 1.45× the gain distillation itself gives over next-token training
- At 0.5B, average zero-shot accuracy improves 1 point across 7 tasks
- Gains survive low-bit quantization and data-limited settings
- Ablations: perturbing the teacher's inputs drives most of the gain; perturbing only the student's inputs helps far less
Related event: Perturbing Inputs Improves Pretraining Distillation, New Paper Finds(4 posts)→
More from Research
- Liquid AI previews on-device continual learning within a fixed memory budget — agihouse_org · 2026-11-19
- AI agent autonomously performs brain surgery on a mouse, choosing drug and target — HealthcareAIGuy · 2026-11-19
- Interaction Nets: Graph-Rewriting Computation That Could Eliminate Caches and Locks — mayfer · 2026-11-19
- NSF and Amazon award 11 Fairness in AI grant projects — tallinzen · 2026-11-19
- Prediction: 2027 could see 1000x more AI-driven scientific breakthroughs — Dr_Singularity · 2026-11-19
- MIT Study: GPT-6 Astra Edits 3.5% of Weights, Lifts Car Task From 0/256 to 1024/1024 — DeepLearningAI · 2026-11-19