Perturbing Inputs Boosts Pretraining Distillation, New Paper Shows

A new paper shows that adding input perturbations during pretraining distillation yields significant gains, with student models matching teacher outputs in the early training phase. The simple trick also works in data-limited settings.

2026-11-16 ~ 2026-11-17 · 3 related posts

1 near-duplicate retellings: DimitrisPapail