Perturbing Inputs Improves Pretraining Distillation
Ziyang Cai, Yihe Dong, Dimitris Papailiopoulos, Shivam Garg
cs.CL, cs.AI
2026-11-09
Blending neighboring tokens for the first 30% of distillation cuts loss another 0.025 nats at 135M and 0.020 at 0.5B, and adds about one point of 0.5B zero-shot accuracy.
Distillation for a small language model matches the student to the teacher on the training text and nowhere else. The teacher's next-token distribution off that text is never shown to the student. In a single pass, clean-text predictions are already dense supervision, so extra behavior off the data is not an obvious win.
Usual augmentation corrupts the student input and still targets the original next token. A teacher can label the corrupted input itself, so the target matches what the student sees. Task distillation with word replacement or adversarial masks, and same-view augmentation in vision, already train this way. The open question is pretraining from scratch, without repeating data.
Each perturbation replaces a one-hot token with a sparse weighted sum of a few vocabulary items. Teacher and student embed that same weight pattern with their own tables, so the perturbation is shared across hidden widths. The loss is forward KL at temperature 1 over the full vocabulary, with no next-token term. The objective never changes. Only the inputs in an early slice of training do.
Blended inputs sit far from natural text, so the perturbed slice stays short and the remaining budget is ordinary distillation on clean text. The learning rate is warmup-stable-decay: 10% warmup, a plateau across the switch, and a final 10% decay that always runs on clean inputs. The baseline is the same initialization and the same data order with that slice removed.
Both students are trained from random initialization. At 135M the architecture is SmolLM2-135M, the frozen teacher is SmolLM2-1.7B, and the budget is 4.2B FineWeb-Edu tokens. At 0.5B the architecture is Qwen2.5-0.5B, the teacher is Qwen2.5-1.5B, and the budget is 10B tokens. The main runs do not repeat data.
At 135M, standard distillation beats next-token prediction by 0.017 nats. All three perturbations go further. Adjacent blending is the largest.
| Method | Baseline | Validation loss |
| 135M adjacent blend λ=0.8, first 30% | standard distillation | −0.025 nats |
| 135M noise σ=0.4, first 50% | standard distillation | −0.019 nats |
| 135M parallel blend λ=0.7, first 30% | standard distillation | −0.018 nats |
| 135M standard distillation | next-token prediction | −0.017 nats |
| 0.5B adjacent blend λ=0.5, first 30% | standard distillation | −0.020 nats |
| 0.5B noise σ=0.75, first 50% | standard distillation | −0.012 nats |
| 0.5B parallel blend λ=0.5, first 15% | standard distillation | −0.012 nats |
That 0.025 is 1.45 times the 0.017 distillation itself adds. Two more seeds of the 0.5B adjacent run land at 0.021 and 0.019, and the paired standard error is 0.0003.
The perturbed fraction has a real optimum. At 135M both blends peak at 30%, and a 70% slice finishes worse than next-token prediction. Noise is looser: 50% is best, and 75% still helps. At 0.5B, adjacent blending again prefers 30%, noise is similar at 30% or 50%, and parallel blending prefers 15% and hurts at 50%. During the blend the 0.5B student lags by about 1 nat, passes the baseline near 47% of training, and finishes 0.020 nats lower.
Small noise barely moves the teacher. At 0.5B, σ=0.4 raises the frozen teacher's loss by 0.07 nats. At σ=1.2 the teacher's loss rises by 3.5 nats and the student ends worse than standard distillation. The best σ is 0.4 at 135M and 0.75 at 0.5B, found by a sweep.
Ablations use 135M noise at σ=0.4 for the first 50%. The three-seed mean gain is 0.015 nats, seeds differ by up to 0.008, and the 0.019 in the table is one of those seeds. Without a teacher, noise and parallel blending beat next-token prediction by only 0.003 nats, about a seventh of the gain with a teacher. Adjacent blending without a teacher is 0.03 nats worse. Perturbing only the teacher recovers about 90%. Perturbing only the student recovers about a third. The two one-sided gains sum to about a fifth more than perturbing both.
Softer targets do not reproduce the gain. Entropy rises from 1.97 to 2.46 nats at σ=0.4. Matching that entropy on clean text, with a temperature or with label smoothing, gains at most 0.003 nats. A weaker teacher for the first half also stays within 0.003. Extra text is not the source either: adjacent blending locked to the baseline's own tokens still gains 0.028 nats on the same seed, a bit above the regular 0.025. Randomly replacing 15% of tokens behaves differently. Student-only replacement gains 0.013, and sending the replaced text to both models gains 0.003, because the original token is gone.
The gain survives reused data. On the same 4.2B budget, 16 epochs over the first 262M tokens, raising AdamW weight decay from 0.1 to 1.0 lowers standard distillation by 0.033 nats. Weight decay 10 diverges. On that tuned baseline, noise still gains 0.023 and 0.020 nats. The same noise without a teacher gains 0.008. On fresh data at weight decay 1.0, noise still gains 0.020.
Quantization is measured at 0.5B on linear-layer weights, output head excluded, groups of 128, activations left in 16 bits. Under int4 GPTQ, adjacent blending's advantage grows from 0.020 to 0.024 nats, and parallel blending and noise grow to 0.017 and 0.015. int4 AWQ leaves the gaps essentially unchanged. int3 round-to-nearest raises standard distillation by 6.2 nats and the two blends by 1.2 and 1.5. Mean excess kurtosis of the linear weights is about 0.5, and about 0.14 for both blends. Under GPTQ the blends' int3 damage is about a quarter lower. Under AWQ the students lose about the same.
On seven lm-evaluation-harness zero-shot tasks at 0.5B, standard distillation averages 52.5%. Noise reaches 53.6% (+1.1±0.4), adjacent blending 53.5% (+1.1±0.3), and parallel blending 52.9% (+0.5±0.3). SciQ moves from 77.3 to 81.3 with noise and to 80.9 with adjacent blending. Both blends drop about a point on LAMBADA. At 135M the average moves by at most 0.6 points, in line with the bootstrap error.
The objective and the token budget stay fixed, and the student is still trained from scratch. At 135M the extra 0.025 nats is larger than the 0.017 distillation itself adds over next-token training. The step is incremental, and the increment is bigger than adding distillation.
Teacher-side perturbation supplies most of the gain. Student-side perturbation supplies about a third. Under repeated data the improvement stacks on a tuned weight decay. The loss gap is still there after int4 quantization. σ, λ, and the perturbed fraction all need a sweep, and the optimum moves with scale. One run is about 20 H100-hours at 135M and about 90 at 0.5B.
The largest student is 0.5B, trained for 10B tokens. Whether the gain holds for a larger student or a longer run is left open. There is no preset rule for the amplitude: the best σ moves from 0.4 to 0.75, and the best parallel-blend λ from 0.7 to 0.5. A 70% perturbed slice at 135M finishes worse than next-token prediction.
The 0.025 nats figure is a single run. Noise gains spread by up to 0.008 nats across seeds, and most settings are trained once. Gaps below about 0.003 nats sit inside the 135M evaluation resolution.
In the teacher-only setup, averaging four noise draws cuts the gain from 0.0122 to 0.0076 nats, a gap the appendix treats as seed noise. The student does not learn the teacher's local response: correlation along the noise direction is at most about 0.03. An overlapping adjacent blend is 0.008 nats worse than standard distillation, and that input also mixes in part of the token being predicted.
Downstream accuracy at 135M does not hold up. The one point at 0.5B is concentrated on SciQ, while LAMBADA drops. The int3 robustness shows up under round-to-nearest. AWQ removes the gap.