Negative Self-Distillation: Learning to Reason by Avoiding Flaws
Rongcan Pei, Zhepei Wei, Shuyao Xu, Xinyu Zhu, Wei-Lin Chen, Yu Meng
cs.CL, cs.LG
2026-09-10
NSD pushes a student off self-written “careless” reasoning, no gold labels. Qwen3-1.7B/4B/8B gain 2.3, 7.5, and 6.0 points on seven math benchmarks.
On-Policy Self-Distillation (OPSD) lets a model teach itself. Gold answers are privileged context, and the teacher puts dense token-level supervision on the student's own samples. That teacher already knows the answer, so the traces come out linear and overconfident. Matching that distribution squeezes out uncertainty and self-correction, the behaviors hard problems actually need.
Label-free RL has the opposite failure. Intuitor rewards self-certainty; TTRL treats majority vote as a pseudo-label. Both skip gold answers and then bet that the model can grade itself. On a weak 1.7B model, high confidence is not high accuracy. Intuitor loses 0.5 points on average.
NSD flips the objective. Do not imitate a privileged good trace. Move away from a bad reasoning mode the model writes for itself.
Training uses MATH questions only, labels discarded. For each problem the student samples a solution, then writes a negative condition, typically an instruction to act as a careless reasoner. A frozen copy of the initial weights runs two forwards:
The gate is Gt = max(0, pneg − pref). A token is penalized only if the negative condition raised its probability. Ordinary grammar that is likely under both contexts gets a zero gate. Raw unlikelihood, −log(1−p), explodes as p approaches 1 and would rip out punctuation and spaces. A sigmoid cap, equivalent to Gt / (2 − pθ), puts weight on low-to-mid confidence reasoning tokens. A pointwise forward KL on the sampled token, with α=0.01, keeps the student from drifting off the reference. Without that KL, training collapses mid-run and the gate stops meaning anything.
One student rollout per example. The loss needs three scalar probabilities, not a full-vocabulary alignment. The two frozen forwards share weights and can run in parallel.
Qwen3-1.7B/4B/8B, non-thinking mode, Avg@8 on seven benchmarks (AIME 24/25/26, HMMT 2025, AMC 2023, OlympiadBench's 675 open-ended items, MATH-500):
| Model | Base | OPSD | Intuitor | TTRL | NSD |
| 1.7B ΔAvg | 0 | +1.1 | −0.5 | +0.3 | +2.3 |
| 4B ΔAvg | 0 | +1.0 | +1.3 | +0.2 | +7.5 |
| 8B ΔAvg | 0 | +0.3 | +1.9 | −0.1 | +6.0 |
AIME 2024 goes from 23.8% to 35.8% on 4B and from 28.8% to 39.6% on 8B. The 1.7B gain is smaller, still with p=0.001; the 4B/8B confidence intervals sit above zero. On Qwen3-4B, reflection tokens such as “wait” average 3.6 per response for the base model, 2.2 for OPSD, 0.8 for Intuitor, and 7.5 for NSD. Negative conditions do not have to be online. Question-only offline conditions get +7.3 on 4B, close to the default online +7.8. Irrelevant Wikipedia noise remains competitive and drops step time from 68s to 54s. Under thinking mode, 4B still gains +3.0 on average.
This is a post-training path that needs neither gold answers nor a stronger external teacher. The signal is which tokens a negative condition inflates, not whether the model thinks it is right, which is why confidence-based RL stalls on small models. One rollout and a scalar loss beat GRPO-style groups of 8 and OPSD's full-vocabulary match on wall-clock.
Gains are large at 4B and 8B, modest at 1.7B. That tracks negative-condition quality. Before treating NSD as a general recipe, check that the base model can write a contrast that actually means something.
The authors say very small or weak models may not produce a useful negative contrast, and the optimization then dulls. The online recipe still generates a negative condition and runs two frozen forwards; wiki-irr removes that generation cost, while the headline numbers still come from the online setup.
All main results are Qwen3 on math. The gate is a probability gap. Nothing proves it isolates reasoning errors rather than style tokens the negative prompt happens to activate. More reflection tokens are not a measured self-correction rate; the paper gives frequencies and case studies. Training is two epochs on MATH. Transfer to code or agent traces is not shown.