Smaller Models, Better Rejects: Preference Distillation Scaling
Rui Cai, Wenhui Zhu, Xiwen Chen, Jincheng Cao, Han Yu, Shayan Mohajer Hamidi, Zelin He, Qiyao Ma, Daiwei Chen, Xuanzhao Dong, Yuanda Xu, Jelena Markovic-Voronov, Kayhan Behdin, Zhengze Zhou, Ran He, Alborz Geramifard, Rohit Jain, Zhe Zhao
cs.LG
2026-09-30
Preference distillation defaults to the student's own responses as rejects; this paper shows frozen smaller models (0.5B-14B) produce stronger rejects for 7B-72B students on code and math, at 0.9-50% of the inference compute.
Preference distillation has a standard recipe: the teacher's output becomes the chosen response, the student's own output becomes the rejected one, and DPO does the rest. Two assumptions sit underneath. First, that the student's own failures are the most informative negatives. Second, that rejects must come from a model at least as large as the student, which means reject-generation compute scales with the student, a real cost when you are distilling 72B models.
The first assumption also has a subtle flaw. After SeqKD on teacher responses, the student's wrong answers tend to be near misses that already carry high probability under the DPO reference policy. High-probability rejects leave DPO with little contrast to exploit.
A LinkedIn-led team tested both assumptions and found they fail. Smaller frozen models generate rejects that are cheaper and train stronger students.
The setup follows the SODA two-stage pipeline: Qwen2.5 students first run SeqKD on execution-verified GPT-4o responses (75,752 code prompts) or DeepSeek-R1 trajectories, then DPO on a disjoint set of 24,248 prompts. Within each student scale, prompts, chosen responses, initialization, objective, and training budget are held fixed; only the reject source varies. Every reject is verified incorrect (fails at least one unit test).
A "smaller Base source" is a frozen vanilla model with strictly fewer parameters than the student; a 14B student can draw rejects from 0.5B/1.5B/3B/7B. Baselines are vanilla Self (the original same-scale model) and post-SeqKD Self (the reference policy that initializes DPO).
To explain the phenomenon, the paper frames reject construction as inverse data design: given a student, which reject distribution should you supply? In a linearized feature model of DPO (fixed features taken as score gradients at initialization, an NTK-style view), each source is scored by transfer mass (how much it moves the update) and adverse share (the fraction pointing the wrong way); their combination is net transfer. A finite-horizon bound says a source helps when transfer mass is large and adverse share is small. Three interventions follow directly:
The main result is clean: all 18 smaller-Base configurations across 7B–72B students beat both Self constructions on code generation; math reasoning shows the same pattern.
| Configuration | Metric | Result |
| 14B student + Qwen2.5-3B rejects | KoDCode avg@4 vs vanilla Self | +3.1 points |
| 14B student + Qwen2.5-3B rejects | MATH-500 avg@4 vs vanilla Self | +1.9 points |
| 72B student + 0.5B rejects | Generation compute vs same-scale Self | 0.9%, a 108.3× reduction |
| All smaller-Base configs | Inference compute vs Self | 0.9%–50.1% |
The interventions confirm the mechanism one by one. Raising the Llama-1B share from 0% to 100% lifts a 14B student from 59.73% to 63.28% avg@4, monotonically. Length-matched gibberish performs like continued SFT, so merely suppressing some sequence is useless; real rejects reassigned to different prompts still beat gibberish; even rejects reduced to a shuffled token inventory recover 40%–92.5% of the real gain. Reference likelihood is the strongest signal: source-level reference likelihood correlates negatively with downstream avg@4 at every scale (Pearson r=-0.97 for the 7B student), and reselection toward lower likelihood beats reselection toward higher likelihood for all five sources; the 1.5B source goes 63.14% vs 61.88%.
Cross-family results hold: Llama rejects fed to Qwen students win all nine code configurations and eight of nine MATH-500 configurations.
For distillation teams this is a configuration you can copy tomorrow: stop sampling rejects from the student, generate them with a 0.5B–3B model, cut inference budget by one to two orders of magnitude, and get better results. The finding covers four student scales, two task families, and two model families.
The design principle transfers beyond the pipeline: useful rejects preserve task-relevant structure while staying loosely coupled to the reference policy. Selecting rejects by low reference likelihood works regardless of generator scale, which makes it an operational rule for anyone constructing negative data.
To be precise about magnitude: gains sit in the 1–3.5 point range. This is an efficiency win plus a mechanism finding, not a new capability.
The authors flag several limits. The theory rests on a linearized feature model whose fixed-feature assumption holds only near initialization. Optimization-level diagnostics (reject fields, reward margins, RC-DPO) fail to recover the smaller-Base-over-Self ordering, so existing DPO diagnostics cannot explain the effect. The scope is verifier-backed code and math only; tasks without a verifier have no way to confirm a reject is actually wrong.
Reading closely raises more questions. All students are Qwen2.5-Instruct with GPT-4o or DeepSeek-R1 teachers; whether the effect depends on these families is untested. Gains narrow visibly at 72B, where sources from 0.5B through 32B cluster between 66.7% and 67.5%, and the data do not support extrapolating further up in scale. The token-shuffling result is itself counterintuitive: why an unordered token set still carries corrective signal is explained only gesturally. External-benchmark gains over Continued-SFT run smaller than in-domain KoDCode gains, and the paper does not dig into that gap.