On-Policy Distillation in Frontier Model Training Isn't a Free Lunch
heghbalz · x · 2026-07-22
As On-Policy Distillation (OPD) becomes a default post-training tool for many open-source frontier models like DeepSeek and MiMO, its potential flaws are surfacing. OPD combines RL's on-policy nature to reduce exposure bias while providing token-level supervision like SFT.
However, an expert points out an inherent failure mode: early mistakes are structurally uncorrectable. When the student model takes an early wrong turn during rollout, the per-token KL divergence computed along that frozen path cannot pull it back. This is proven to be a built-in flaw of the objective function rather than noisy gradients, meaning even a perfect teacher cannot correct this structural learning bias.
More from Research
- NLI alone rejects 5/6 new agent memories; a 0.45 similarity gate fixes it all — Alternative_Pin9598 · 2026-10-26
- AI folks, stop saying "biology" when you mean drug development — KevinKaichuang · 2026-10-26
- ostris pre-training "Primal VAE": 1-channel latents match modern 8x VAEs via patching — ostrisai · 2026-10-26
- Shanghai AI Lab-led team releases 100-hour egocentric wearable dataset for humanoid training — ChuckDBrooks · 2026-10-26
- ConvFill: A 0.6B Talker Buys Time While a Large Reasoner Whispers the Real Answer — andimarafioti · 2026-10-26
- Flow Reasoning Models turn flow matching into iterative reasoners, hitting 99.5% on Sudoku-Extreme — alec_helbling · 2026-10-26