REOPOLD: selective teacher feedback makes on-policy distillation for small reasoning models stable
burkov · x · 2026-11-13
Small models can gain reasoning from stronger models via on-policy distillation: the student generates its own responses and gets teacher feedback. But standard methods apply feedback equally across tokens, so a few extreme negative signals can dominate training, suppress useful reasoning paths, reduce diversity, and sometimes leave the student worse than before. The article introduces REOPOLD, which adjusts which teacher signals affect training, how strongly negatives count, and how these rules evolve over time, tested on math, visual, and tool-using reasoning tasks.
More from Models
- Ditto rolls out uncensored inference to all users — bittingthembits · 2026-11-13
- Google says Gemini 4 Hydrogen expands query fan-out and the range of sites Search links to — OnlineInference · 2026-11-13
- 124B model in 22GB: Cactus Quants hits 1.418 bits/weight, 47 tok/s on an M5 Max — pbaylies · 2026-11-13
- PostTrainBench: Fable 5.2 ties 5.1 at 44.6%, speculating Anthropic capped AI research ability — scaling01 · 2026-11-13
- Gemini 4 Hydrogen lands in Devin, scoring 38.2% on FrontierCode 2.0 behind GPT-6.1 Sol — scaling01 · 2026-11-13
- Gemini 4 Carbon comes within 1.9 points of Fable 5.2 on Code4Scene at ~1/10 the cost — ZhitingHu · 2026-11-13