Why SFT generalizes worse than RL: off-policy data, not the objective
a_karvonen · x · 2026-10-03
Maxime Labonne highlighted research arguing that SFT generalizes worse than RL not because of the SFT objective itself, but because the data is off-policy: rewriting expert trajectories to resemble the base model's own outputs can match or beat on-policy methods, with less forgetting as a bonus.
Research scientist akarvonen added his take: on-policy training has a much higher signal-to-noise ratio because it focuses only on the metric of interest rather than tone or response structure, so it perturbs the model far less.
More from Models
- Mystery Model 'Gestalt' Tops New Frontier Design Benchmark by 214 Points — soleio · 2026-11-19
- mrfakename releases abliterated GLM-5.3-Flash and Qwen3.8-Flash-Next — mrfakename0 · 2026-11-19
- Physician: AI competition now measured in weeks, GPT-6 versions 7 days apart — ramos_casals · 2026-11-19
- micro1's flow qc model screens training data for benchmark leaks before release — Exp_Mark · 2026-11-19
- Leaker predicts new DeepSeek paper this week, scheduled for November 11 — teortaxesTex · 2026-11-19
- Gemini 4 Neon rollout begins: leaked UI shows 3-tier Gemini 4 lineup with Carbon locked — testingcatalog · 2026-11-19