SFT before RL helps small models but hurts large ones, Base Labs experiments show
cwolferesearch · x · 2026-10-30
Base Labs distilled GPT-5.6-Sol reasoning traces via SFT before RL on Qwen models from 0.6B to 8B. Key findings:
- SFT raises pre-RL accuracy but doesn't guarantee better post-RL performance
- Small models (0.6B/1.7B) benefit from SFT+RL; large models (4B/8B) see little or no gain, sometimes worse
- Scaling SFT data (0.5M–4M tokens) doesn't change the trend
Mechanism: GRPO needs mixed correct/incorrect rollouts for learning signal. SFT teaches small models to solve some problems, enabling RL to start. Large models can already find correct solutions via sampling; SFT only reduces next-token entropy, hurting exploration. Takeaway: skip pre-RL SFT for models that already solve a subset of problems.
More from Research
- RLNF: Training AI Models on Human Preferences Directly from Brain Signals — alex_verem · 2026-10-30
- Impagliazzo, Paturi and Williams launch site to scrutinize OpenAI's claimed ETH refutation — fortnow · 2026-10-30
- Know, then Draw: UCSD & Adobe Cut Image-Gen Training Compute 48x While Boosting DPG-Bench 10% — Yucheng Mao · 2026-10-30
- Live mechinterp session hunts 'neurons of sadness' in Qwen3.5 — burny_tech · 2026-10-30
- RT-SAFE Benchmark Shows Faster Reasoning Doesn't Mean Safer Agents in Real-Time Embodied Environments — Lianhuiq · 2026-10-30
- LeCun team's LeHami: symplectic Hamiltonian flow keeps JEPA world models from collapsing latent states — ylecun · 2026-10-30