NeurIPS paper: with mixed data and RL-matched scale, SFT can generalize as well as or better than RL
kastnerkyle · x · 2026-11-17
A NeurIPS 2026 paper, "Rethinking 'RL Generalizes, SFT Memorizes': The Role of SFT Data" (NUS/Yale/Cisco Foundation AI, arXiv:2611.14912), argues the community consensus since DeepSeek-R1 may stem from an unfair comparison:
- Standard setups give RL a huge rollout budget per prompt while SFT trains on a tiny data slice
- On Qwen2.5/Qwen3, carefully mixing data sources beats single-teacher distillation, and scaling SFT data to match RL's rollout budget lets SFT generalize on par with RL
- Combining both factors, SFT can generalize even better than RL
- LLM annotations characterize solution distributions in RL rollouts: larger datasets cover more tail methods, which provide generalizable reasoning signals
- Backed by training-dynamics analysis of shallow transformers under both RL and SFT
More from Research
- BrickFlow uses SE(3) flow models to reconstruct buildable LEGO assemblies from images — Peter Kulits · 2026-11-17
- Meta-model writes weight updates from one paragraph, beating LoRA on many benchmarks — gdm3000 · 2026-11-17
- SPARE preprint: 2.55x faster decoding for linear attention on a 1.3B model — krishnan · 2026-11-17
- Study estimates ~12% of philosophy published today may be AI-generated — leland_mcinnes · 2026-11-17
- Berkeley EECS & Statistics open all-rank faculty search in AI for Physical Sciences — berkeley_ai · 2026-11-17
- Origami as physical reservoir computing: powering robotic crawling in Scientific Reports — cephaloform · 2026-11-17