Why SFT+RL Enhances Reasoning Compositional Generalization
tw_killian · x · 2026-07-11
This paper investigates why combining SFT and RL often yields the best results in post-training reasoning models.
The authors propose that this advantage stems from compositional generalization: the model treats reasoning as a combination of reusable "atomic modules." The paper formalizes this using a hierarchical latent variable selection model and concludes that:
- SFT provides broad coverage of reasoning trajectories, allowing the model to learn reusable atomic modules;
- RL explores and recombines these modules, pushing beyond the SFT distribution to achieve stronger compositional generalization;
- Experiments show that the best results typically occur when SFT provides sufficient coverage while RL explores new combinations.
A reshared comment also mentioned the author's view that "reasoning creativity" can be understood as the model recombining known skills to solve new problems.
More from Research
- Speculative Diffusion Paper Maps the Speed-Quality Pareto Frontier via Two Tradeoffs — UCLA · 2026-10-26
- Git Records Fall Short for Auditing AI's Role in Research Repositories, Paper Finds — Koutian Wu · 2026-10-26
- What a 'Fused DeepLIFT' Actually Means: Gradient Overrides Without Hooks — jmschreiber91 · 2026-10-26
- New paper: AI agents negotiate badly under uncertainty and break deals on purpose — menhguin · 2026-10-26
- Cherimoya v0.3.1 Ships Fused DeepLIFT, ~7x Faster Attribution for Genomics Models — jmschreiber91 · 2026-10-26
- Proto-tools, a universal infrastructure layer for generative biology, now open source — jmschreiber91 · 2026-10-26