Why SFT+RL Enhances Reasoning Compositional Generalization
tw_killian · x · 2026-07-11
This paper investigates why combining SFT and RL often yields the best results in post-training reasoning models.
The authors propose that this advantage stems from compositional generalization: the model treats reasoning as a combination of reusable "atomic modules." The paper formalizes this using a hierarchical latent variable selection model and concludes that:
- SFT provides broad coverage of reasoning trajectories, allowing the model to learn reusable atomic modules;
- RL explores and recombines these modules, pushing beyond the SFT distribution to achieve stronger compositional generalization;
- Experiments show that the best results typically occur when SFT provides sufficient coverage while RL explores new combinations.
A reshared comment also mentioned the author's view that "reasoning creativity" can be understood as the model recombining known skills to solve new problems.
More from Research
- UniAction: unified action conditioning lets one world model handle mixed action types — ZeYanjie · 2026-10-27
- Artificial Analysis breaks down all sub-evaluations in Intelligence Index v4.3.2 — ArtificialAnlys · 2026-10-27
- Infini-AI Lab open-sources Hermit, a framework for agent harness evolution and continual learning — BeidiChen · 2026-10-27
- Meta FAIR's Self-Direction turns agent failures into reusable LoRA skills, +13.8pp on Terminal-Bench — karen_ullrich · 2026-10-27
- DONIC hits 86.79% on bimanual dexterous tracking, crushing prior SOTA's 9.43% — liuziwei7 · 2026-10-27
- Why don't more LLMs ship 10M-token context? A researcher asks about attention tradeoffs — agarwl_ · 2026-10-27