Inside Post-Training: How SFT and RL Enhance Model Combinatorial Generalization

Recent academic studies have deeply explored the underlying mechanisms of why combining Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) achieves optimal performance in post-training reasoning models. Researchers point out that the core of this improvement lies in the model's "combinatorial generalization"—treating reasoning as a combination of reusable "atomic modules," thereby demonstrating "reasoning creativity" to solve novel problems.

Division of Labor Between SFT and RL

From a mechanistic perspective, SFT and RL play complementary roles. A research hypothesis summarized by @tw_killian clearly defines their division of labor: pre-training establishes the overall distribution of "possible concepts," SFT demonstrates how these logical concepts can be arranged and combined, and RL drives the model to explore new concept orderings when facing new contexts and tasks. Specifically, SFT provides rich combinatorial reasoning trajectories, while RL breaks these complex trajectories down into independently callable atomic skills and routing strategies.

How RL Drives Skill Reuse

Even with only final answer rewards and no intermediate step supervision, RL demonstrates a strong driving force. Research forwarded by @heghbalz indicates that RL first reinforces the model's original basic skills, then combines these skills into more complex reasoning processes, ultimately forming a repeatedly callable skill library. Control experiments show that this mechanism allows the model to solve hold-out problems that base models almost cannot solve, exhibiting stronger generalization on unseen combinations and varying reasoning depths.

2026-07-10 ~ 2026-07-11 · 5 related posts