Study: Training a Single Layer Can Match Full-Parameter RL
机器之心 · wechat · 2026-07-08
A joint team from the University of Minnesota, Peking University, and Amazon challenged the implicit assumption that RL post-training should uniformly update all Transformer layers. Introducing a "layer contribution" metric, they discovered that RL gains are highly concentrated in a few intermediate layers. On Qwen3-1.7B, training only the 10th layer recovers 114% of the gains from full-parameter RL, while the weakest layer recovers only 28%, showing a 4x difference between the best and worst layers.
Experiments spanned 7 models, 2 model families, 3 RL algorithms (GRPO/Dr.GRPO/GiGPO), and 3 task types: math reasoning, code generation, and agentic decision-making. The paper demonstrates that single-layer training strategies based on these findings can consistently outperform standard full-parameter RL training, offering new insights for RL post-training algorithm design.
Related event: Training Single Transformer Layer Can Match Full-Parameter RL, Study Finds(2 posts)→
More from Research
- TRL adds Score Centering, a simple fix for RLHF training instability — lvwerra · 2026-10-27
- Latent-Foresight: world models that learn to predict their own future — abursuc · 2026-10-27
- UK AISI papers: no good safeguards yet against misuse of biological AI tools — davidmanheim · 2026-10-27
- CMU's XYBench: LLMs answer queries literally and reinforce user misconceptions — valentina__py · 2026-10-27
- arXiv advice from T. Dietterich: wrong primary category will almost surely get your paper on hold — tdietterich · 2026-10-27
- arXiv has 4,000+ papers on hold amid submission surge; moderator shares how to avoid it — tdietterich · 2026-10-27