Preventing Entropy Collapse: SAF Framework Enhances LLM Math Reasoning
Yifan Ding · hf · 2026-08-03
This paper introduces the Stable Advantage Fusion (SAF) framework to address challenges in combining reinforcement learning with on-policy distillation (OPD). The authors find that fusing RLVR (Reinforcement Learning with Verifiable Rewards) and OPD advantages with a fixed coefficient triggers entropy collapse due to magnitude and temporal miscalibrations.
SAF resolves these issues via a lightweight, four-stage pipeline applied to the OPD advantage: a sparsify-then-compress mechanism for magnitude control paired with a warm-up-then-anneal mechanism for temporal control. Evaluated on Qwen3-1.7B/4B/8B across seven math reasoning and code generation benchmarks, SAF consistently prevents entropy collapse, improving aggregate scores by 0.51-2.70% over fixed-coefficient fusion while ensuring stable training.
More from Research
- SMPL-X C++ Plugin for UE5.8 dramatically speeds up 3D human body rendering — Michael_J_Black · 2026-08-03
- lossfunk Research: Diffusion Models Internally Represent Visual Illusions — paraschopra · 2026-08-03
- ICLR Introduces New Measures as AI-Generated Papers Flood Submissions — gabriberton · 2026-08-03
- Tencent Hunyuan's E-Bench: Top LLMs Score Under 74% in Real-World Multi-Step Agent Tasks — 腾讯混元 · 2026-08-03
- SIAM Highlights 'Subgame Perfect' Algorithm Design Blending Theory and Practice — prof_grimmer · 2026-08-03
- SQLite Hit by Fake Critical CVEs Generated by LLM Slop — jedisct1 · 2026-08-03