RL Training Failure: How Format Rewards Destroy Qwen's Reasoning
malliktwts · x · 2026-08-03
A developer accidentally reproduced classic reward-hacking and catastrophic forgetting while building GRPO from scratch and fine-tuning a Qwen2.5-0.5B model.
- Setup: The model was trained on the GSM8K dataset for 420K tokens without a KL-divergence penalty, using a +0.5 format reward to enforce a specific output structure.
- Results: Accuracy collapsed from 32% to 8%, and perplexity spiked from 3.6 to 15.4. Although format compliance hit 100%, the model's reasoning ability was destroyed.
- Root Cause: In GRPO's group-relative advantage calculation, a format reward given to everyone cancels out in the group mean. It provided zero learning signal, simply inflating the reward curve while the actual correctness signal quietly died.
- Fix: Dropping the format reward entirely and optimizing strictly for correctness prevents the reasoning collapse.
Related event: GRPO Training Pitfalls: Format Rewards Destroy LLM Reasoning(2 posts)→
More from Research
- Simplex Diffusion Models paper talk by EPFL/DeepMind's Justin Deschenaux on Oct 19 — ArnaudDoucet1 · 2026-10-16
- Meituan LongCat team's Data World synthesizes data environments to train data agents — meituan-longcat · 2026-10-16
- FDE-Gym benchmark shows huge gap on real AI delivery tasks: top model scores 0.563 — DeepLethe · 2026-10-16
- ForcingBooster halves computation of autoregressive video diffusion with patch-wise denoising — VAST-AI · 2026-10-16
- Baidu open-sources FlashMask: up to 1.55x faster than FlashAttention-4 for long-context training — BaiduResearch · 2026-10-16
- SemEval-2027 Task 9 opens: multimodal news framing analysis across 4 languages — preslav_nakov · 2026-10-16