RL Training Failure: How Format Rewards Destroy Qwen's Reasoning
malliktwts · x · 2026-08-03
A developer accidentally reproduced classic reward-hacking and catastrophic forgetting while building GRPO from scratch and fine-tuning a Qwen2.5-0.5B model.
- Setup: The model was trained on the GSM8K dataset for 420K tokens without a KL-divergence penalty, using a +0.5 format reward to enforce a specific output structure.
- Results: Accuracy collapsed from 32% to 8%, and perplexity spiked from 3.6 to 15.4. Although format compliance hit 100%, the model's reasoning ability was destroyed.
- Root Cause: In GRPO's group-relative advantage calculation, a format reward given to everyone cancels out in the group mean. It provided zero learning signal, simply inflating the reward curve while the actual correctness signal quietly died.
- Fix: Dropping the format reward entirely and optimizing strictly for correctness prevents the reasoning collapse.
Related event: GRPO Training Pitfalls: Format Rewards Destroy LLM Reasoning(2 posts)→
More from Research
- AI Deceives Humans Perfectly in Social Deduction Games; Removing Safety Guardrails Reduces Lying Ability — alex_verem · 2026-08-03
- Estimating Frontier Model Sizes: Claude Fable at 4.5T Parameters — anpaure · 2026-08-03
- AI Impacts Academia: Pure Mathematics Faces Paradigm Shift — RexDouglass · 2026-08-03
- Can Weaker Models Replicate Frontier Discoveries with Hints? Exploring LLM Basins of Attraction — danshipper · 2026-08-03
- OpenAI Paper Explores 8 Agentic AI Applications in Scientific Computing — markjeffrey · 2026-08-03
- Meta Open-Sources SAM 3D Body for Robust 3D Human Mesh Recovery from Single Images — tom_doerr · 2026-08-03