Researchers Uncover 'Value Flattening' in PPO Critics — Sparse Supervision on 3 States Fixes It
Shanghai-AI-Laboratory · hf · 2026-09-17
Researchers identify a systematic failure mode in PPO critics for RL-based LLM training, dubbed Value Flattening: true state values estimated from multiple Monte Carlo continuations shift sharply across intermediate states while critic predictions remain flat. The effect reproduces in FrozenLake and worsens as state spaces grow.
- Cause: an implicit variance penalty in the critic loss plus redundant updates from temporally correlated states with similar gradients
- Fix — SP³O (SParse PPO): apply the value loss to only a few well-separated states per response
Experiments on Qwen3-Base show SP³O with just three supervised states per response mitigates Value Flattening and consistently improves the learned policy across model sizes and evaluation suites.
More from Research
- Gowers responds to letter on maths and AI signed by 25 Fields medallists — tak3sh8 · 2026-09-17
- World models share one architecture — the tokenizer is where methods diverge — abursuc · 2026-09-17
- TMLR tightens desk rejects amid submission deluge, quizzes authors on their own papers — RexDouglass · 2026-09-17
- Anatomy of modern world models: a tokenizer compresses states, a module predicts the next — abursuc · 2026-09-17
- Speculative decoding: small draft model proposes tokens, big model verifies in one pass — HowDevelop · 2026-09-17
- Researcher proposes a journal for vibe-coded papers, with AI models as reviewers — peter_richtarik · 2026-09-17