Paper reveals why PPO value functions fail, proposes BPCO for stable training
heghbalz · x · 2026-08-27
Addressing the industry shift to GRPO due to the difficulty of training critics in LLM reasoning, a new paper identifies the root causes of PPO instability: ratio clipping acting weird on low-probability tokens, bootstrapped error accumulation, and GAE breaking on variable response lengths.
The proposed solution, BPCO (Best-Practice Critic Optimization), stabilizes value estimates by providing the critic network with "cheat codes"—full reference answers and rubrics hidden from the policy model during training. Results show solid performance when scaling from 1.5B to 30B MoE models.
Related event: New Paper Explains PPO Value Function Failures, Proposes BPCO(2 posts)→
More from Research
- Gwern: Evolution as Backstop for Reinforcement Learning — CatAstro_Piyush · 2026-08-27
- RSI-Exam Benchmark: Quantifying Recursive Self-Improvement in LLMs — cihangxie · 2026-08-27
- Gaussian fiddling brings facial expressions to Clug — repligate · 2026-08-27
- Lightwheel and Hugging Face release 100k-hour egocentric dataset for Physical AI — vanstriendaniel · 2026-08-27
- PyTorch Ecosystem Adds Perforated, TokenSpeed, and 8 Others — zhyncs42 · 2026-08-27
- V-Rubrics: Improving Visual Faithfulness via Rubric-Based Reinforcement Learning — liuziwei7 · 2026-08-27