Fixing PPO Training Noise: Value Estimation as Classification Task
joecole · x · 2026-08-08
In reinforcement learning with binary rewards, standard PPO typically trains the critic using MSE regression. However, minor value errors can significantly amplify noise in advantage functions.
A recent paper proposes a simple fix: stop regressing scalar values and treat value estimation as a classification task. The authors replace the MSE head with a categorical predictor over value bins (HL-Gauss PPO), trained using cross-entropy, and then decode the distribution back to a scalar for standard GAE.
Tested on Qwen models across math and reasoning tasks, this method consistently outperforms standard PPO and DAPO, yielding lower-variance advantages and better calibration.
More from Research
- Microsoft Research Proposes Unified Agent Architecture for Long-Horizon Tasks — anselm · 2026-08-08
- HiGraph Fixes Agent Memory: Hierarchical Subgraph Rewriting Cuts Token Costs — anselm · 2026-08-08
- AI Safety Expert Recommends Paper on International AI Governance Architecture — S_OhEigeartaigh · 2026-08-08
- Paper Share: Tsallis Regularized Optimal Transport and Ecological Inference — FrnkNlsn · 2026-08-08
- Bittensor Competition Proves Decentralized AI Alignment: Steering Model Thoughts Without Retraining — bittingthembits · 2026-08-08
- 1B Parameter DNA Model Rivals 40B Giant, 2330x Faster — huggingface · 2026-08-08