Beyond Entropy: CPO for correctness-aware RLVR
tencent · hf · 2026-07-20
Beyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimization
The paper argues that entropy is a poor correctness signal in reinforcement learning with verifiable rewards (RLVR) because it cannot tell useful uncertainty from harmful confusion.
It proposes Contrastive Policy Optimization (CPO), which shapes advantages using token-level contrastive disagreement between a reference-guided distribution and a vanilla generation distribution. The authors claim this disagreement is a reliable indicator of token-level correctness. They also show:
- On-policy Distillation is a special case of CPO when the posterior distribution comes from an external teacher model.
- CPO addresses the zero-advantage problem.
- Experiments on in-domain and out-of-domain benchmarks show CPO significantly outperforms entropy-based RLVR methods while generalizing well.
- Further analysis suggests correct and incorrect responses naturally support exploration and exploitation respectively, and balancing the two yields the best performance.
Related event: CPO Optimizes Correctness-Aware Advantage Shaping in RLVR(2 posts)→
More from Research
- Could 10k agents discover learning methods beyond backprop, or just tweak existing ones? — SeunghyunSEO7 · 2026-09-11
- TRACES grades the process, not the answer: six-dimension eval for open-ended AI science — Faheem_uh · 2026-09-11
- Apodex launches TRACES, a benchmark grading AI on open-ended discovery instead of known answers — Faheem_uh · 2026-09-11
- Cognition's SWE-2 uses a KKT duality argument in RL to shift the effort Pareto curve — YouJiacheng · 2026-09-11
- VidMap uses RoMa coarse matching on all frames, fine-scale only for keyframes — ducha_aiki · 2026-09-11
- Bug Hunt Bench author: leaderboard noise is about 2-3 points — PawelHuryn · 2026-09-11