Best Practice Critic Optimization
Penghui Qi, Xiangxin Zhou, Wee Sun Lee
cs.LG, cs.AI, cs.CL
2026-08-25
BPCO is a single-rollout actor-critic recipe: DPPO, reward-range value heads, Monte Carlo critic targets, raw advantages, and length-adaptive GAE. From 1.5B to 30B-A3B math models it matches or beats Dr.GRPO that samples 16 responses per prompt.
Group methods such as GRPO compare several answers to one prompt and skip a critic. Every token in an answer then shares the same outcome advantage, and each prompt costs many rollouts. A critic could emit token-level advantages from one trajectory, but common PPO recipes for LLMs are brittle. Ratio clipping treats low- and high-probability tokens unevenly. Bootstrapped value targets feed critic error into the policy. A fixed GAE λ downweights the terminal reward on long answers.
Two implementation choices make this worse. A linear value head can leave the known reward range. Batch advantage normalization keeps unit-scale advantages even after residual signal should shrink.
BPCO starts from a verl default recipe and adds pieces on a tiny solvable set: 1,460 math problems that DeepSeek-R1-Distill-Qwen-1.5B can already solve. A healthy optimizer should drive training reward near 100%. Failure is treated as an optimization bug.
The six steps are:
The critic baseline already includes decoupled GAE and length-adaptive GAE, but keeps an unbounded head and advantage normalization. BPCO differs from that baseline only by bounding values and keeping raw advantages. All methods use DPPO. The group baseline is Dr.GRPO with 16 responses per prompt, with fewer distinct prompts so trajectory count matches.
Main scores live in figures, not a final AIME table. The comparisons that the text supports are these.
On DeepScaleR (40.3K problems, 24k-token cap) with the 1.5B model, BPCO variants stay above both the group and critic baselines on training reward, explained variance, and AIME 2025 avg@32. The BPCO+Ans curve moves avg@32 into the low 30s while the baselines remain lower. Removing the value bound slows learning. Restoring advantage normalization inflates advantage magnitude again. A reference answer as privileged input speeds training and raises AIME; official solutions, present for only 7.3K problems, help a little. On the tiny set, privilege also overfits earlier.
On Qwen3-30B-A3B and its Base variant, trained on DAPO-Math-17k with a 12k cap, BPCO still beats the critic baseline. On A3B the critic baseline stops improving AIME 2025 after 100 steps. Against the group baseline, BPCO is better on A3B and comparable on Base.
On OpenRubrics with Qwen3-4B-Base and a frozen instruct judge, both BPCO variants learn faster than the group and critic baselines; the group baseline later catches up. Feeding the rubric only to the critic raises explained variance without raising final reward.
Sixteen samples per prompt is expensive for long CoT. If a single rollout plus a critic can match group-relative advantages, async single-trajectory RL becomes realistic. BPCO is less a new named algorithm than a default stack assembled from DPPO, VC-PPO, and VAPO that survives a "must fit the tiny set" audit.
Bounded values and raw advantages are small diffs worth copying. Privileged critics are worth trying when a reference answer exists, not as a universal add-on.
Evidence is math and rubric rewards only. The method needs a known reward range. Privileged variants need evaluator-side information. Critic compute is not reported as wall-clock. AIME is shown in plots without a final mean±std table. Fitting a tiny solvable set is not the same as best generalization. On the rubric task the group baseline ties late, so the single-rollout edge is sample efficiency, not a higher ceiling.