Bellman Policy Optimization
Zhuoqing Song, Haotian Xu, Xikun Zhang, Lidong Bing
cs.LG, cs.CL, math.OC
2026-09-15
BPO turns Policy Mirror Descent into a critic-free trajectory loss. On Qwen3-30B-A3B-Base it peaks at 50.5% Avg@32 on AIME 24-26, +3.1 over CISPO, +11.0 over GRPO.
RL with verifiable rewards is the default way to train reasoning models. GRPO-style methods sample a group of answers per prompt, normalize rewards inside the group, and plug those advantages into a PPO-style clip. Under outcome supervision every token in a response shares one advantage, so a learned critic is already optional. Naive Policy Mirror Descent still wants advantages or Q-values at every prefix. The aim is the same PMD update, using only the terminal reward and the prompt’s expected reward.
Generation is a finite-horizon MDP with a terminal verifier. Non-terminal advantage is a difference of successive state values; the sum along a response collapses to R(x,y)−V(x). Summing the PMD stationarity condition across tokens yields a trajectory residual: terminal reward minus initial value, minus per-step log ratios and reverse KL. Minimizing the squared residual has the same unique optimum as PMD on states reachable under the rollout policy.
Four approximations make a loss. Linearize the residual at π=μ so the residual factor becomes R−V. Estimate V(x) by the group mean and the prompt weight by inverse group std, recovering GRPO’s normalized advantage. Replace full-vocabulary KL with binary KL (this token versus the rest). The gradient multiplier becomes (1−μ)/(1−π), a ratio of complementary probabilities. That ratio explodes as π→1, so the code adds smoothing ε and a cap C. The loss still looks like GRPO; the importance ratio r=π/μ is replaced by ω=(1+ε−μ)/(1+ε−π). The run uses ε=0.1, C=3.0, and GRPO’s clip mask.
Backbone: Qwen3-30B-A3B-Base. Data: English DAPO-Math-17k. Only the policy loss changes versus GRPO-ClipHigher, GSPO, CISPO, and DPPO. Each step: 256 prompts, 16 responses, 16384-token cap, 400 training steps with eight optimizer updates each. Metric: Avg@32 on AIME 24/25/26.
Peak mean:
| Method | AIME24 | AIME25 | AIME26 | Avg |
| GRPO-ClipHigher | 45.6 | 34.8 | 38.0 | 39.5 |
| GSPO | 50.3 | 35.5 | 44.6 | 43.5 |
| CISPO | 52.7 | 39.0 | 50.4 | 47.4 |
| DPPO | 55.8 | 39.2 | 44.2 | 46.4 |
| BPO | 57.4 | 41.0 | 53.0 | 50.5 |
That is +3.1 over CISPO and +11.0 over GRPO. After 400 steps BPO sits at 49.4 versus 45.5 for DPPO. BPO leads all three contests. Ablations on Qwen3-4B-Base report similar scores across a band of ε and C.
RLVR already wants to skip critics. BPO supplies a PMD-shaped reason: under terminal rewards the intermediate values telescope, and the practical change is the importance weight, not the sampling stack. If a stack already runs GRPO or CISPO, swapping that multiplier is a small patch. A 3-to-11 point gap on contest math is large enough to try.
The equivalence theorem is about the squared trajectory residual. The shipped loss adds linearization, binary KL, smoothing, and clipping. All numbers are math, one 30B-A3B base, no code or chat. Group-mean V(x) is noisy for tiny groups or all-0/1 rewards, the same pathology as GRPO. AIME26 leakage is the reader’s problem. Hyperparameter sweeps are on 4B.