Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening
Yizhuo Li, Jianhao Yan, Yun Luo, Zhi Wang, Futing Wang, Rong-Xi Tan, Kanghui Tian, Ganqu Cui, Ning Ding, Peilin Zhao, Yafu Li, Yu Cheng
cs.LG, cs.AI
2026-09-16
PPO critics stay flat while MC state values swing inside a response. SP3O supervises a few spaced tokens. Qwen3-4B math avg@32 rises from PPO 37.60 to 45.57.
PPO for long-horizon LLM reasoning asks a critic to track state values along a response so GAE can form token-level advantages. If a reasoning step has already doomed the answer, or nearly locked it in, the value should move. This paper shows that it often does not. Monte Carlo continuations from the same policy can swing by 0.5 or more between nearby tokens, while the PPO critic stays nearly flat and sometimes moves the wrong way. The authors call this Value Flattening. It shows up on correct and incorrect responses, across training checkpoints of Qwen3-4B-Base on DAPO-Math-17k. In stochastic FrozenLake, growing the maze from 32 to 128 under a fixed training setup makes the critic map smoother and less accurate.
With terminal-only rewards and γ=λ=1, every token in a response is regressed to the same sampled return R. Split the MSE and one term is the variance of the critic's predictions inside that response. Dense token-level supervision therefore penalizes value movement directly.
LLM states are prefixes. Adjacent states differ by one token, hidden representations and gradients stay highly aligned, and cosine similarity falls only as token distance grows. Dense updates then pile many similar gradients onto neighboring positions.
SP3O changes only which states enter the critic loss. The actor objective, rollouts, and return targets stay the same. The default anchors sit at relative positions 0.3, 0.6, and 0.9, plus 0.95 when the response is at least 6144 tokens, so the tail is covered. The critic still emits a value at every generation step; only the training signal is sparse and spaced.
Qwen3-4B-Base and Qwen3-8B-Base trained on DAPO-Math-17k. Math scores are avg@32, out-of-distribution scores avg@4.
| Model | Method | Math avg | OOD avg |
| Qwen3-4B-Base | PPO | 37.60 | 51.95 |
| Qwen3-4B-Base | GRPO | 39.26 | 56.44 |
| Qwen3-4B-Base | SP3O | 45.57 | 59.28 |
| Qwen3-8B-Base | PPO | 48.50 | 64.38 |
| Qwen3-8B-Base | GRPO | 47.91 | 64.91 |
| Qwen3-8B-Base | SP3O | 50.51 | 66.37 |
Against PPO, the 4B model gains 7.97 math points and 7.33 OOD points. The 8B gains are smaller but still positive. On AIME26 the 4B moves from 14.69 (PPO) to 22.08; on ZebraLogic, from 9.90 to 19.20. Versus own-policy MC profiles, SP3O cuts MSE by 36%, 11%, and 21% at 30%, 60%, and 90% progress. Response-level discrimination does not drop, within-response variation rises, and the median effective rank of hidden states goes from 4.33 to 5.63. Online actor updates are smaller; responses are longer.
Ablations: K=3 is best, K=4 and 8 still beat dense PPO, K=16 and 64 fall back toward the dense baseline. At K=3, random anchors score 36.59, below PPO's 37.60; fixed 0.3/0.6/0.9 scores 45.57. Removing the tail anchor drops accuracy from 45.57 to 44.10 and raises repetition from 1.12 to 18.33.
A common reaction to unstable critics is to drop them and run GRPO. The mechanism here is more specific: regressing every token to one terminal return actively flattens within-response values. Supervising a few spaced anchors is close to a one-line sampling change, and on 4B it beats both PPO and GRPO in and out of domain. If a stack still trains a value head, this is a cheap first experiment.
The 8B gains over GRPO are about 2.6 math points and 1.5 OOD points. There are no numbers above 8B.
There is no standalone limitations section. Experiments cover only Qwen3-Base at 4B and 8B, one math corpus, binary terminal rewards, and a zero KL coefficient. The variance-penalty decomposition assumes a shared target at every position; process rewards or γ, λ other than 1 are untested. VinePPO's extra continuations appear only as a diagnostic, not as a training-cost baseline. FrozenLake shows the pattern worsening with state-space size, which does not by itself prove the same scaling for long chain-of-thought. Run variance on the K sweep is described as non-negligible, so the best K may move with model and task.