Trust the Critic More
Kaiyue Wen, Luke Bailey, Arvind Mahankali, Tengyu Ma
cs.LG, cs.AI
2026-09-30
Stanford's AC2 scores 10k-token action chunks with a learned critic instead of waiting for terminal rewards, beating GRPO's peak with 2.5x fewer decoding FLOPs on Qwen3-4B.
Mainstream RLVR methods like GRPO treat an entire rollout as one action. A proof runs to 50k tokens, the verifier returns a single score, and every token gets the same advantage. A trajectory that reasons correctly until a final arithmetic slip is penalized as a whole. Beyond coarse credit assignment, the bigger cost is compute: no reward exists until termination, and in long-form reasoning nearly all training FLOPs go to decoding.
Actor-critic is the textbook fix: learn a value function that scores intermediate states. The LLM RL community has mostly rejected this on the assumption that learned critics are too inaccurate, and recent methods that train a critic at all use it only as a variance-reducing baseline. This Stanford paper asks the question directly: how far can a critic actually be trusted?
AC2 (Actor-Critic with Action Chunking) turns "trust the critic" into a per-problem switch. Each step:
Three choices make the scores reliable:
A quarter of ready problems keep running to completion (auditing), which keeps critic error measurable and supplies real-reward targets for critic training.
Qwen3-4B-Thinking-2507 trained on FineProofs-RL (5,200 olympiad proof problems), evaluated on IMO-ProofBench (60 problems, scored 0 to 7 by DeepSeek-V4-Flash):
| Method | Peak score | Decoding FLOPs to reach GRPO's peak |
| GRPO | 18.50% | 1.99e20 |
| AC2 | 20.57% | 0.79e20 |
The 2.5x gap has two sources: 25% fewer steps (90 vs 120) and fewer tokens per step, since trajectories no longer need to finish. A control, Prefix GRPO (same replayed prefixes, full rollouts, terminal rewards), falls well behind AC2, so the prefix state distribution alone does not explain the gains. The global readiness threshold is crossed at the end of step 7; by step 200 roughly 70% of sampled problems run on critic scoring.
Ablations separate what matters. Without local readiness: 13.10% at step 40 versus 15.33% with it. Chunks shrunk to 2k tokens peak at 16.62% and fall to 13.57% by step 120. A buffer holding only high-scoring trajectories leads at step 60 (17.35% vs 16.88%) then collapses to 14.67% by step 110. Dropping auditing (18.66% peak), a stale replay buffer (17.90%), and group size 1 all still work.
Diagnostics at step 80: the group-mean value tracks true mean reward with MAE 0.065 versus 0.211 for single prefix predictions, and critic-based advantages correlate with terminal-reward advantages at only 0.388. Not precise, but directionally enough to learn from.
Decoding is the dominant cost of long-reasoning RL, and AC2 shows that policy updates do not require every trajectory to finish. The 2.5x decoding saving maps directly to GPU hours. The larger payoff is the design space unlocked: with no terminal-reward dependency, stale prefixes can be reused off-policy, and the authors sketch beam search or MCTS over action chunks as follow-ons, none of which fits current RLVR frameworks. Nothing in the method is specific to proofs.
The authors' own stated limit: local readiness requires epoching the training data, because complete rollouts are needed to measure per-problem critic error. In the large-scale single-epoch regime the mechanism does not apply, and adapting it is left as future work.
Other concerns: a single model (Qwen3-4B) and a single dataset; evaluation rests on an LLM judge whose own noise is not quantified; the peak comparison comes from 60 problems with no variance reported; and the 2.5x figure counts decoding FLOPs for rollouts, while the critic's own forward passes per continuation are not broken out separately.