Le Critique: Privileged Value Functions for LLM Reinforcement Learning
Siddarth Venkatraman, Matthieu Dinot, Laurence Aitchison
cs.LG
2026-08-17
Conditioning the critic on oracle answers or leave-one-out group responses lifts Qwen3-4B RL above the GRPO mean baseline on all four reasoning tasks, with explained variance on Sudoku rising from 0.03 to 0.75.
Since DeepSeek-R1, the default in LLM reinforcement learning is critic-free: GRPO samples a group of responses per prompt and centers advantages on the group mean. Learned value functions were largely abandoned, on the grounds that they need extra infrastructure and a poorly fit critic can hurt the policy. But group-relative methods pay two structural costs. Advantages are sequence-level: every token in a response shares one number, so there is no token-level credit assignment. And large groups amplify the straggler effect: training waits for the slowest rollout in each group, which throttles throughput and worsens off-policyness. A value function solves both in principle. This paper, from Mistral AI with Mila and Université de Montréal, asks how to make the critic's payoff justify its cost.
Two complementary strategies.
Privileged Value Functions (PVF). RL training usually holds information the policy cannot see but that would sharpen return prediction enormously: oracle answers for math problems, gold patches for code repair, verifier rubrics, or the other K−1 responses in the rollout group together with their rewards. PVF routes all of it into the critic. The critic conditions on both the policy's token history and this privileged context; the policy never sees any of it.
The legality hinge is the baseline admissibility condition: a baseline stays unbiased as long as its extra information is conditionally independent of the current token given history. A fixed reference answer passes. Future tokens, realized rewards, and feedback generated by the current response do not. The critic therefore cannot peek at how this particular response turns out, but may freely look at its siblings. Privileged information changes only the gradient variance, never the policy objective. That is the sharpest contrast with self-distillation, which introduces a new distribution-matching objective and shifts the optimal policy, requiring careful control over what the teacher sees.
The privileged signal is picked per task: ground-truth answers on Reasoning Gym; on CodeIO, which has no reference solution, the critic gets the other three group responses with their returns, reducing value prediction to an in-context aggregation task; on Sudoku, the complete solved grid, since judging whether an intermediate move fits a globally solvable puzzle otherwise requires implicitly solving the puzzle inside the critic.
TETHER. When the critic is poorly fit, a value baseline can be worse than the group mean. TETHER interpolates: baseline = (1−ρ)·leave-one-out group mean + ρ·token-level value. ρ is fit each batch by least squares against which mixture best predicts observed returns, then EMA-smoothed. One detail carries the unbiasedness: the current batch's advantages use the previous batch's ρ, since fitting on a batch's own returns would make its baseline depend on them. ρ=0 recovers the group baseline, ρ=1 the pure value baseline, and no task-specific tuning is needed.
Qwen3-4B-Instruct-2507 (Qwen3.5-4B for MiniF2F), mean reward over the final 50 steps:
| Task | Group-mean GRPO | Plain VF | PVF |
| Reasoning Gym (K=1) | n/a | 0.528 | 0.559 |
| Reasoning Gym (K=8) | 0.556 | 0.569 | 0.574 |
| CodeIO (K=4) | 0.518 | 0.517 | 0.542 |
| Sudoku (K=4) | 0.271 | 0.220 | 0.306 |
PVF is the best method in all four settings. The Sudoku gain is the largest (0.271→0.306, while plain VF drops to 0.220), which the authors attribute to the long multi-turn horizon. Explained variance beats plain VF in every environment; on Sudoku it rises from 0.032 to 0.751, and the variance gap tracks the final reward gap task by task.
For TETHER (adding MiniF2F): Reasoning Gym 0.581, CodeIO 0.534, MiniF2F 0.324, all above the mean baseline; Sudoku lands at 0.255, short of the mean baseline's 0.271 but well above plain VF's 0.220. The fitted ρ does move off zero during training.
This is a direct counterattack on "LLM RL doesn't need a critic", and a well-aimed one: instead of fighting GRPO on infrastructure, it exploits a capability only a critic has, namely consuming information the policy cannot see. For practitioners: on tasks with reference answers or verifiers, PVF is close to a free upgrade; for teams with an existing GRPO pipeline, TETHER is the lowest-risk entry point into value functions, since it degrades toward the mean baseline when the critic is weak. The value-training infrastructure is open-sourced as prime-values, built on PRIME-RL. The incremental-improvement caveat stands: everything here is at 4B scale, well short of frontier post-training.
The authors list three: value inference and training require dedicated GPU allocation (VF/PVF runs use one to two extra nodes over the mean baseline), experiments match inference trajectory counts rather than total compute, and everything is limited to 4B models with responses up to 32,000 tokens. Beyond the self-report: group sizes only cover K∈{4,8}, and the claim that larger groups would not preferentially help the mean baseline rests on argument, not experiment. The appendix shows that moving λGAE from 1 to 0.999888 noticeably changes VF and PVF performance, while all main experiments pin λ=1 ; that sensitivity suggests the tuning burden of the value route is not actually solved. And TETHER fails to fully recover the mean baseline on Sudoku, so adaptive interpolation is not a guaranteed floor.