Mistral paper: Privileged Value Functions give LLM RL critics hidden context for stronger signal
jm_alexia · x · 2026-08-19
Mistral AI, together with Mila and Université de Montréal, released Le Critique: Privileged Value Functions for LLM Reinforcement Learning, arguing value functions have been grossly underutilized in LLM RL. Two key contributions:
- Privileged Value Functions (PVF): the critic conditions on context hidden from the policy — reference solutions, verifier rubrics, other group samples and rewards — producing a more informative token-level baseline that reduces policy-gradient variance without exposing those clues to the policy.
- TETHER: instead of trading off group and value baselines, it adaptively interpolates between the leave-one-out group mean and the learned value baseline based on how reliable the value function is.
Paper and code are open-sourced.
More from Research
- WorldCrafter: Video World Model with Implicit 3D-Aware Memory Opensourced — _akhaliq · 2026-09-22
- SaaS sales conversations dataset with 100K+ English dialogues trends on Hugging Face — DeepMostInnovations · 2026-09-22
- onPanda: token-level correction tool cuts alignment data annotation time by 52% — stepfun-ai · 2026-09-22
- Randomized step sizes make Metropolis–Hastings robust to tuning, study finds — michaelchchoi · 2026-09-22
- Exact MCMC via Bernoulli factories for proposals with intractable normalizing constants — michaelchchoi · 2026-09-22
- TorchDR 0.5 lands with native multi-GPU support, beating cuML by 3.71x on a single GPU — leland_mcinnes · 2026-09-22