Parameter Exploration for RLVR via Variational Learning
Vatsal Venkatkrishna, Nico Daheim, Iryna Gurevych
cs.LG, cs.AI, cs.CL
2026-08-11
IVON adds Bayesian noise to the weights, sampling several policies for GRPO instead of tuning temperature. 7B models beat GRPO on math at near-equal FLOPs and produce fewer zero-advantage groups.
GRPO is the workhorse algorithm for RLVR (reinforcement learning with verifiable rewards), the recipe behind today's reasoning models. For each prompt it samples a group of G rollouts and uses the group's mean reward as the baseline: rollouts above average get reinforced, below average get suppressed. The mechanism has a built-in weak spot. Whenever every rollout in a group is fully right or fully wrong, every advantage is zero and the group contributes nothing to the gradient. Training spins its wheels. These are zero-advantage groups, and they get more frequent as the model sharpens, because a sharper model more often solves a problem every time or fails every time.
The standard exploration lever is sampling temperature: crank it up and the token distribution flattens, so you draw more varied completions. But temperature only reshapes the variance of the distribution at each position. It cannot reorder which tokens are preferred, and too much of it destabilizes training. The "different" rollouts you get are still samples of one distribution. The authors ask whether there is an exploration axis orthogonal to temperature.
Their answer is to move exploration from action space (token probabilities) into parameter space (the weights). Perturb the weights themselves, sample several distinct policies, let each trace its own sampling path, then estimate reward.
The noise is not arbitrary. They use IVON, a variational Bayesian optimizer (think a Bayesian cousin of Adam that also tracks a posterior variance per weight). Each weight is perturbed as θ̂ = m + σ⊙z with z standard normal, and σ = 1/√(λ(h+δ)) where h is IVON's diagonal Hessian estimate and λ is the single knob. Smaller λ means more noise and more exploration. The Hessian weighting matters: noise concentrates on the weights that matter most for the loss. Equal-magnitude isotropic noise does not work, as an ablation confirms.
The 3PO family (Perturbed Parameter Policy Optimization) differs in how the perturbed weights are consumed:
Two 7B models, math on AIME 2024-26, AMC, MATH-500, Minerva, code on LiveCodeBench-v6. Baselines are GRPO and GRPO augmented with action-space tricks like temperature scheduling (Polaris) and entropy regularization.
| Model | Method | Math avg Pass@1 |
| OLMo-3-1025-7B | GRPO | 42.99 |
| OLMo-3-1025-7B | C3PO | 44.04 |
| Qwen2.5-Math-7B | GRPO | 45.36 |
| Qwen2.5-Math-7B | C3PO | 46.88 |
The average gain is about 1 to 1.5 points at near-identical FLOPs, but it concentrates on the harder benchmarks. On the toughest split, Qwen's AIME 2026 jumps from 20.42 to 26.25, almost six points. On code, C3PO reaches 15.17 on LCBv6, clearly above GRPO, and matches GRPO's final reward within the first half of training. The authors' read: the harder the task and the weaker the post-pretraining model, the larger the payoff from parameter-space exploration.
The training dynamics matter as much as the scores. C3PO and M3PO rescue more zero-advantage groups than GRPO throughout training and produce fewer malformed or incorrect rollouts. That lands directly on the opening problem: widening the exploration axis breaks the all-correct-or-all-wrong deadlock and gets gradient flowing again.
C3PO is acutely sensitive to λ. At λ=10⁸ the noise is too large and training collapses to a 0.0 average; λ=10⁹ is optimal; λ=10¹⁰ drops back. An ablation also shows that replacing IVON's Hessian-scaled noise with equal-magnitude isotropic noise scores 42.12, below GRPO. What does the work is not "add noise" but "add curvature-aware noise."
RLVR is the main route from SFT to stronger reasoning, and zero-advantage groups are a recognized bottleneck on it, one that worsens as models grow smarter. This paper hands practitioners an exploration lever orthogonal to temperature that costs almost no extra compute. The value shows up most when action-space methods have exhausted temperature scheduling and training starts idling.
Be honest about the size of the win. The absolute average gain is modest, around 1 to 1.5 points, and on MATH-500 and AMC, C3PO is roughly tied with or slightly below GRPO. The real separation is on hard benchmarks like AIME. Reading it as an incremental improvement that looms largest on hard tasks and weak baselines is more accurate than a paradigm shift.
The authors list several. Only 7B models were tried, limited by compute, and they expect larger models to benefit more but do not show it. Although FLOPs are matched, M3PO and C3PO run about 1.5× slower in wall-clock time because current RL stacks assume a single model and lose parallelism, a systems limitation rather than an algorithmic one. Only a diagonal posterior was used; structured non-diagonal estimators were not tried. Test-time multi-sample scaling was left out.
Two things stand out as open questions. First, C3PO is so sensitive to λ that it collapses at the wrong setting, and the optimum is model-dependent (10⁹ for OLMo, 10¹⁰ for Qwen), so moving to a new model means sweeping λ first. Second, because the gains concentrate on hard tasks and weak baselines, there is no evidence yet that the method still wins on already-strong models and saturated benchmarks.