A Cheap Verifier is Good Enough: LLM Post-training is Robust to Erroneous Rewards
Andreas Plesner, Curtis Northcutt, Francisco Guzmán, Anish Athalye
cs.LG, cs.AI, cs.CL
2026-09-27
11k H100-hours of Qwen3 GRPO: Gemma 4 26B trails the best verifier by 1 point at 98.8% lower grading cost; the cheapest 80% pick trails by 3 points on average.
Once RL post-training leaves math and code, the reward is no longer a unit test. An LLM ticks rubric items that mix correctness, tone, and safety into a weighted scalar. The working assumption is simple: a more accurate grader should produce a stronger policy. The paper's classroom analogy is a TA who marks 10% of homework wrong versus one who marks 30% wrong. Students of the better TA should learn more.
On semi-verifiable tasks there is no answer key. Accuracy here means agreement with a more expensive frontier model used as a golden verifier. Handshake AI and ETH Zurich spent about 11.3k H100 GPU-hours on two questions: whether that agreement ranks training verifiers, and whether expensive judges actually train better policies.
Training is GRPO with 8 rollouts per prompt, advantages standardized by the within-prompt group standard deviation, 120 prompts per batch, constant learning rate 5e-6, and no effective KL penalty. Trainees are Qwen3 1.7B, 4B, and 8B; PRBench uses 8B only. HealthBench scores are a signed weighted sum of binary verdicts, normalized by total positive weight, so a perfect response scores 1.
Verifiers sit in three layers. The oracle is unattainable. Golden verifiers are Claude Opus 4.6, Gemini 3.1 Pro, and GPT-5.5, one criterion per call at thinking=medium. Training-time verifiers grade up to three criteria per call at thinking=low. A screen of 52 models on HealthBench-hard found agreement accuracy from 50.9% to 95.5% against Opus. Twelve went into full post-training, including Gemma 4, GPT oss, Grok 4.20, a Qwen3 self-judge, and weak LFM2 24B. Unstable APIs left some runs incomplete, so Grok 4.20 coverage is uneven.
Agreement is accuracy, precision, recall, and F1 on fixed 20- or 40-task slices, computed per golden then averaged. Policy eval uses larger disjoint sets: 100 hard plus 40 non-hard HealthBench tasks, and about 53-55 PRBench tasks per domain. Two cheap picks are scored after the fact: the cheapest candidate with accuracy at least 0.80 (cost-reducing, often GPT oss 20B), and a fixed Gemma 4 26B (balanced). The second pick used observed training scores. It is not a rule you can apply before running the sweep.
Agreement is informative and still fails to pick the winner. On HealthBench-all, Qwen3 8B scores 0.37 with GPT oss 20B and 0.46 with Grok 4.20, even though GPT oss 20B has the higher F1. Open-weight Gemma 4 26B scores 0.45. On HealthBench-hard, Gemma 4 31B (F1=0.83) and 26B (F1=0.76) both land at 0.34. Weak judges can wreck training: LFM2 24B has F1=0.54; the 8B trainee still beats its no-training baseline, while the 1.7B trainee falls below as mean response length drops from 808 tokens at step 9 to about one token by step 49. The Qwen3 8B self-judge has F1 around 0.76 and a hard score of 0.08, below the untrained 0.16.
| Training verifier | HealthBench-all 8B | F1 (all) |
| Grok 4.20 | 0.46 | 0.71 |
| Gemma 4 26B | 0.45 | 0.74 |
| GPT oss 20B | 0.37 | 0.81 |
| LFM2 24B | 0.34 | 0.53 |
| Qwen3 8B self-judge | 0.11 | 0.74 |
| No-training baseline | 0.33 | n/a |
On PRBench the scores bunch within a few points. Finance-all normalized: Gemma 4 31B at 0.46, Gemma 4 26B and most other candidates at 0.45, baseline 0.23; GPT oss 120B drops to 0.38, LFM2 to 0.34. Legal-all: Gemini 3 flash 0.43, Gemma models and GPT oss 20B around 0.41-0.42, baseline 0.30.
Across 18 cells (3 datasets x 2 splits x 3 goldens), Gemma 4 26B sits 0.010 behind the best eligible training verifier, captures 95% of available uplift, and cuts grading cost 98.8% versus the golden protocol. The accuracy-floor cheap pick sits 0.027 behind, captures 82%, and cuts 99.7%. Those averages hide a 6.2-point miss on HealthBench-all, where the cheap pick captures 46% of available uplift. The paper states the averages do not make verifiers interchangeable.
Most of the dollar gap is price, not tokens. On a HealthBench probe, Claude Opus 4.6 costs $133.20 per 1,000 tasks, Gemma 4 26B $0.45, GPT oss 20B $0.08. Golden protocols use 11.6 calls per task, training protocols 4.3. Opus uses about 3.9x the tokens of Gemma 26B and about 296x the dollars. Under a matched batch=3, thinking=low protocol the token ratio falls to 1.44x. Self-hosted Gemma 4 26B grades a HealthBench rollout in about 0.51 H100-seconds, or roughly 14 H100-hours for the 96k rollouts of a 100-step run. These figures are grading API cost, not the full training bill, and no policy was trained with a golden verifier.
Teams already running rubric RL can treat the strongest-judge default as a cost curve. Grok 4.20 was in the pool, and open Gemma 4 26B still sits next to the top. A large share of screened models cluster between 78% and 87% accuracy; F1 alone does not buy the training ranking.
This is an observational sweep, not a new algorithm. Both cheap picks are retrospective. The balanced choice locked Gemma 4 26B after seeing training scores, so the 1-point gap does not transfer to a new task without that sweep. Squeezing cost on HealthBench-all gives back 6.2 points. Weak judges and the self-judge collapse, so cheap is not automatically good.
Trainees are Qwen3 only, up to 8B, with one GRPO recipe and no KL. Weaker verifiers did not get their own hyperparameters. Unstable APIs left incomplete runs and uneven Grok 4.20 coverage. Repeats were not controlled seed replications, so gaps mix training noise with verifier effects. Goldens remain LLMs: pairwise agreement is 84.4%-90.8%, unanimous on 81.5% of criteria. Two experts reviewed 50 Opus HealthBench labels and found no errors; that sample cannot certify the goldens. Agreement is measured on frozen responses, not on each trained policy's outputs, and there is no dedicated reward-hacking test. On PRBench, mean response length grows from about 1.9k-2.4k tokens to near the 5,000 cap, with 35%-96% truncated, so endpoint scores sit under that length constraint.
The metric is also the wrong object. Unweighted criterion F1 is not the signed, group-normalized advantage GRPO actually trains on. Relative ranking of two rollouts is not what F1 measures.