Cheap verifiers match costly ones in LLM post-training, saving up to 99.7% of grading cost
iScienceLuvr · x · 2026-09-29
A new arXiv paper, "A Cheap Verifier is Good Enough," spends over 11,000 H100 GPU-hours testing whether verifier accuracy predicts post-training outcomes for LLMs.
- Setup: Qwen3 trainees (1.7B–8B) trained on HealthBench and PRBench tasks across medical, legal, and finance domains, with frontier LLMs serving as "golden verifiers."
- Finding: higher verifier agreement does not consistently identify the best training verifier; expensive verifiers need not outperform cheap ones, and open-weight Gemma verifiers produce strong training outcomes.
- Cost: low-cost verification choices cut grading costs by 98.8%–99.7% versus golden protocols, with average post-training score gaps of only 1–3 points — though individual settings saw larger losses, so verifier choice is not fully interchangeable.
More from Research
- Economist John Horton: keep AI-generated papers out of human venues, publish on GitHub — soumitrashukla9 · 2026-09-29
- Harvard's Clinical Informatics Lecture Series hosts "AI and Mental Health" with Dr. John Torous — zakkohane · 2026-09-29
- HCOMP 2026 study: imagery scales skew annotator and reviewer performance — windx0303 · 2026-09-29
- EleutherAI open-sources Bergson, a scalable data attribution library for LLMs, with an EMNLP 2026 oral paper — BlancheMinerva · 2026-09-29
- Bergson data attribution library released with first public MAGIC implementation — BlancheMinerva · 2026-09-29
- EleutherAI tests whether EK-FAC data attribution can block subliminal learning — results are 'solidly mid' — BlancheMinerva · 2026-09-29