VerifierBench: 222 Tasks Expose How Unreliable Agent-Generated Verifiers Are
Jianing Yin · hf · 2026-10-23
A team from UPenn, UIUC, UC Berkeley, NYU and others releases VerifierBench, a benchmark studying whether LLM agents can write reliable executable verifiers.
- Setup: given an instruction and environment, agents write a verifier that must accept every valid solution and reject invalid ones — spanning tool-call, state-based, and code verification across 222 tasks with 4,993 hidden test cases including alternative valid solutions.
- Key finding: agent-generated verifiers frequently wrongly reject valid solutions due to hidden assumptions; the paper recommends testing that verifiers accept diverse correct answers.
- Why it matters: broken verifiers can repeatedly reward violations or discard valid solutions in RL, making verifier reliability critical to agent training.
More from Research
- Sophont's Holon-S1 predicts genes and biomarkers from routine pathology slides — iScienceLuvr · 2026-10-23
- Glasgow study: raised eyebrows and slow blinks make speakers look confident — Limor_Raviv · 2026-10-23
- Microsoft's step-by-step AI diagnosis research lands Science cover — erichorvitz · 2026-10-23
- IFM open-sources RL360, the agentic RL infra behind K2 Horizon's training — rupspace · 2026-10-23
- Wayfarer discovers RL options from a single experience stream, SOTA on hard Atari games — MarlosCMachado · 2026-10-23
- PhysVidBench accepted at EMNLP 2026: do video gen models grasp physical commonsense? — aykuterdemml · 2026-10-23