VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents
Jianing Yin, Ziyi Chen, Hao Wang, Yiyou Sun, Zijian Jin, Dawn Song, Daoan Zhang
cs.SE, cs.AI
2026-10-20
On 222 tasks and 4,993 hidden cases, 19 models score 10.6 to 62.1 normalized balanced accuracy. Verifiers reject invalid solutions more reliably than they accept valid ones.
A solver is finished once it finds one working answer. A verifier has to accept every valid way of completing the task and reject mistakes, shortcuts, and attacks. In data-oriented recursive self-improvement, the agent builds its own training data, and the verifier decides which trajectories get a reward and which enter the next training set. A wrong verifier pays the same violation out again and again.
Agent tasks also depend on tool calls, the final environment state, and code changes. Matching one reference solution cannot show whether the verifier killed another completion the task allows. Those discarded strategies then never reach the training set.
VerifierBench draws tasks from six open-source agent benchmarks. Screening starts at 267 tasks. Verifier4Verifier (V4V), a six-check audit, drops 45 and keeps 222 tasks with 4,993 hidden cases. Tool-call accounts for 95 tasks (BFCL v4, ACEBench), state for 80 (τ2-bench, AppWorld, AgentDojo), and code for 47 (SWE-bench Verified). All 37 Terminal-Bench tasks are removed. On 35 of them the test-output parser counts skipped tests as passes.
Cases form four groups. The reference solution (222) and valid variants (1,344) are labeled PASS. Variants split into cosmetic edits (840) and alternative implementations or paths (504). Ordinary errors (2,629) and attacks (798) are labeled FAIL. Positives are 1,566 cases, 31.4% of the suite.
The generator sees only the instruction and environment a solver would see. The reference verifier and the hidden cases stay withheld. Self-tests on solutions the agent writes are its only evidence. After submission the verifier is frozen and run on every case. State and code tasks may call runpublic to replay actions or run tests. Generation and grading use separate execution budgets.
The headline metric is BAnorm = TPR + TNR - 100. TPR is the task-averaged rate at which positive cases are accepted. TNR is the task-averaged rate at which negative cases are rejected. Both rates are computed inside each task, then averaged across tasks. A verifier that always emits a valid decision scores 0. Perfect agreement scores 100. Missing verifiers, timeouts, and crashes stay in the denominator. Attack false-positive rate, FPRh, is reported separately.
V4V checks replay against construction labels, repeatability, provenance, consistent labels on identical inputs, exposure of trivial verifiers, and mutation detection. Trivial means always accept, always reject, or accept only the reference solution. All 1,410 mutation configurations are detected. When a construction label and the source verifier disagree, the task requirements win. All 222 retained tasks pass the six checks, replay agrees, and the reference label is the source verifier's decision.
Nineteen models run in one harness, Pi. Each task gets 5,400 seconds and each reply at most 32,768 output tokens. Every model is scored on all 222 tasks.
| Model | BAnorm | TPR | TNR | Attack FPR |
| GPT-6-astra | 62.1 | 67.4 | 94.6 | 7.4 |
| Claude Opus 4.8 | 54.0 | 67.7 | 86.3 | 13.5 |
| Kimi K3 | 53.4 | 70.2 | 83.2 | 13.4 |
| GPT-5.6-sol | 52.9 | 56.8 | 96.0 | 0.8 |
| Doubao Seed 2.0 Mini | 10.6 | 21.1 | 89.5 | 3.7 |
Scores run from 10.6 to 62.1. Every model has a lower TPR than TNR: TPR spans 21.1% to 70.2%, TNR 79.2% to 96.3%. Of positive cases that do not pass, 57.1% to 100% are explicit FAILs. The models are rejecting valid solutions.
Averaged across models, TPR is 79.2% on tool-call tasks, 27.7% on state tasks, and 40.2% on code tasks. On state tasks no model's TPR reaches 50%. The leader changes with task type: GPT-5.6-sol leads tool-call at 85.2, Claude Opus 4.8 leads state at 34.6, and Kimi K3 leads code at 76.9. A low attack acceptance rate is not reliability. On state tasks GPT-5.6-sol has an FPRh of 0.0% while accepting only 23.0% of positives, for a BAnorm of 16.1. On code tasks 17 models show an FPRh of 0.0% at the reported precision, and their BAnorm still runs from -25.9 to 76.9.
All-reject verifiers return a valid FAIL on every case of a task. They add 0 to BAnorm there, yet they produce 87.9% to 96.0% of positive false rejections in the archived records.
The harness moves the ranking. The best of 36 configurations is Claude Opus 4.8 with Claude Code at 66.2, 12.2 above its Pi score of 54.0. Gemini CLI raises Gemini 3.5 Flash by 15.0 and lowers Gemini 2.5 Pro by 9.2. Codex lowers four GPT models by 4.1 to 26.6. GPT-5.6-sol falls from 52.9 to 26.2. No harness improves every model it runs. Raising reasoning effort barely moves the total: for GPT-6-astra on Pi, HIGH versus LOW changes BAnorm by +0.5, up on code and down on state. Higher effort burns the budget. For that same model, among 127 state and code tasks, runs that exhaust 20 runpublic calls rise from 4 to 77 on Pi and from 21 to 98 on Claude Code.
A targeted read of 16 verifiers across 12 tasks covers 119 positive FAILs and 90 distinct cases. On a banking task, the tool allowlist omits a permitted read-only query, all 9 positives fail before any environment execution, and the self-test never calls that query. A Django verifier rejects any patch that edits init.py. All four positives edit that file. The self-test only edits query.py, and it passes.
When a verifier filters synthetic training data, being too strict is the quiet failure. An all-reject verifier can post a zero attack-acceptance rate and still discard every strategy for the task. BAnorm penalizes that constant output. FPRh alone does not.
Confirmatory self-tests only try solutions the extra check already allows, so more tests at the same spot change nothing. A correctly rejected negative does not justify the check either: an invalid patch can trip a real requirement and the extra rule at once. Other runs finish without a single complete positive PASS and submit anyway. Eight verifiers in that group reject all 72 of their positive judgments at evaluation time, without executing the environment.
The suggested fix is to tie every check to the task requirements and to probe it with a permitted variant. That remedy is not measured.
State-task verifiers are not ready to serve as rewards. The best TPR there stays under 50%, and the best BAnorm is 34.6.
Scores measure agreement with retained reference verifiers on a finite, constructed suite. Dropping 45 tasks does not certify the remaining labels. Agreement between a construction label and the reference decision is not semantic proof. Rankings are single-run point estimates, with no repeated generations and no significance tests. The 36 configurations tie prompts, tools, and reasoning settings together, so a harness gap cannot be assigned to one component.
The concentration numbers use archived per-task records for nine models plus a trace batch for GPT-6-astra, not the final per-task records of all 19. The code and trace review selects verifiers that explain failures already seen, mostly from one model. Attribution comes from reading source, not from intervention experiments, and the traces do not reconstruct the verifier version at each self-test. The mechanisms are identified. Their prevalence is not.
Positives are only 31.4% of cases, and process-violation attacks number 2. A wall of 0.0% attack acceptance on code tasks can mean the attacks are weak, or that all-reject verifiers fail the attacks for free. Once those verifiers pin a task's TNR at 100, part of the impression that rejection is the strong side is the scoring setup.