VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks
Caiqi Zhang, Rujun Han, Zifeng Wang, Zoey CuiZhu, Nigel Collier, Tomas Pfister, Chen-Yu Lee
cs.AI, cs.MA
2026-10-01
VeriHarness makes the generator's own model an agentic verifier with workspace tools. Revision lifts five benchmarks 6.2 points on Gemini 3.5 Flash and 6.4 on Claude Opus 4.8.
Long-horizon agents now ship reports, spreadsheets, and patches. Those artifacts do not come with unit tests or proof checkers. At the frontier the generator is already the strongest model on hand, so test-time verification has to run on that same model, with the same tools, and with no reference answers or grading rubrics.
Repeated sampling produces a pool of rollouts (full attempts) that often contain complementary correct claims. Majority vote treats the most frequent answer as true. An LLM-as-a-judge only reads the rollouts and never touches source files or constraints in the workspace.
On ten-rollout pools of Claude Opus 4.8 on APEX-Agents, 34% of 917 consensus claims are graded wrong. Among 775 disputed claims, 74% contain a correct candidate, while the most frequent value is right only 47% of the time. Consensus can lock in a shared error. Disagreement often holds the right alternative that voting would discard.
VeriHarness is a training-free, plug-and-play verification harness: a workspace, evidence tools, a checking protocol, and a skill library wrapped around a frozen model. Generator and verifier share that model. Each rollout's artifact, final state, and trace sit in the workspace as files, next to the task environment. The protocol does not depend on artifact type.
Two investigations run in separate contexts. The disagreement resolver finds claims where rollouts conflict and drops values the environment contradicts. It picks checks by a qualitative information-gain rule. If evidence kills every candidate, it writes a new value only when the evidence establishes one; otherwise the claim stays open. The consensus challenger has no rival value to test. It first proposes how a shared claim could fail, then looks for counter-evidence: recompute totals, check a shared reading against the named source, and scan the task description for requirements every rollout omitted. Omissions have to be turned into check targets, because they appear in no artifact.
A skill is a short text, optionally with a script, that names a reusable failure mode and how to check it. Tools run checks. Skills say what is worth checking. A fresh context adjudicates both evidence records, picks the base rollout with the fewest evidence-backed defects, and writes a revision plan. Merging the two investigations into one context costs 0.8 points.
A worked case: two rollouts cite a draft and report FY2025 revenue as USD 100m; a third cites the final report at 120m; all three mark the figure USD. The resolver keeps 120m after checking version history. The challenger finds the currency should be EUR. Adjudication merges both edits.
Five workspace benchmarks, N=10 frozen rollouts per task, same model as generator and verifier. Benchmark graders never reach the verifier.
On Gemini 3.5 Flash the single-rollout average is 47.2. Majority vote reaches 47.5, best-of-N judging 47.3, pairwise tournament 49.1, LLM-as-a-Verifier 49.5. An agentic verifier with the same workspace and tools but no protocol or skills scores 49.4, about half the selection gain. VeriHarness selection reaches 51.6 (+4.4). Evidence-backed revision reaches 53.4 (+6.2). On Claude Opus 4.8 the path is 49.6 to 53.7 (+4.1) to 56.1 (+6.4).
| Method | Avg | APEX | WSB | WorkBuddy | SB-2 | JobBench |
| Flash single | 47.2 | 48.1 | 56.5 | 73.2 | 27.3 | 30.9 |
| Flash + revision | 53.4 | 54.8 | 65.4 | 78.2 | 32.2 | 36.2 |
| Opus single | 49.6 | 35.8 | 60.5 | 79.0 | 30.1 | 42.8 |
| Opus + revision | 56.1 | 47.5 | 67.3 | 83.7 | 34.3 | 47.6 |
The largest revision jump is Opus on APEX-Agents, 35.8 to 47.5 (+11.7). Aggregating the pool into a new artifact without evidence checks stays at 49.9 on Flash and 52.3 on Opus, below the full harness. The selection oracle sits at 62.7 and 63.1. Inside Gemini CLI, Claude Code, or Codex, Flash stays within two points of the dedicated implementation on every benchmark; Opus falls back to the single-rollout level on JobBench.
Skills can grow from failure feedback with the model, protocol, and tools held fixed. A library evolved from empty improves held-out scores by 11.0 points on APEX-Agents (38.6 to 49.6) and 5.7 on SpreadsheetBench 2 (32.8 to 38.5), beating the frozen human library (45.3 and 35.7). Evolving from the human library adds another 6.8 and 3.7.
Ablations on Opus: resolver alone 54.3, challenger alone 52.6, both 56.1, from 49.6. Selection gains sit almost entirely in pools that disagree: +0.6 over the pool mean on 154 unanimous APEX-Agents pools, +6.1 on 262 disputed ones. The challenger never refuted a claim every rollout had right. About 70% of the shared errors it left standing were cases where it checked a correct intermediate result and the error sat one step later.
This is test-time verification with the same model and tools, no stronger judge, and no fine-tuning. For anyone already running long-horizon workspace agents, it is a verification layer that can sit on existing CLIs.
The gain is incremental. Averages move +6.2 and +6.4, while the selection ceiling is still 62 to 63. Majority vote barely moves. Extra points come from checking the workspace, and from a protocol plus skills that specify what to check. Environment access without that protocol recovers about half the selection gain.
The authors release about 26,000 rollouts across five benchmarks and two models, produced at over $100,000. Later work can compare selection and revision on this pool without paying the generation bill again. The verification record ties each claim to a check, evidence, and verdict, and they mark it as a possible claim-level training signal for the generator.
The paper lists four limits: a ten-rollout generation budget, wall-clock cost from multi-turn investigation, no scalar score per rollout, and a same-model study that cannot separate harness gain from a stronger judge. The intended use is reports, workbooks, and patches, where quality outranks latency.
Selection gain on unanimous pools is almost zero. The blind spot on shared errors needs prior knowledge in the skill library, and skill evolution uses grader feedback on a development set, so that stage is not unsupervised. Held-out scores in the evolution study are not the full Table 2 splits. When Opus runs inside existing CLIs, JobBench falls back to the single-rollout score: the protocol can move, the gain may not. The challenger never false-rejected a unanimously correct claim. That is safety. It is also conservatism.