VeriHarness Paper: Disagreement Resolver Plus Consensus Challenger Lifts Agentic Verification by 6.4 Points
mikeflache · x · 2026-10-05
Researchers at Google (Caiqi Zhang, Rujun Han, et al.) released VeriHarness on arXiv, a framework for strengthening LLM-agent verification on long-horizon tasks with a fixed base model and no reference answers or rubrics at test time.
Key findings and method:
- Central observation: across repeated samples, disagreement often exposes correct alternatives while consensus can conceal errors — so consensus is treated as a potential vector for hidden errors, not default truth.
- VeriHarness turns the base LLM into an agentic verifier with a workspace, evidence tools, and reusable verification skills, split into two lanes: a disagreement resolver that checks competing claims against environmental evidence, and a consensus challenger that probes shared claims and searches for omitted requirements.
- Their findings guide selection and revision of the final artifact.
Results: Across five long-horizon workspace benchmarks and two frontier models, VeriHarness achieves the highest selection scores among baselines; evidence-backed revision adds average gains of 6.2 points with Gemini 3.5 Flash and 6.4 points with Claude Opus 4.8 over a single rollout.
Related event: Google's VeriHarness Turns Consensus on Its Head for LLM Verification(3 posts)→
More from coding & agent
- What's the Actual Difference Between AI Models and AI Agents? — HarperSCarroll · 2026-11-20
- Victor Taelin's Bend2 talk from SF Lean Meetup explains the rationale behind the language design — pbaylies · 2026-11-20
- Drex Router cuts small-game build cost to $1.26 vs $13.44 on Claude Opus alone — testingcatalog · 2026-11-20
- User teaches AI companion to ignore "drop it" said to his dog, then fixes the over-pause — manosaie · 2026-11-20
- NVIDIA Dynamo cuts RL rollout bottlenecks with KV-aware routing and delta weight updates — NVIDIAAI · 2026-11-20
- LangChain Academy Answers: Can a Better Model Fix a Broken Agent? — LangChain · 2026-11-20