VeriHarness Paper: Disagreement Resolver Plus Consensus Challenger Lifts Agentic Verification by 6.4 Points
mikeflache · x · 2026-10-05
Researchers at Google (Caiqi Zhang, Rujun Han, et al.) released VeriHarness on arXiv, a framework for strengthening LLM-agent verification on long-horizon tasks with a fixed base model and no reference answers or rubrics at test time.
Key findings and method:
- Central observation: across repeated samples, disagreement often exposes correct alternatives while consensus can conceal errors — so consensus is treated as a potential vector for hidden errors, not default truth.
- VeriHarness turns the base LLM into an agentic verifier with a workspace, evidence tools, and reusable verification skills, split into two lanes: a disagreement resolver that checks competing claims against environmental evidence, and a consensus challenger that probes shared claims and searches for omitted requirements.
- Their findings guide selection and revision of the final artifact.
Results: Across five long-horizon workspace benchmarks and two frontier models, VeriHarness achieves the highest selection scores among baselines; evidence-backed revision adds average gains of 6.2 points with Gemini 3.5 Flash and 6.4 points with Claude Opus 4.8 over a single rollout.
Related event: Google's VeriHarness: Disagreement May Beat Consensus in AI Evaluation(2 posts)→
More from coding & agent
- Ethan Mollick wants an email system built for humans and AI agents to share — emollick · 2026-10-05
- Dev says image gen is the only thing keeping him on Codex over Claude — CtrlAltDwayne · 2026-10-05
- SwiftUI Gesture Debug Panel Tested Against Open-Source App Ice Cube — Dimillian · 2026-10-05
- Qwen Flash Next Q4 hits 17.5 tks on Mac Mini M5 with carousel expert streaming and dual-SSD tricks — turtleninja99 · 2026-10-05
- xAI ships Grok Bots marketing guides: six specialist bots run the full launch loop — alexcovo_eth · 2026-10-05
- Agents can test your code in ways that took weeks to build before — and catch human-introduced bugs — remilouf · 2026-10-05