Alibaba and Cornell Paper: Independent Critic Boosts Coding Agents by up to 9.5 Points
dair_ai · x · 2026-10-27
A paper from Alibaba and Cornell, "Rethinking Verification in Agent Harnesses," systematically studies verification design for coding agents — essential reading for harness builders.
Setup
- 17 verification designs compared across 7 frontier coding models on Terminal-Bench and SWE-bench Verified, plus a new VisualBench for visual/interactive artifacts
- Four intervention points along the agent lifecycle: initial instructions, execution-time reminders, completion gates, and an independent critic at the end
Key findings
- Instructions and reminders: small, model-dependent gains
- Completion gates: larger gains but potentially much higher cost
- Independent critic offers the best accuracy-cost tradeoff: +5.9–9.5 points on Terminal-Bench at 1.13–1.25x cost; +3.9–6.6 on SWE-bench Verified at 1.00–1.04x cost
- On VisualBench, four designs exceed baseline by 13.2–23.8 points, surpassing gaps between evaluated Lo-L2 models
- Additional review rounds add up to 3.5 more points on Terminal-Bench
Conclusion: independent review is a practical axis of test-time scaling for agent harnesses.
More from coding & agent
- Anthropic Reveals 12 New Prompting Rules for Claude Opus 5.5 — JeremyCMorgan · 2026-10-28
- TraceTrial-Bench separates agent success from harness reliability: 31.1% of runs mislead — _akhaliq · 2026-10-28
- NVIDIA open-sources a LoRA post-training workflow to localize Nemotron with synthetic personas — NVIDIAAI · 2026-10-28
- Hobbyist builds 'Sandman' bot fleet dashboard to gamify context management — alexcovo_eth · 2026-10-28
- Apple's iPrefBench: mobile agents execute well (91.6%) but personalize poorly (47.7%) — apple · 2026-10-28
- Alibaba Tongyi MAI releases MAI Gym, a 9B open GUI agent hitting SOTA under 10B — AdinaYakup · 2026-10-28