Cornell & Alibaba paper: independent critic review gives best accuracy-cost tradeoff for agent verification
Zhengxin Zhang · hf · 2026-10-28
The paper Rethinking Verification in Agent Harnesses (Cornell + Alibaba, Oct 25, 2026) studies verification as an axis of test-time scaling.
- Framework spans four lifecycle layers: initial instructions, mid-turn reminders, completion gates, and independent critic review.
- Compares 17 designs across 7 frontier coding models on Terminal-Bench and SWE-bench Verified, and introduces VisualBench for evaluating visual/interactive artifacts.
- Findings: initial instructions and mid-turn reminders yield modest, model-dependent gains; completion gates give larger gains but cost substantially more; independent critic review offers the best accuracy–cost tradeoff — +5.9–9.5 points on TB at 1.13–1.25× baseline cost, +3.9–6.6 on SWE at 1.00–1.04×, and +13.2–23.8 net preference points on VisualBench.
Directly useful for designing test-time scaling in agent harnesses.
Related event: Alibaba-Cornell Paper: Independent Critics Boost Coding Agent Accuracy(3 posts)→
More from coding & agent
- Hook-level agent security wins: Microsoft, Anthropic and OpenAI all shifted from gateways to runtime hooks — jacob_posel · 2026-10-28
- Supabase launches Compute in private alpha: run AI agents and long-running jobs next to your Postgres data — dshukertjr · 2026-10-28
- Zero-AI-at-runtime design generator hits run 8, new aesthetics expected by run 10 — Afinetheorem · 2026-10-28
- Gunning for 60fps on a 200MHz CPU: dev shares Dreamcast optimization challenge — AIandDesign · 2026-10-28
- EnvLunch turns executable environments into free verifiable agent training data — KingsCollegeLondon · 2026-10-28
- Is agent recovery an underrated problem? Devs debate checkpointing and idempotency — CuteClass9864 · 2026-10-28