Agentic Benchmark Checklist paper shows flawed agent benchmarks skew results by up to 100%
ddkang · x · 2026-09-19
A 26-author paper from Berkeley/Stanford-adjacent researchers (Percy Liang, Matei Zaharia, Ion Stoica, Daniel Kang among them) shows many agentic benchmarks suffer from flawed task setup and reward design:
- SWE-bench Verified uses insufficient test cases; TAU-bench counts empty responses as successes — such issues can under- or overestimate agent performance by up to 100% in relative terms
- The authors introduce the Agentic Benchmark Checklist (ABC), guidelines synthesized from benchmark-building experience, best-practice surveys, and previously reported issues
- Applying ABC to CVE-Bench, a benchmark with a particularly complex evaluation design, reduces performance overestimation by 33%
The 39-page paper offers an actionable standard for rigorous agent evaluation.
More from coding & agent
- Butterfly traces multi-agent failures with graphs, 62.5% accuracy vs 53.8% for log prompting — JeremyCMorgan · 2026-11-04
- Free course covers the 4 hard problems of production generative media agents — Al_Grigor · 2026-11-04
- Quota reset night: Codex back to 100% while Claude shows just 10% left — rudrank · 2026-11-04
- Self-improving software is inevitable: closing the agent-observability loop — bibryam · 2026-11-04
- Building a 7-GPU 132GB VRAM local inference rig: consolidate on EPYC or split across two systems? — infectiousstupidity · 2026-11-04
- Founder of 50-person, 12-country team says he talks to AI agents more than humans — Scobleizer · 2026-11-04