Paper Finds Four Silent Bugs in Agent Leaderboards: 13-Point Gap Was a Tie
akorecebov · reddit · 2026-11-10
The paper "Silent Instrument Errors Reorder Agent Leaderboards" (arXiv 2611.04399) audited only public run artifacts from HAL and tau2-bench and found four bugs that never threw an error yet reorder leaderboards:
- Cost columns double-counted parent/child trace spans — exactly 2× cost in 70 of 134 runs, reshaping cost-vs-accuracy charts.
- One agent ended chats in ways that never triggered the grader; ungraded runs were stored as failures. Grading them turns a 13-point gap into a tie.
- Archived trajectories logged the grader's replay actions as agent actions — 68% of airline records.
- Re-grading tau2 with a newer evaluator moved scores -7 to +2.5, and one telecom number by 82 points, silently.
Cheap fixes: report graded fraction, sum cost from root spans only, tag who wrote each logged action, pin evaluator versions and publish diffs. The poster admits they never checked whether their own eval checker actually ran on every item.
More from coding & agent
- Re-running an old castle prompt: Fable 5.2's procedural Blender output is a visible jump — aitrendz_xyz · 2026-11-10
- Fable 5.2 plus a few Opus subagents produces a complex 3D scene in ~30 minutes — aitrendz_xyz · 2026-11-10
- Leetcode interviews will die as agentic coding rounds become the new hiring standard — rachittshah · 2026-11-10
- 10 open-source GitHub repos for voice AI: from local ASR to zero-shot voice cloning — Shruti_0810 · 2026-11-10
- As Arc sunsets, a dev builds Earful, an AI tool that aggregates product reviews into verdicts — BLCNYY · 2026-11-10
- Claude's Chrome Extension Slammed for Being Unable to Control Already-Open Tabs — Angaisb_ · 2026-11-10