Beaker autonomously finds 100+ grading errors across five popular AI benchmarks
VictorSanh · x · 2026-11-20
Beaker autonomously flagged more than 100 errors in task instructions and grading across five widely used AI benchmarks, with at least one verified error in each — a finding amplified by Hugging Face CEO Clement Delangue's retweet.
Notable failures:
- Agents' Last Exam: the grader accepts 5.9 as an exact match for 5 (decimals truncated before comparison) and only scores the JSON file, ignoring whether the required dashboard exists
- AutomationBench: nine email checks for a scheduling task that never verifies the Calendly office hours were created
- Harvey LAB: expects $0 while the source spreadsheet records $8 million
- Terminal Bench Science: the same report passes or fails depending on an unstated tie-break
- Terminal Bench 2.0: asks for three report fields but defines only two
Since teams use benchmark scores to pick models, decide what to ship, and tune agents, misleading grades undermine all of those decisions. Beaker's dataset review compares instructions, grading rules, and source files, flags mismatches with cited evidence, and suggests corrections.
More from coding & agent
- Burkov spots Codex quirk: suggested follow-up knows the agent screwed up, but the agent doesn't — burkov · 2026-11-20
- QuixiInfer author: tell Claude 'defeat is not an option' to make it write kernels for your GPU — QuixiAI · 2026-11-20
- OpenAI's Codex repo has 27,802 open issues: agent harness maintenance compared — SuaveSteve · 2026-11-20
- Dev argues terminal/tmux is the wrong form factor for managing 5-10 AI agents — DuaneJRich · 2026-11-20
- Volcano launches Agent Cloud: self-healing deploy platform built for coding agents — rohanpaul_ai · 2026-11-20
- subapi open-sourced: turn your Claude subscription into a local API endpoint — Last-Transition249 · 2026-11-20