AI agent audits 5 top benchmarks, finds 100+ grading errors including 5.9 scored as exact match for 5
VictorSanh · x · 2026-11-21
VictorSanh's team pointed their Beaker AI agent at 5 of the industry's most-used benchmarks and it autonomously flagged 100+ likely grading errors, with at least one verified in each:
- Agents' Last Exam: full marks without building the dashboard; the grader accepts 5.9 as an exact match for 5 (decimals truncated before comparison);
- AutomationBench: task asks for weekly office hours, but all 9 checks only inspect email, never the schedule itself;
- Harvey LAB: rubric expects $0 while the source spreadsheet records $8M;
- Terminal Bench Science: an unstated tie-break flips the same report from pass to fail;
- Terminal Bench 2.0: asks for 3 report fields but defines only 2.
Teams use these scores to pick models, decide what to ship and tune agents — misleading graders put all those decisions on shaky ground.
Related event: AI Agents Audit Five Major Benchmarks, Find 100+ Scoring Errors(3 posts)→
More from Models
- Google launches first interactive Gemma 4 explainer teaching how LLMs work in the browser — DynamicWebPaige · 2026-11-21
- InferenceBench recap: Anthropic led for half a year before Google's dramatic comeback — maksym_andr · 2026-11-21
- Persimmon built a safety framework from scratch for human-like user models — gharik · 2026-11-21
- Persimmon is now open to everyone, safety filters credited for the decision — gharik · 2026-11-21
- Persimmon launches pro and ultra API tiers, ends waitlist — CShorten30 · 2026-11-21
- RCTBench: Only 3 of 9 Decision Models Hit 1,000 Guests in RollerCoaster Tycoon 2 — bo_wangbo · 2026-11-21