AI agent audits 5 top benchmarks, autonomously finds 100+ likely grading errors
VictorSanh · x · 2026-11-21
A team pointed an AI agent at 5 of the industry's most-used benchmarks and it autonomously found 100+ likely grading errors:
- Agents' Last Exam: full marks with no dashboard built; a 5.9 score recorded as 5
- AutomationBench: asks for weekly office hours; all 9 checks are about email
- Harvey LAB: rubric says $0, source spreadsheet says $8M
- Terminal Bench Science: an unstated tie-break rule
Hugging Face's Victor Sanh replied that building datasets is hard and offered to help fix them.
Related event: AI Agents Audit Five Major Benchmarks, Find 100+ Scoring Errors(3 posts)→
More from coding & agent
- Grok Bot Gets Its Own Email Addresses to Sign Up and Handle Promos for You — elonmusk · 2026-11-21
- CapCut × ChatGPT plugin turns chat into a video generator and editor — SimplyAnnisa · 2026-11-21
- Dev builds daily X-digest Grok bot that auto-blocked 19 reply bots, heading to marketplace — petergyang · 2026-11-21
- DeepMind's Antigravity Robotics Plugin: 2 Prompts, ~20 Minutes to Command a Real FR3 Duo Arm — jackyliang42 · 2026-11-21
- Asking Astra 6.2 to Review Code Written by Opus 5.5 Yields Almost Nothing — rickasaurus · 2026-11-21
- Security & Web3 Firewall MCP Server Hits Official Registry: Detects Tool-Poisoning and Rug-Pull Drift — ezelech · 2026-11-21