Epoch AI's First Benchmark Audit Flags 9 of 15 as Flawed

On September 18, Epoch AI launched Benchmark Reviews, a program to systematically audit AI benchmarks themselves. Of the first 15 covered, only 4 earned a Verified rating, 9 were judged Flawed, and 2 could not be reviewed due to insufficient information. The result directly challenges the credibility of widely cited model evaluations and drew broad community discussion.

Confirmed

• Epoch's first batch: 4 Verified, 9 Flawed, 2 undetermined
• For Humanity's Last Exam, 6 questions from each of 8 subjects (48 total) were sampled; 46% had substantive problems
• Terminal Bench 4.0: cross-checked with GitHub issues, 30 of 66 tasks (45.5%) had grading defects such as exploitable graders, answer leakage, and rejection of correct solutions
• DeepSWE 1.1 was found flawed; the DeepSWE team welcomed the review and public discussion
• Epoch researcher Michelle Campeau noted Anthropic itself disclosed that about 40% of items in its Critical Point benchmark were problematic
• PostTrainBench v1.1 was among the first Verified benchmarks
• @burnytech relayed that RL-related evaluation environments have even worse problems

Unconfirmed

• Terminal Bench maintainers challenged the audit's rigor and rejected the reading that "TB 4.0 is broken"; @DimitrisPapail likewise argued grading defects do not mean the benchmark is wholly broken; Epoch has not publicly responded
• alexgshaw (relayed by @simonguozirui) questioned whether the audit approach is fair, since benchmarks are software and software inevitably has bugs

Why it matters

• Benchmark scores underpin model comparisons and leaderboards; most mainstream benchmarks failing audit casts doubt on current model-vs-model claims
• The program offers a public reference for distinguishing trustworthy benchmarks, but the methodology and the meaning of the Flawed label are already drawing pushback from maintainers and community members
• Even Anthropic's own benchmark was self-reported as heavily flawed, reminding users to weigh data quality when citing scores

2026-09-18 ~ 2026-09-20 · 22 related posts

• [source] Epoch AI launches Benchmark Reviews: only 4 of first 15 benchmarks earn Verified status — xeophon · 2026-09-18
• Epoch AI launches Benchmark Reviews, audits 15 benchmarks: 4 verified, 9 flawed — Jsevillamol · 2026-09-18
• Epoch AI audits 15 AI benchmarks, finds 9 flawed — 46% of sampled HLE questions broken — morqon · 2026-09-18
• Audit finds 9 of 15 AI benchmarks flawed: 45.5% of Terminal Bench 4.0 tasks broken — ajratner · 2026-09-18
• Epoch AI audits AI benchmarks: 9 of 15 reviewed benchmarks found flawed — joecole · 2026-09-18
• Epoch AI audits 15 AI benchmarks: 4 Verified, 9 Flawed; PostTrainBench passes and preps v1.2 — maksym_andr · 2026-09-18
• Audit flags 45.5% of Terminal Bench 4.0 tasks as broken; maintainers push back on rigor — DimitrisPapail · 2026-09-19
• [source] Terminal-Bench 4.0 scoring flaws hit <3% of rollouts, despite Epoch finding 30/66 defective tasks — DimitrisPapail · 2026-09-19
• Epoch AI Launches Benchmark Reviews, Finds 9 of 15 Agent Benchmarks Flawed — burny_tech · 2026-09-19
• Terminal Bench audit backlash: known issues affect under 3% of leaderboard, researcher says — AlexGDimakis · 2026-09-19
• Epoch AI's benchmark audit calls 9 of 15 flawed; TB4 authors push back — DimitrisPapail · 2026-09-19
• Epoch AI audits 15 benchmarks; Anthropic says 40% of Critical Point physics questions are broken — Jsevillamol · 2026-09-19
• [source] Epoch AI finds 46% of audited Humanity's Last Exam questions have accuracy-altering errors — charles_irl · 2026-09-19
• Epoch AI audits 15 AI benchmarks: 9 of 15 found flawed — burny_tech · 2026-09-19

8 near-duplicate retellings: Jsevillamol · stochasticchasm · iamrobotbear · pvncher · Jsevillamol · Jsevillamol · simonguozirui · Jsevillamol