Epoch AI finds 46% of audited Humanity's Last Exam questions have accuracy-altering errors
charles_irl · x · 2026-09-19
Epoch AI's benchmark review delivers a "Flawed" verdict for Humanity's Last Exam, the knowledge benchmark created by the Center for AI Safety, Scale AI, and independent contributors.
- Method: sampled 48 questions (6 each from 8 categories), used Fable 5 to surface errors, then manually audited each against an evidence bar.
- Result: 22 of 48 questions (46%) had substantial accuracy-altering errors.
- Breakdown: 12 questions impossible to answer as written (missing assumptions, contradictory premises, multiple valid interpretations, objective answers demanded for subjective questions); 10 well-posed questions could produce false negatives, 5 of which could also produce false positives.
- Example: a CS question asks for a 4-point DFT over a sequence with 8 samples.
- The review covers the original benchmark, not HLE-Rolling or HLE-Verified.
Related event: Epoch AI's First Benchmark Audit Flags 9 of 15 as Flawed(22 posts)→
More from Models
- Musk cites unsustainable Grok Bot costs as Hermes offers one-command agent migration — Teknium · 2026-11-04
- Hobbyist rewrites Qwen 0.6B weights with 39.8M tokens, no gradients, in 3h12m — Altruistic_Heat_9531 · 2026-11-04
- Same leak, reshared: ChatGPT's five family plan prices spotted again — heyshrutimishra · 2026-11-04
- KOL visits StepFun's Shanghai office, hints a powerful new model is coming — teortaxesTex · 2026-11-04
- Fable 5.5 delayed as Anthropic reportedly bets OpenAI's Astra 6.2 won't beat it — haider1 · 2026-11-04
- ByteDance open-sources Axion 1.0-35B agent foundation model with full reproducible recipe — pmttyji · 2026-11-04