Audit finds up to 12% of GPQA, MMLU-Pro, and MMMU-Pro questions were broken
pawofdoom · reddit · 2026-07-29
Audit finds up to 12% of GPQA, MMLU-Pro, and MMMU-Pro questions were broken
A Reddit post points to a paper and repository auditing several popular benchmarks for malformed questions, wrong answer keys, and multiple plausible answers. The audit found that about 12% of questions in GPQA-Extended, MMLU-Pro, and MMMU-Pro were verifiably broken.
After removing the bad items, top models reportedly reached around 98% on the cleaned versions, up from the low 90s on the originals. The author also released -Clean benchmark versions, a full ledger of flagged items, dual original-vs-clean scoring, lm-eval-harness tasks, and Hugging Face datasets.
Related event: Study Reveals Heavy Contamination in Major LLM Benchmarks(2 posts)→
More from Research
- Tsinghua professor's ApexIntelligence raises $50M to build self-evolving foundation models — DeryaTR_ · 2026-10-16
- Fluids Don't Compute—They Contribute the Nonlinear Step, Sandberg Argues — anderssandberg · 2026-10-16
- Anders Sandberg on fluid computation: error budgets tighten quadratically over time — anderssandberg · 2026-10-16
- How the fluid computation works: force field advects particle to 0/1 boxes — anderssandberg · 2026-10-16
- FlashAttention creator Tri Dao joins Radical Numerics as scientific advisor — exnx · 2026-10-16
- SAIR Launches Open Math OS, Offering Free Compute and 100+ Tools for Community-Owned Math AI — Thom_Wolf · 2026-10-16