Study Reveals 12% of Questions in Major LLM Benchmarks Are Broken, Skewing Scores
pawofdoom · reddit · 2026-07-29
A researcher auditing mainstream LLM benchmarks discovered significant data pollution in evaluations like GPQA, MMLU-Pro, and MMMU-Pro.
- Alarming Error Rate: Approximately 12% of the questions across these benchmarks were verifiably broken, featuring malformed questions, incorrect answer keys, or multiple realistic answers.
- True Model Performance: Top models were previously topping out at 92-93% on GPQA-Diamond, which the author suspected was due to these flawed questions. When evaluated on the fixed -Clean versions of the benchmarks, top models scored around 98%.
- Open Resources: The author has released a full paper alongside -Clean versions of the benchmarks, complete with a flagged-candidate ledger, lm-eval-harness tasks, and Hugging Face datasets.
Related event: Study Reveals Heavy Contamination in Major LLM Benchmarks(2 posts)→
More from Models
- Google restricts Gemini Pro access for Google Plus subscribers, matching free tier — EmergencyLess8861 · 2026-10-16
- Meta open-sources Muse Spark: runs on two Mac Studios, license bans passing output off as human-written — mawcopolow · 2026-10-16
- Golden Gate Claude explainer resurfaces: interpretability premise still holds — HarperSCarroll · 2026-10-16
- Hugging Face livestreams release of a new open-weight model — huggingface · 2026-10-16
- Step 5 Preview hits #2 on OpenRouter weekly leaderboard with 19.5T tokens in one week — StepFun_ai · 2026-10-16
- Meta's open Muse Spark needs 493GB of memory — far beyond any consumer laptop — TejasKumar_ · 2026-10-16