Open-source leaderboard analysis: AI labs report benchmark scores 6.4% higher than independent testers
jakecoolguy · reddit · 2026-10-19
The creator of open-source leaderboard whatmodel.sh matched 611 comparisons across 180 models and 103 benchmarks from 1,629 published results, finding labs' self-reported scores average 6.4% higher than independent testers on identical tests: 375 pairs favor the lab, 216 favor independent testers, 20 are tied.
The largest gaps appear on SWE-bench Verified and MBPP (median +9.0, with labs higher in 92% of SWE-bench pairs), followed by ARC Easy, AIME 2024 and GSM8K; MMLU, Winogrande and ARC Challenge show little to no gap. The dataset (CC BY-SA 4.0) and code (MIT) are fully open with sources and licenses recorded.
More from Models
- ChatGPT web can actually listen to audio, but the same model can't via API or Codex — Possible_Statement84 · 2026-10-23
- MiniMax M3.1 Flash free for 3 days on 0G, with 1M context and 512K output — MiniMax_AI · 2026-10-23
- TLA+ expert corrects Grok: TLC only exhausts finite state spaces, not all behaviors — tianyin_xu · 2026-10-23
- OpenAI blocks manual reset with 13% usage left: 'no reset needed right now' — AccomplishedSugar490 · 2026-10-23
- Meta's 816B Muse Spark Open scores 39, losing to 180B Qwen3.8-Flash-Next at 40 — BTA_Labs · 2026-10-23
- Gemini refuses to generate image in Docs, blames document size limit — viksit · 2026-10-23