Same Model, Two Runs, 1.7-Point Gap: Benchmark Variance Underestimated
PawelHuryn · x · 2026-10-14
After burning through his Muse subscription, the author re-ran benchmarks across several models and found single-run scores diverge noticeably from multi-run averages.
Headline results (max effort)
- Astra 5.1: 45
- Fable 5.1: 43
- Muse Wonder Full 1.0: 42 (previously 44)
- Opus 5.5: 41.7
- Gemini 4 Argon (high): 40.7
Variance exposed by repeat runs
- Muse Wonder Full 1.0 max: 44, 40 (mean 42.0)
- Muse Wonder Full 1.0 xhigh: 44, 39 (mean 41.5)
- Muse Wonder 1.0 max: 30, 36, 29 (mean 31.7)
Conclusion: running three times shifts the median score by 1.7 points versus a single run; variance here looks high, and ideally each effort level should be run 4-5 times. xhigh and max are nearly indistinguishable, meaning one of those scores will likely drift with more runs.
More from Models
- Reports of mass account bans on Meta's free Muse model after heavy free-tier abuse — op7418 · 2026-10-14
- 256 real mobile jobs: Claude Haiku 5.5 ties GPT-6 Luna despite 23.5-point benchmark gap — LovesWorkin · 2026-10-14
- Hugging Face CEO: media obsession with reasoning LLMs ignores 3M+ useful models on HF — marktenenholtz · 2026-10-14
- Dev theo Logged Every Claude and Codex Request Into ClickHouse — Has Nearly Abandoned Codex — scaling01 · 2026-10-14
- User awaits Gemini 4 Argon in Antigravity to benchmark it against CodeX and Claude Code — HarrySurden · 2026-10-14
- New transformers decision pipeline brings unified serving for jev/clef/von model families — art_zucker · 2026-10-14