Bayesian take: LLM benchmark noise means all models are probably tied
remilouf · x · 2026-10-22
Developer remilouf ran a Bayesian analysis and concluded LLM benchmarks are extremely noisy — "all models are probably tied at this point." In a follow-up he adds they got a +5 to +13 95% credible interval on tau2 bench, with more to come. The take feeds a growing skepticism that leaderboard gaps fall within noise.
More from Models
- Flint int4 hits 122.6 tok/s on Mac mini matching Qwen3.6-35B quality at 3.5x speed — _rockt · 2026-10-22
- OpenAI's New Pricing Slammed for Zero Transparency as $500 Tier Reveals No Usage Numbers — Effective_Tap_9786 · 2026-10-22
- OpenAI's pricing pages never state actual usage limits, power users cry foul — Monstermage · 2026-10-22
- Nemotron 3.5 Super VL: NVFP4 on one Blackwell GPU beats BF16 on two, ~2.2x per-GPU throughput — TheZachMueller · 2026-10-22
- LFM2.5-24B-A2B Tested and Found Lacking: Looping Outputs, Broken Code, Full Configs Posted — tracagnotto · 2026-10-22
- Decision Index 0.4: 224 models scored on private benchmarks only to curb benchmaxxing — multimodalart · 2026-10-22