+5 to +13 95% credible interval on tau2 bench: more benchmark-noise evidence
remilouf · x · 2026-10-22
Following his claim that LLM benchmarks are so noisy all models are probably tied, remilouf adds concrete numbers: a +5 to +13 95% credible interval on tau2 bench, with a fuller analysis coming soon.
More from Models
- StockBench sim: open-source models sweep top spots, GPT-5 loses to buy-and-hold — PtrPomorski · 2026-10-22
- Cohere's CulturalRiddles: 28 LLMs score 79.3% on cross-cultural riddles, but just 22% in some cultures — Cohere_Labs · 2026-10-22
- Rumors: Next Chinese AI Models Coming in Under 7 Days, Hacks 'Just the Prelude' — kimmonismus · 2026-10-22
- How much is an Anthropic Team subscription actually worth in tokens? Reddit tries to do the math — Acrobatic_Pool_713 · 2026-10-22
- Tencent open-sources Hunyuan-MT2: a 440MB offline translation model running on phones in 33 languages — alexcovo_eth · 2026-10-22
- Open tool computes API vs. RunPod self-hosting breakeven across 450+ models, with VRAM sizing — zackku · 2026-10-22