1.2M Calls Study: Even at Temperature 0, LLM Benchmarks Vary Up to 24%
lulzxdxdxd · reddit · 2026-10-13
An arXiv paper (2610.13823) using 1.2M model calls across 3 benchmarks finds that even at temperature 0, 0.07–10% of rows change between identical runs, and up to 24% on one task — implying small benchmark score differences may be pure noise, with direct implications for leaderboard credibility.
More from Models
- Claude Opus 5.5 maps 5,285 km coast-to-coast US route, sampled every 10m — garrytan · 2026-10-13
- 19 models that run on smartphones now beat GPT-4, once top for a full year — emollick · 2026-10-13
- Grok Bot Not Included in the Cheapest $8 Subscription Tier — op7418 · 2026-10-13
- New open-source OCR model hits 85.9% SOTA on olmocr bench at 4B params, but is it enough? — srchvrs · 2026-10-13
- MiniCPM-V-4.7-1B, a lightweight on-device multimodal model, trends on Hugging Face — openbmb · 2026-10-13
- Meta's Muse Wonder Full hits 64.6% on Vals Index with far better token efficiency — EdwardSun0909 · 2026-10-13