Analysis of 31k LLM benchmarks: within-day variation 2.8 pts
ionutvi · reddit · 2026-08-29
Analysis of 31,352 hourly LLM benchmark scores reveals a within-day variation of 2.8 points versus an 8.4-point variation between days. The author open-sourced AIStupidLevel, a platform for continuous capability drift monitoring with 169,858 runs. It features four test suites and a smart router that switches models based on real-time performance metrics like stability and tool-calling reliability.
Related event: 31K Hourly Benchmarks Show LLM Scores Swing 3x More Across Days(2 posts)→
More from Infra
- Microsoft unveils in-house 3T-class MAI base models: 80B active params, 50T training tokens detailed — SinclairWang1 · 2026-10-16
- Jensen Huang says Nvidia doesn't need to backstop CoreWeave as its cost of capital falls — ayushthakur0 · 2026-10-16
- Zuckerberg: Meta's Muse Spark Open weights out today, runs on Mac Studios — AIatMeta · 2026-10-16
- Microsoft unveils MAI-Base-2: 2.8T-param MoE, 4-12x more pre-training FLOP-efficient than DeepSeek — ChengleiSi · 2026-10-16
- Local RAG pipeline: retrieval sets the ceiling, not the model — Arindam_1729 · 2026-10-16
- AI Data Center Cooling Playbook: $16B+ in Public Orders, Modine Down 7.5% Then Up 46% — FinanceYF5 · 2026-10-16