31,352 repeated LLM measurements show day-to-day scores vary 3x more than within-day runs
ionutvi · reddit · 2026-09-07
The founder of AI Stupid Level shared a longitudinal analysis based on 31,352 repeated benchmark score observations across 49 models: within-day score standard deviation was 2.80 points, while between-day standard deviation of daily medians was 8.43 points — roughly 3x — suggesting a single benchmark score shouldn't be treated as a permanent property of a model.
The author stresses this is descriptive, not proof that providers silently change models: sampling noise, task composition, serving-side behavior, infrastructure, and benchmark changes can all produce drift. Their methodology includes:
- Repeated trials instead of single runs
- Execution-based scoring over LLM-as-judge
- Benchmark versioning: no silent comparison against old baselines after config changes
- Separating capability failures from provider/availability failures
- Recording systemfingerprint metadata, while noting an unchanged fingerprint proves nothing
- Cross-model correlation to attribute movement to provider-side changes
A second core issue is benchmark contamination: a continuously running benchmark whose tasks are fully public gradually becomes part of the environment models are optimized against, motivating a split between methodological transparency and evaluation-set secrecy. The argument: LLM benchmarks should shift from leaderboards toward continuous observability systems. A methodology PDF is publicly available.
Related event: 31,352 Repeat Tests Show LLM Benchmark Scores Drift Heavily Day to Day(2 posts)→
More from Models
- Kimi K3.1 Pro tops MathArena open models, closing frontier gap to about 6 months — teortaxesTex · 2026-10-26
- Ruff creator Charlie Marsh: frontier models find infinite bugs in any software — charliermarsh · 2026-10-26
- Dev testing one skill across Opus 5.5 and Sol 6.1 finds instruction following 'all over the map' — lucasmeijer · 2026-10-26
- supermemory unveils learner-1, claiming continual learning for any model — julianweisser · 2026-10-26
- Anthropic says Opus 5.5 cuts cost per task by ~40% vs Opus 5 — taufiqintech · 2026-10-26
- AI search APIs ranked: top 12 within 7 points, yet prices differ nearly 7x — FinanceYF5 · 2026-10-26