31,352 repeated benchmark runs show LLM scores drift 3x more across days than within a day
ionutvi · reddit · 2026-09-07
A benchmark platform founder argues LLM evaluation should be treated as a longitudinal measurement problem, not a leaderboard problem, since API-served models change behind the scenes without version transitions.
Key findings:
- 31,352 repeated score observations across 49 models
- Within-day score std dev: 2.80 points; between-day daily median std dev: 8.43 points — roughly 3:1
- The author stops short of claiming providers change models daily (too many confounders), but argues temporal variation must be measured, not treated as noise
Methodology:
- Versioned benchmark configs; only compare compatible longitudinal observations
- Repeated execution-based evals instead of LLM judges
- Separate availability failures from valid outcomes, track serving/version metadata, run change-point detection
- Addresses benchmark contamination by publishing methodology while withholding the live task bank
The author invites criticism on time-series units, distinguishing drift from infrastructure effects, benchmark secrecy boundaries, and alternatives to change-point detectors.
Related event: 31,352 Repeat Tests Show LLM Benchmark Scores Drift Heavily Day to Day(2 posts)→
More from Models
- GPT-6.1 Terra rolls out to all Plus, Pro, Enterprise and Edu users — OpenAI · 2026-10-27
- OpenAI launches GPT-6.1 Terra with 50% lower API prices than GPT-5.6 Terra — OpenAI · 2026-10-27
- In cyber, intention matters little: Chintala on model info leaks, and what users should do — brhydon · 2026-10-27
- Why don't more LLMs ship 10M-token context? A researcher asks about attention tradeoffs — agarwl_ · 2026-10-27
- Ortet's first health eval: 34 open-weight models, 50 tasks, generic rankings fall short — thegautamkamath · 2026-10-27
- VLM Run launches OCR Playground for side-by-side comparison of GLM-OCR, DeepSeek-OCR-2 and more — spillai · 2026-10-27