31,352 Repeat Tests Show LLM Benchmark Scores Drift Heavily Day to Day

Based on 31,352 repeated benchmark observations across 49 models, an evaluation platform founder shows LLM scores drift substantially over time, with day-to-day variance about three times the intra-day variance (intra-day std ≈2.80), arguing benchmarks should be treated as longitudinal measurements rather than snapshots.

2026-09-07 ~ 2026-09-07 · 2 related posts

• 31,352 repeated benchmark runs show LLM scores drift 3x more across days than within a day — ionutvi · 2026-09-07
• 31,352 repeated LLM measurements show day-to-day scores vary 3x more than within-day runs — ionutvi · 2026-09-07