31K hourly LLM benchmarks show 8.4-point day-to-day variation, 3x within-day noise
ionutvi · reddit · 2026-08-29
The author built a continuous evaluation pipeline and analyzed 31,352 hourly benchmark scores across 49 model identifiers (coding, deep reasoning, tool calling, high-frequency canaries):
- Within-day variation was 2.8 points vs 8.4 between days (3×), suggesting isolated hourly moves are mostly stochastic noise while sustained daily-window changes are the stronger signal for performance drift.
- Coding answers are actually executed, not just model-judged; tool-calling is tested in isolated Docker environments; each task runs 5 times and is aggregated.
- The pipeline aggregates to daily medians and applies sequential change-point detection with minimum-effect thresholds before flagging degradation or recovery.
- At screenshot time it detected a 32% sustained decline in Gemini 3.1 Flash Lite, classified as critical.
The system is open-sourced as AIStupidLevel (MIT), totaling 169,858 benchmark runs and 88M+ tokens, monitoring 22 models across 6 providers, and powers an OpenAI-compatible router that selects models by current performance, stability, latency and cost.
Related event: 31K Hourly Benchmarks Show LLM Scores Swing 3x More Across Days(2 posts)→
More from Infra
- Autonomous sells personal AI computers from $26,100 that keep data in the room — dee_hw · 2026-10-15
- vLLM's bitwise-consistent RL: zero KL on Nemotron NVFP4, 91.3% throughput on DeepSeek-V4-Flash — DynamicWebPaige · 2026-10-15
- vLLM achieves bitwise-consistent RL training: DeepSeek-V4-Flash hits 0 diff over 100 steps — vllm_project · 2026-10-15
- vLLM Achieves Bitwise-Consistent RL Training, Fixing KL Blowups and Score Collapse — vllm_project · 2026-10-15
- Analyst Ben Bajarin: The AI Industry Needs a Lot More Compute — BenBajarin · 2026-10-15
- Next-gen GraalVM benchmarks: 4x faster startup, 2x less memory, no build step needed — RealGeneKim · 2026-10-15