LLM Evaluation Is a Forecasting Problem: Stale Benchmark Scores Pick the Wrong Model
micahgoldblum · x · 2026-10-15
A new arXiv paper by Rahul Thomas, Adam Block, and Micah Goldblum argues LLM evaluation should be treated as a time-series forecasting problem, since models train on past data but deploy into the future.
Key findings:
- Benchmark scores go stale fast: on LiveCodeBench, using 12-month-old data to predict next month's scores more than doubles prediction error versus 1-month-old data.
- Across 19 models and 12 datasets, the Chronos-2 forecaster cuts next-year prediction error by 37% versus pooling all past scores.
- Model choice flips: on WildTime-arXiv, standard pooling picks Llama-3.1-70B-Instruct in all 11 target years, even though Qwen3-32B-thinking is actually best in 10 of them.
- Distribution shift hits training too: in an early NIST competition, models with <1% validation error (Census-worker digits) fared far worse on high-schooler digits.
The authors recommend keeping timestamps, weighting recent data more heavily, and running time-aware forecasters over eval history. Code and data are released.
Related event: LLM Evaluation Is a Forecasting Problem, New Research Argues(2 posts)→
More from Models
- Brin says Gemini 4 puts Google on the 'RSI treadmill'; critics demand hard data — daveholtz · 2026-10-15
- Meta launches Muse Wonder models, claiming SOTA on personal and professional agent tasks — AIatMeta · 2026-10-15
- Third-party eval featured on Jev site: 98.7% cheaper, 48x faster, 89% vs 88% accuracy vs GPT-5.6 Terra — generativist · 2026-10-15
- TypeSafe ships Jev 1.14.0 with image inputs, longer context under experimental 'Danger Zone' flags — usamawahabkhan · 2026-10-15
- Meta's Muse Wonder 1.0 Enters PINNACLE Top Ten, Beating Claude Opus 5.5 and GPT-6 — ryanshrout · 2026-10-15
- Meta's Muse Wonder 1.0 enters Signal65 PINNACLE top ten with 33% fewer errors than Spark — LudovicCreator · 2026-10-15