Benchmark scores decay: treating LLM evals as forecasting shows time drift hurts deployment
micahgoldblum · x · 2026-10-15
Researcher micahgoldblum posted a long thread arguing that LLMs are trained on past data and deployed in the future, yet we evaluate them as if time doesn't exist.
Key findings:
- Across pretraining corpora (FineWeb-Edu, DCLM, Nemotron-CC), document length, compressibility, POS, domain and embedding distributions all drift over time; within-corpus drift can rival between-corpus gaps
- The biggest domain shift hit in 2017 when Common Crawl expanded coverage; AI-detector scores and words like "delve" spiked after ChatGPT, but most trends predate LLMs
- Pretraining per year and evaluating across years shows loss grows smoothly with train-test year distance, and past→future transfer is harder than the reverse — exactly the deployment direction
- Framing eval as forecasting, time-series models like Chronos-2 over eval history gain 6-8pp in hit-rate over a historical-mean baseline
Practical advice: keep timestamps and weight recent data more; for fixed SFT budgets, recent data improved next-year accuracy in every tested setting.
Related event: LLM Evaluation Is a Forecasting Problem, New Research Argues(2 posts)→
More from Research
- Cohere Labs: more coding agents isn't a scaling strategy — when to delegate work — pvncher · 2026-10-15
- UCLA paper: productivity tech like AI can trigger a workplace "rat race" that hurts welfare — soumitrashukla9 · 2026-10-15
- Deterministic Miller-Rabin primality testing now in P after last week's breakthrough, ~O((log n)^9.9984) — geoffreyirving · 2026-10-15
- Initialization matters far more than expected in LLM-driven scientific discovery, new study finds — BenBlaiszik · 2026-10-15
- RLVR is just RL — RLHF is the weird one, argues lateinteraction — lateinteraction · 2026-10-15
- Mathematician models OpenAI's 372-theorem drop as a self-organized criticality collapse — littmath · 2026-10-15