WhatIf-Bench exposes extrapolation gap in time-series foundation models, skill drops to -7%
irinarish · x · 2026-11-04
Lacuna Research and collaborators released WhatIf-Bench, a large-scale benchmark (37,665 tasks) built from simulators of data centers, power grids, road traffic and water networks to test intervention-aware time-series forecasting.
Key findings:
- Time-series foundation models can read covariates in context, enabling scenario analysis: set future covariates and forecast how the target responds
- When a future event replays values the model has seen, models score 48-85% skill over a naive baseline
- When asked to extrapolate to novel covariate values, skill collapses to between -7% and 50%, with regression baselines falling the most — a clear "extrapolation gap"
- Frontier LLMs score highest overall, ahead of the best time-series foundation models
More from Research
- Library of Layers (LoL): Paper Proposes Adaptively Ordering Transformer Layers at Runtime — Thrumpwart · 2026-11-04
- Researchers Warn of 'Vibe Science': AI-Slop ICLR Submissions Full of Bold Claims, Broken Logic — IanArawjo · 2026-11-04
- TAMU professor releases 400+ page AI-assisted tutorial deck on generative models — ShuiwangJi · 2026-11-04
- DeepMind mafia seeding a new wave of AI for Science startups in the UK — ersatzben · 2026-11-04
- MIT CSAIL fixes small models by hand-editing weights — hype or old idea rebranded? — ziv_ravid · 2026-11-04
- Robots need failure data: learned sims handle rendering, MuJoCo keeps contact honest — chris_j_paxton · 2026-11-04