2026-10-02
ICML 2024 theory: black-box dynamical systems reconstruction models provably cannot recover unobserved regimes of multistable systems; SOTA algorithms fail in 50-run tests.
Dynamical systems reconstruction (DSR) learns the governing equations of a system straight from time series. State-of-the-art methods already handle both short-term forecasting and long-term invariants such as attractor geometry on synthetic benchmarks like the Lorenz system. Those benchmarks share a convenient property: for common parameters a single global attractor exists, so "generalization" only ever means more of the same regime. Real systems in neuroscience, ecology, climate or finance are multistable: several attractors, for example a spiking limit cycle next to a resting fixed point in one neuron model, coexist in the same state space. Empirical data usually covers one basin of attraction, while a proper scientific theory should say what happens in regimes never observed. The paper notes that prior work on anticipating bifurcations and tipping points quietly assumed some observations from the target domain, which is closer to transfer learning. The question here is when true out-of-domain generalization is possible at all, and the answer is blunt: for black-box models, essentially never.
Two error notions make "generalized to an unseen regime" checkable. The statistical error compares occupation measures, the share of time trajectories spend in each region of state space, via a Sliced Wasserstein distance. The topological error requires matching Lyapunov exponent signs, a close maximum exponent, and a small Hausdorff distance between limit sets. Learnability is imported from statistical learning theory: a problem is strictly learnable if every model with zero training-domain error also has zero test-domain error. Three theorems carry the paper:
On a bistable Duffing oscillator, three SOTA algorithms (reservoir computing, Neural ODE, shPLRNN) trained on four trajectories from one basin, 50 independent runs each, reconstruct the observed attractor and never the unobserved one. SINDy with the correct library generalizes across basins; drop the cubic Duffing term from the library and it collapses to a local approximation at best. The failure is structural: more data from one basin changes nothing, while data from both basins quickly yields a full reconstruction.
Three mechanisms explain the failure. Initialization carries a simplicity bias toward monostable systems: at the Glorot gain commonly used in DSR (σ=0.3), every randomly initialized shPLRNN had attractor entropy of zero, a single fixed point; raising the gain produces richer dynamics but networks that barely train. Generalizing solutions are saddles: retrain an already-generalizing model on single-basin data and its error on the second basin grows about 20-fold, and removing the second basin's data lets spurious minima appear in the loss landscape. Those solutions are also sharp: by minimum radius, generalizing valleys are narrower than non-generalizing ones, the opposite of the usual flat-minima story, while SGD prefers wide valleys.
This is a theory-grade warning for a whole branch of AI4Science: black-box models fitted to climate, brain or ecosystem time series cannot be extrapolated to regimes they have not seen. Zero training error and pretty short-term forecasts say nothing about cross-regime predictions, and the failure never shows up on training curves. The diagnosis localizes the problem too: without structural priors, generalizing solutions sit in sharp, saddle-like regions that current training systematically avoids. The follow-up directions are explicit, initialization and training procedures that encourage multistability, or accepting that physical priors are unavoidable. Anyone building time-series foundation models for science should read this first.
All experiments are on low-dimensional synthetic systems (Duffing, Lorenz-like, spatially extended Lorenz-96); no real-world data appears, and the authors position the work as laying theoretical foundations. The optimistic SINDy result hinges on the library containing the right terms, an assumption polynomials for real systems likely violate; enlarging the library makes the problem underspecified. The paper diagnoses but does not deliver a working fix. The theory assumes access to the data-generating process on the training domain; noise and sampling bias are not treated.