Learning Chaos Without Seeing Chaos: Extrapolation of Global Dynamics in Autoregressive Transformers
Yilun Liu, Yi Zhang, Ganyu Wu, Sikuan Yan, Mengyue Wang, Alois Knoll, Volker Tresp, Yunpu Ma
cs.LG, nlin.CD
2026-09-30
An 8k-parameter transformer trained only on fixed-point data extrapolates period doubling to period 128, with scaling ratio 4.6687 within 5e-4 of Feigenbaum's constant.
Autoregressive models are trained to do one thing: look at a history, predict the next state. At inference their own predictions are fed back as inputs, which turns the model into an autonomous dynamical system and shifts the input distribution from real states to model-generated ones. If training covered only a narrow parameter regime, does the model's self-generated dynamics stay faithful to the real system outside that regime?
The team from LMU and TU Munich picked the harshest test case: bifurcation. In the logistic map, every trajectory settles to a fixed point for r below 3; past 3, orbits of period 2, 4, 8 appear in turn and the system descends into chaos. Training data came only from r ∈ [2, 3], an interval that never shows a single period doubling, while closed-loop evaluation covers [2, 4]. Crossing a bifurcation demands the learned dynamics stay correct where long-run behavior changes qualitatively, which is stricter than ordinary OOD generalization. Reservoir computers, RNNs and Neural ODEs have reported similar extrapolation before; evidence on transformers was mixed, including a 2026 report of failures on collapse transitions.
The setup is deliberately minimal:
One choice is deliberate: the feed-forward block is token-wise, so parameter information can only reach state computations through attention, leaving a clean surface for causal interventions. Verification is stricter than eyeballing bifurcation diagrams: a blind scan finds candidate transitions without consulting known bifurcation values, local grids are adaptively refined, and each flip counts as verified only if an autodiff solve of orbit closure plus multiplier crossing −1 lands inside the trajectory bracket, with stability continuation confirming the crossing on both sides.
| Setting | Metric | Result |
| Transformer (logistic) | verified consecutive flips | 7, up to period 128 |
| Transformer (logistic) | finite-order scaling ratio | 4.6687 (same-order reference 4.669132; Feigenbaum constant 4.6692) |
| Transformer (sine) | period-doubling cascade | full qualitative cascade recovered |
| Lorenz (joint ρ, σ) | fixed/nonfixed classification | 593/625 cells agree; wing switching reproduced in 362/375 |
| Lorenz (28, 10) | largest Lyapunov exponent | 1.0404 vs 0.9046 reference; sign correct at all 10 tested pairs |
| Hopf (−0.1, 1) | outer cycle radius | 0.948 vs 0.942 reference |
The scaling ratio is the hardest number. The Feigenbaum constant is universal: different unimodal maps have different bifurcation locations yet share the same limiting ratio. The fitted critical exponents, 2.00 for the logistic transformer and about 2.10 for sine, land in the quadratic universality class, so what was recovered is structure, not a pretty picture. Shrinking the training window to [2.9, 3] or shifting it down to [2, 2.5] still yields the cascade, confirmed in 11 of 12 window-seed combinations by the coarse detector.
Emergence has a timeline: no doubling by 2,000 updates, the first at 3,000, three by 4,000. One-step accuracy and closed-loop fidelity also dissociate: between 6,000 and 10,000 updates, OOD one-step MSE improves from 1.10×10⁻³ to 2.41×10⁻⁴ while closed-loop OOD Wasserstein-1 worsens from 0.0296 to 0.0867.
Causal interventions pin the mechanism. Patching the second-layer attention output with the activation from a different parameter transfers almost the entire parameter effect (R > 0.99999); patching the first layer is negligible (R = 0.000151). Blocking second-layer attention to the parameter token during closed-loop generation pushes OOD W1 from 0.03497 to 0.60605 and wipes out the detected flips. The dominant layer differs across seeds, so the routing is learned rather than architectural; permuting parameter labels during training destroys the extrapolation in every run.
For anyone building world models or scientific surrogates, the paper offers a checkable position: a learned predictor carries scientifically meaningful structure in the dynamical system obtained by recursively applying it, beyond what pointwise accuracy or internal representations capture. Bifurcation placement, universality scaling, Lyapunov signs and attractor selection become read-outs for what a model has internalized.
For interpretability work it is a clean 8,304-parameter testbed with no positional encodings and a two-token-type design, where patching and closed-loop blocking support causal attribution; the recipe should transfer to larger models.
The boundary should be stated plainly: this is an existence proof on low-dimensional smooth systems, not a general law.