2026-10-01
Blending RNN states with data each step provably bounds gradients on chaotic time series; EEG/ECG reconstructions beat LSTM, LEM and Neural ODE by wide margins.
Reconstructing a dynamical system from a time series means training a generative model whose own free-running trajectories reproduce the attractor geometry and temporal statistics of the data. Whenever the underlying system is chaotic, gradient training hits a wall that no architecture fixes: the same product of Jacobians that makes nearby trajectories diverge exponentially appears inside BPTT loss gradients, so the gradients blow up too. Prior work proved that gated units (LSTM, GRU), specialized recurrent architectures and gradient clipping do not solve this, while hard stability constraints prevent the maximum Lyapunov exponent from ever going positive, so the model cannot be chaotic in the first place. Sparse teacher forcing helps but requires knowing the system's Lyapunov exponents and still leaves gradients unbounded in principle. Real recordings such as EEG, ECG or climate series are mostly chaotic, so in practice the field was stuck on simulated Lorenz-style benchmarks.
One extra step per training iteration: after computing the loss, do not feed the RNN state directly to the next step. Blend it with the data-derived target state, ztilde = (1-a)zt + azbart. This generalized teacher forcing (GTF) traces back to a 1992 suggestion and had never been systematically studied.
The blend multiplies every Jacobian in the gradient chain by (1-a). The paper proves that with a = 1 - 1/sigmamax, where sigmamax is the supremum of the largest singular value over all Jacobians the network can express, the Jacobian-product norm stays bounded for arbitrarily long horizons. Any chaotic RNN necessarily has sigmamax > 1, so a suitable a always exists. Forcing is training-only, so at test time the network still evolves freely and can remain chaotic.
Computing sigmamax exactly is infeasible (a shallow PLRNN has 2^L linear regions), so the authors derive adaptive GTF: choose a so that the Jacobian chain spanning a training sequence stays close to identity, annealing down from a = 1 during training. The companion architecture is the shallow PLRNN, a one-hidden-layer ReLU recurrent network that can be rewritten as a dendPLRNN, keeping fixed points and cycles semi-analytically computable for scientific analysis.
Medians over 20 runs on two real-world datasets:
| Method | ECG Dstsp | EEG Dstsp | EEG latent dim |
| shPLRNN+GTF | 4.3±0.6 | 2.1±0.2 | 16 |
| dendPLRNN+id-TF | 5.8±0.6 | 3±1 | 105 |
| RC | 5.3±1.7 | 14±7 | 448 |
| LSTM-TBPTT | 15.2±0.5 | 30±21 | 160 |
| LEM | 16.3±0.2 | 10.2±1.5 | 76 |
ECG is a cardiac series delay-embedded into 5 dimensions; EEG is 64 channels over 60 seconds. GTF attains the lowest state-space distribution distance on both, several times better than LSTM, LEM and Neural ODE, and it reconstructs the EEG with only 16 latent variables versus 105 for dendPLRNN and 64 to 160 for the rest. SINDy diverges outright on both real datasets. On simulated benchmarks (Lorenz-63, Lorenz-96, multiscale Lorenz-96) every method is close to the ceiling. About 90 percent of GTF runs yield consistently good reconstructions while competitors are high-variance. Adaptive GTF matches the grid-searched fixed-alpha variant.
For practitioners this is a portable training trick: one line of state interpolation plus an adaptive coefficient stabilizes long-horizon recurrent training without touching the architecture or needing prior knowledge of the system. For the dynamical-systems-reconstruction field it sets a new standard on the hardest real-world data while pushing reconstruction dimensionality below the number of observed variables.
The authors themselves note that the gains are coupled to the shPLRNN architecture: a naive application to dendPLRNN did not reproduce the boost, and tanh activations performed worse. The alpha estimate relies on replacing forced-state Jacobians with data-inferred ones, backed by annealing rather than proof. Only two real datasets, one subject each, are tested. Generalization to ordinary time-series prediction or classification is stated as an outlook without experiments.