Parallel-in-Time Training of Recurrent Neural Networks for Dynamical Systems Reconstruction
Florian Hess, Florian Götz, Daniel Durstewitz · 2026
Read Parallel-in-Time Training of Recurrent Neural Networks for Dynamical Systems Reconstruction by Florian Hess, Florian Götz, Daniel Durstewitz (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12683.txt.
Introduction
Hess, Götz and Durstewitz study parallel-in-time training of recurrent neural networks for reconstructing nonlinear systems from data. Sequential models usually force linear runtime in sequence length. Recent parallel algorithms achieve log time by splitting along sequence length. The paper examines two classes of parallel algorithms, both built on linked scans. The first class covers models with linear self-run motion and a nonlinear readout, like modern state space models. The second covers general curved models, opening new avenues for long-sequence rebuilding.
Catalog record
| CREATOR | Florian Hess, Florian Götz, Daniel Durstewitz |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 81843 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.12683
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Parallel-in-Time Training of Recurrent Neural Networks for Dynamical Systems Reconstruction Florian Hess, Florian Götz, Daniel Durstewitz Abstract Reconstructing nonlinear dynamical systems (DS) from data (DSR) is a fundamental challenge in science and engineering, but it inherently relies on sequential models. Recent breakthroughs for sequential models have produced algorithms that parallelize computation along sequence length T T , achieving logarithmic time complexity, 𝒪 ( log T ) \mathcal{O}(\log T) . Since sequence lengths have been practically limited due to the linear runtime complexity 𝒪 ( T ) \mathcal{O}(T) of classical backpropagation through time, this opens new avenues for DSR. This paper studies two prominent classes of parallel-in-time algorithms for this task, both of which leverage parallel associative scans as their core computational primitive. The first class comprises models with linear yet non-autonomous dynamics and a nonlinear readout, such as modern State Space Models (SSMs), while the second consists of general nonlinear models which can be parallelized using the DEER framework. We find that the linear training-time recurrence of the first class of models imposes limitations that often hinder learning of accurate nonlinear dynamics. To address this, we augment DEER with Generalized Teacher Forcing (GTF), a novel variant within the more general nonlinear framework that ensures stable and effective learning of nonlinear dynamics across arbitrary sequence lengths. Using GTF-DEER, we investigate the benefits of training on extremely long sequences ( T > 10 4 T>10^{4} ) for DSR. Our results show that access to such long trajectories significantly improves DSR if the data features long time scales. This work establishes GTF-DEER as a robust tool for data-driven discovery and underscores the largely untapped potential of long-sequence learning in modeling complex DS. 1 Introduction Understanding and predicting the behavior of complex nonlinear systems from neural circuits and climate dynamics to fluid flows and ecological networks is a central aim across the natural and engineering sciences ( 6 ; 36 ; 74 ; 35 ) . A particularly ambitious goal is to reconstruct the underlying dynamical system (DS) directly from observed time series, a problem known as dynamical systems reconstruction (DSR). Beyond short-term forecasting, DSR requires that the learned model faithfully reproduces the long-term statistical and geometric properties of the true system, such as attractor geometry, power spectra, and Lyapunov exponents. Central to the definition of DS is the flow operator, which provides a recursive rule of how the DS evolves in time 71 ; 38 . DSR methods that approximate it are inherently recursive ( 27 ; 52 ) , such as RNNs, commonly trained by backpropagation through time (BPTT; ( 81 ) ). While several well-documented pathologies of BPTT – most notably exploding gradients under chaotic dynamics ( 52 ) – can be successfully mitigated by control-theoretic training algorithms such as sparse and generalized teacher forcing (STF/GTF) ( 52 ; 33 ) , the computational cost of BPTT remains fundamentally linear in the sequence length T , rendering training on problems with long intrinsic timescales prohibitively expensive. Indeed, DSR applications have historically been confined to modest sequence lengths to keep training tractable ( 11 ; 33 ; 76 ) , leaving open the question of whether learning from substantially longer sequences offers any benefit for reconstruction quality. Recent advances in parallel-in-time sequence modeling offer a path forward, and we examine two common paradigms with respect to their performance in DSR. Linear training-time recurrences with a nonlinear readout, as instantiated by modern SSMs, admit trivial parallelization via linear scans and avoid chaos-induced exploding gradients by construction ( 52 ; 55 ; 82 ) . By exposing a duality between these linear SSMs and nonlinear RNNs, we show, however, that this convenience comes at a cost: common diagonal parameterization of the linear recurrence used during training imposes structural limitations that often prevent the model from learning accurate nonlinear dynamics. Moreover, its training suffers from exposure bias, degrading autoregressive roll-outs at test-time ( 4 ; 62 ) . The second paradigm – general nonlinear RNNs parallelized via DEER ( 43 ) – is in principle better suited to DSR, but naive application fails on chaotic data because the Jacobian products driving DEER’s Newton updates diverge whenever the underlying dynamics exhibits positive Lyapunov exponents ( 52 ; 24 ) . Our main methodological contribution is to resolve this tension by combining DEER with Generalized Teacher Forcing (GTF) ( 33 ) . The resulting algorithm, GTF-DEER , inherits the long-sequence scalability of DEER while retaining GTF’s chaos-taming properties: GTF turns the model into a stable DS during training such that GTF-DEER enjoys the average-case \mathcal{O}((\log T)^{2}) scaling established in 24 , even when the underlying system is chaotic. Empirically, GTF-DEER delivers speedups of up to 870\times over sequential training while matching or improving reconstruction quality. Equipped with GTF-DEER, we then revisit the central empirical question: does training on much longer sequences actually improve DSR? Leveraging the ability to train stably on trajectories of length T>10^{4} , we find that access to long trajectories allows the DSR model to efficiently capture long time scales if present in the data, yielding substantial gains in long-term statistics, which an equivalent linear SSM based model cannot match. Taken together, our results establish GTF-DEER as a direct replacement of sequential training while highlighting long-sequence training as a largely untapped lever for DSR. 2 Related work Dynamical systems reconstruction Data-driven methods for DSR fall into two broad categories: The first approximates the vector field that underlies the observed dynamics, assuming the process is governed by differential equations. Sparse Identification of Nonlinear Dynamics (SINDy) and its variants ( 14 ; 13 ; 15 ; 51 ; 17 ) are particularly popular in the physical sciences and enjoy quick training through least-squares regression, but rely on pre-defined function libraries and struggle with noisy, non-stationary and partially observed empirical data. Neural ODE/PDE methods ( 16 ; 37 ; 2 ; 39 ; 1 ) offer universal approximation capabilities and can be augmented with physical priors ( 61 ; 42 ) , but are difficult to train in practice. The second and larger class of models directly approximates the flow operator through black-box universal approximators based on neural networks, including neural operators ( 41 ; 47 ) , Koopman operators ( 48 ; 56 ; 12 ; 53 ; 3 ; 80 ) and hybrids thereof, reservoir computers ( 59 ; 58 ; 75 ; 60 ; 57 ; 23 ) , and RNNs trained through BPTT ( 77 ; 76 ; 11 ; 33 ; 63 ; 9 ) , often accompanied by specialized control techniques that ease optimization and address exploding gradients under chaos ( 52 ; 11 ; 33 ; 64 ) . The latter approach in particular achieves SOTA performance on a wide-range of benchmark systems and performs well even on challenging empirical systems ( 33 ; 79 ; 10 ) , which has led to its adoption as a backbone in foundation models for DSR ( 31 ) . Efficient sequence modeling A central challenge in sequence modeling is the stable capture and retrieval of long-term dependencies ( 5 ; 34 ; 30 ; 29 ; 55 ; 82 ) . One practical bottleneck when using autoregressive models such