On the dimension of pullback attractors in recurrent neural networks

Muhammed Fadera 1 · 2025

Read On the dimension of pullback attractors in recurrent neural networks by Muhammed Fadera 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-11357.txt.

Introduction

Muhammed Fadera bounded the size of the hidden attractor inside a reservoir computer in 2025. Reservoir computing trains only a readout while a fixed recurrent network echoes its input, and such systems rebuild chaotic attractors with uncanny fidelity, matching even fractal dimensions. A conjecture credits this to the network embedding the chaos in its own state space before learning.

Taking a non-autonomous view, Fadera targets the pullback attractor, the set of states the reservoir settles into as it is driven, which training and prediction roughly trace. He proves its box-counting dimension is no larger than that of the space of input sequences under the product structure. The catalog creator field appends an affiliation number; the head byline governs. Readers get a limit theorem for a learning machine: the echo cannot be more complex than its input stream.

Catalog record

CREATORMuhammed Fadera 1
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-11357

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

On the dimension of pullback attractors in recurrent neural networks Muhammed Fadera 1 Abstract Recurrent neural networks trained via the reservoir computing paradigm have demonstrated remarkable success in learning and reconstructing attractors from chaotic systems, often replicating quantities such as Lyapunov exponents and fractal dimensions. It has recently been conjectured that this is because the reservoir computer embeds the dynamics of the chaotic system in its state space before learning. This conjecture has been established for reservoir computers with linear activation functions and remains open for more general reservoir systems. In this work, we employ a non-autonomous dynamical systems approach to establish an upper bound for the box-counting dimension of the pullback attractor, a subset of the reservoir state space that is approximated during training and prediction phases. We prove that the box-counting dimension of the pullback attractor is bounded above by the box-counting dimension of the space of input sequences with respect to the product topology. In particular, for input sequences originating from an N in N_{\text{in}} -dimensional smooth dynamical system or their generic continuously differentiable observations, the box-counting dimension of the pullback attractor is bounded above by N in N_{\text{in}} . The results obtained here highlight the fact that, while a reservoir computer may possess a very high-dimensional state space, it exhibits effective low-dimensional dynamics. Our findings also partly explain why reservoir computers are successful in tasks such as attractor reconstruction and the computation of dynamic invariants like Lyapunov exponents and fractal dimensions. 1 1 footnotetext: Global Systems Institute, University of Exeter, Exeter, EX4 4QE, Devon, UK 1 Introduction Reservoir computing (RC) framework has offered a computationally efficient way to train recurrent neural networks by randomly generating the internal weights and keeping them fixed while focusing training only on the output layer [ 25 , 34 ] . This training procedure allows reservoir computers (RCs) to avoid the notorious vanishing and exploding gradient problem [ 39 ] , and is one of the main contenders for biologically plausible temporal learning [ 12 , 13 ] . Since a large class of RCs are universal approximators of fading memory mappings on left infinite input sequences [ 19 , 15 ] , they have achieve state-of-the-art performance on several benchmarks, including time-series classification [ 22 , 24 ] , systems identification [ 26 ] , adaptive filtering [ 51 ] , attractor reconstruction [ 33 ] , anticipating tipping points and critical transitions [ 38 , 37 ] , etc. Beyond the theory of universal approximations, it is unclear how RCs are able to achieve such performance on these tasks. In the past few years, efforts have been made to understand the decision-making process of such systems by studying them as input-driven dynamical systems [ 47 , 7 , 14 ] . In the case of classification problems with discrete-valued inputs, it has been shown, for example, in [ 47 , 8 ] that echo state networks (ESNs) trained with output feedback take advantage of stable fixed points and their relative locations to solve such tasks. As well as in the continuous output case, errors in ESNs trained on such tasks have been identified with the emergence of spurious/untrained attractors [ 14 , 8 ] . Transitions to such attractors may cause errors over short time steps in classification problems with discrete-valued inputs and over longer time steps in continuous output problems [ 14 ] . Even with such erroneous transitions, it is known, for example, in tasks such as attractor reconstruction, that RCs can maintain the general dynamics of the system they were trained on well beyond when their predictions start to diverge. This property of RCs allows them to learn dynamic invariants, such as Lyapunov exponents and fractal dimensions, of the system they were trained on [ 33 , 20 , 21 , 41 ] . Although it is known that the asymptotic dynamics of a reservoir computer synchronises with those of the driving input dynamics [ 30 ] , this synchronisation may not be sufficient for explaining the performance of reservoir computers [ 17 ] . In [ 20 ] , the authors conjectured that reservoir computers driven with generic continuously differentiable ( C^{1} ) observations of a smooth dynamical system on an m -dimensional compact manifold embed the dynamics of the dynamical system in its state space. This conjecture has been settled in the case of linear reservoirs [ 18 ] . If the conjecture holds in the general case, it will explain why reservoir computers achieve good performance in tasks such as attractor reconstruction. In this paper, we focus on establishing bounds on the dimension of the synchronised dynamics of the reservoir when driven with a certain class of input sequences. In particular, we investigate the box-counting dimension of the pullback attractor in the nonautonomous dynamical system arising from a large class of RCs of the form \displaystyle\mathbf{x}[n+1]=g(\mathbf{u}[n+1],\mathbf{x}[n]) (1) where the inputs \mathbf{u}[n]\in U\subseteq\mathbb{R}^{N_{\text{in}}} , the states \mathbf{x}[n]\in X\subseteq\mathbb{R}^{N_{r}} , and g is some continuous function on a compact set U (the space of inputs) and a compact set X (the state space). We show that under mild conditions on g , the box-counting dimension of the pullback attractor associated with input sequences coming from a finite-dimensional subset of U^{\mathbb{Z}} is bounded above by the box-counting dimension of this subset in the product topology. In the case that the input sequence comes from trajectories of a bi-Lipschitz homeomorphism \phi on U , we show that the box-counting dimension of the pullback attractor is bounded above by the box-counting dimension of U . Remarkably, this upper bound does not require g to satisfy the echo state property (ESP). However, if g satisfy ESP and the set U contains an open set, we show that the pullback attractor associated with all input sequences in U^{\mathbb{Z}} also contains an open set in the reservoir space and thus has dimension N_{r} . The remainder of this paper is organised as follows. Section 2 describes notations, background and some preliminary results used in this paper. The space U^{\mathbb{Z}} of input sequences is introduced in Section 2.2 together with the product topology on it. We then introduce the space of trajectories of a homeomorphism \phi on U as a subset of U^{\mathbb{Z}} . The dynamics of RNNs are presented in Section 2.3 and how they are trained via the reservoir computing paradigm. After introducing RNNs as nonautonomous (skew-product) dynamical systems in Section 2.4 , the notion of a pullback attractor and its characterisations are introduced. Section 3 contains the main contribution of this paper. It begins with the definition and properties of the box-counting dimension. We then give the Theorem of Vishik and Chepyzhov [ 9 ] , providing an upper bound for the box-counting dimension of the pullback attractor for finite-dimensional input sequence spaces (Theorem 3.1 ). We show in Section 3.1 that if g(u,x) is continuously differentiable on U\times X , then the corresponding nonautonomous dynamical system satisfies the conditions of Theorem 3.1 . We apply Theorem 3.1 to the space of trajectories of \phi on U , which is shown to have box-counting dimension equal to that of U (Theorem 3.4 ). In Section 3.2 , we shift

Read the complete text