Attention-Enhanced Reservoir Computing as a Multiple Dynamical System Approximator

Felix Köster, Kazutaka Kanno, Atsushi Uchida · 2025

Read Attention-Enhanced Reservoir Computing as a Multiple Dynamical System Approximator by Felix Köster, Kazutaka Kanno, Atsushi Uchida (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-05852.txt.

Introduction

Felix Köster, Kazutaka Kanno, and Atsushi Uchida teach a reservoir computer to juggle several chaotic systems at once, in a 2025 paper. Reservoir computing predicts time series with a fixed random network and a trained readout, but one weight set usually fits one system. Adding an attention layer lets the readout reweight features on the fly.

The upgraded machine learned the Lorenz and Rössler systems, the Henon map, the Duffing oscillator, and the Mackey-Glass equation in a single training, then switched among them without retraining. It beat the plain reservoir on accuracy, forecast horizon, and the shape of the predicted chaos. Weather-like modeling and control stand to gain.

Catalog record

CREATORFelix Köster, Kazutaka Kanno, Atsushi Uchida
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-05852

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Attention-Enhanced Reservoir Computing as a Multiple Dynamical System Approximator Preprint: APS/123-QED Felix Köster, Kazutaka Kanno, Atsushi Uchida Abstract Reservoir computing has proven effective for tasks such as time-series prediction, particularly in the context of chaotic systems. However, conventional reservoir computing frameworks often face challenges in achieving high prediction accuracy and adapting to diverse dynamical problems due to their reliance on fixed weight structures. A concept of an attention-enhanced reservoir computer has been proposed, which integrates an attention mechanism into the output layer of the reservoir computing model. This addition enables the system to prioritize distinct features dynamically, enhancing adaptability and prediction performance. In this study, we demonstrate the capability of the attention-enhanced reservoir computer to learn and predict multiple chaotic attractors simultaneously with a single set of weights, thus enabling transitions between attractors without explicit retraining. The method is validated using benchmark tasks, including the Lorenz system, Rössler system, Henon map, Duffing oscillator, and Mackey-Glass delay-differential equation. Our results indicate that the attention-enhanced reservoir computer achieves superior prediction accuracy, valid prediction times, and improved representation of spectral and histogram characteristics compared to traditional reservoir computing methods, establishing it as a robust tool for modeling complex dynamical systems. I Introduction The combination of reservoir computing (RC) and other machine learning concepts has emerged as a promising field of information processing, utilizing the development of more rapid and efficient computational schemes Shen et al. 2017 . Central to this area is the concept of RC, a scheme that utilizes the dynamic properties of any dynamical system Jaeger 2001 ; Appeltant et al. 2011 ; Tanaka et al. 2019 ; Torrejon et al. 2017 ; Kotooka et al. 2021 . RCs leverage a fixed, randomly connected reservoir and train only the output layer, making them computationally efficient and suitable for a variety of tasks, including time-series prediction and pattern recognition Jaeger 2001 ; Tanaka et al. 2019 , while making hardware implementations easy and versatile. Chaotic time-series prediction is known to be challenging owing to its inherent sensitivity to small deviations Strogatz 2000 . A prime example of this challenge is weather forecasting, which is typically limited to short-term predictions due to the fundamentally chaotic nature of weather systems. The framework of RC has been widely implemented using a variety of physical devices, such as electronic circuits Appeltant et al. 2011 , spintronic devices Torrejon et al. 2017 , soft robots Nakajima et al. 2015 , and nanostructure materials Kotooka et al. 2021 , showing the broad application range. It also highlights that many reservoir computer can be leveraged in a wide area, where the utilization can often easily be accomplished via a simple measurement process. One of the advantages of RC is the easy implementation of the reservoir without high computational cost because input weights and network weights are randomly fixed, and only the output weights are trained using easy and fast ridge regression, thus, making the output layer linear. Therefore, classical RC is often limited due to inflexibility of the fixed output weight structure when faced with the complexities in chaotic time series Jaeger 2001 , while also having a bad scalability due to the characteristics of ridge regression and linear output layers. Advancements in neural network architectures, such as the attention mechanism, have opened new pathways for enhanced prediction accuracy and improved adaptability Vaswani et al. 2017 ; Bai et al. 2018 . To further enhance the predictive capabilities of RC, we proposed integrating the attention mechanism within a photonic setup Köster et al. 2024 . By incorporating an attention layer as the output layer, the system gets the capacity to prioritize specific readouts in each time step, potentially yielding a more dynamic comprehension of the data Bai et al. 2018 . In this study, we use an attention-enhanced reservoir computer (AERC) with an echo state network to demonstrate its capability to learn multiple attractors simultaneously with the same set of weights and to acquire the emergent ability to switch between learned attractors without explicit training. We perform chaotic time-series prediction by deploying five different benchmark tasks, i.e. prediction of the Lorenz system Lorenz 1963 , Rössler system Rössler 1976 , Henon map Henon 1976 , Duffing oscillator Duffing 1918 , and Mackey-Glass delay differential equation Mackey and Glass 1977 , showcasing the robustness and adaptability of the attention mechanism. These diverse tasks allow us to investigate how well the attention mechanism adapts to various types of chaotic dynamics and how effectively it improves prediction accuracy. By training the AERC on these five different tasks with a single set of weights, we demonstrate its ability to simultaneously learn and predict multiple attractors, a significant advancement over traditional RC. II Reservoir Computing and Attention Mechanism II.1 Classic Reservoir Computing Input \mathbf{x}_{l} Reservoir \mathbf{r}_{l} Output Layer \bar{\mathbf{y}}_{l} Attention Layer \mathbf{W}_{\text{att},l} Error ||\bar{\mathbf{y}}_{l}-\mathbf{y}_{l}||_{2} Training \nabla F(\textbf{W}_{\text{net}},\textbf{R}) Figure 1 : The float chart diagram of the AERC layout. Via the inputs reservoir states are collected. With the reservoir states and a randomly initialized attention layer an output is generated, which is then utilized in a gradient descent approach to optimize the attention layer. The dashed arrows show the training process, which are deactivated after training. RC is a framework for computation that utilizes a fixed, randomly connected, and high-dimensional dynamical system, called a reservoir, along with a trained readout layer to perform tasks such as time series prediction, pattern recognition, and classification. Mathematically, the state of the reservoir at time-step l , denoted by \mathbf{r}_{l} , is updated based on the previous state and the current input \mathbf{x}_{l} as follows: \mathbf{r}_{l}=\tanh(\mathbf{W}_{\text{res}}\mathbf{r}_{l-1}+\mathbf{W}_{\text{in}}\mathbf{x}_{l}+\mathbf{b}) Here, \mathbf{r}_{l} is the reservoir state vector in \mathbb{R}^{N\times 1} , where N represents the number of reservoir nodes or readouts. The matrix \mathbf{W}_{\text{res}} in \mathbb{R}^{N\times N} is the internal reservoir weight matrix, \mathbf{W}_{\text{in}} in \mathbb{R}^{N\times M} is the input weight matrix, and \mathbf{x}_{l} in \mathbb{R}^{M\times 1} is the input vector, with M representing the number of input dimensions. The vector \mathbf{b} in \mathbb{R}^{N\times 1} is the bias vector. The linear readout layer is trained to map the reservoir states to the desired output. The output \bar{\mathbf{y}}_{l} at the l -th data point is given by: \bar{\mathbf{y}}_{l}=\mathbf{W}_{\text{out}}\mathbf{r}_{l} where \mathbf{W}_{\text{out}} in \mathbb{R}^{T\times N} is the output weight matrix, with T representing the number of output dimensions. The feature matrix \mathbf{R} , used for training the readout layer, consists of reservoir states collected over time. Each row of \mathbf{R} represents a sample (a reservoir state at a particular time-step), and each column represents a feature (a particular node of the reservoir state vector): \mathbf{R}=\begin{bmatrix}\mathbf{r}_{1}^{\top}\\ \mathbf{r}_{2}^{\top}\\ \vdots\\ \mathbf{r}_{N}^{\top}\end{bmatrix}=\begin{bmatrix}r_{1,1}&r_{1,2}&\cdots&r_{1,L}\\ r_{2,1}&r_{2,2}&\cdots&r_{2,L}\\ \vdots&\vdots&\ddots&\vdots\\ r_{N,1}&r_{N,2}&\cdots&r_{N,L}\end{bmatrix} where L is the total number of measured time steps, and N is the number of reservoir nodes. The readout weights \mathbf{W}_{\text{out}} are typically trained using linear regression to minimize the error between the predicted output \bar{\mathbf{y}}_{l} and the desired output \mathbf{y}_{l} : \mathbf{W}_{\text{out}}=(\mathbf{R}^{\top}\mathbf{R}+\lambda\mathbf{I})^{-1}\mathbf{R}^{\top}\mathbf{Y} where \mathbf{Y} is the matrix of desired outputs, \lambda is a regularization parameter, and \mathbf{I} is the identity matrix. Throughout this paper, we fixed the spectral radius of the reservoirs to 0.9 and the average connection degree of the input matrix and topology matrix to 0.1 . System Total Time Sample Points Step Size Samples in one Lyapunov Time Lorenz 375

Read the complete text