Towards a Comprehensive Theory of Reservoir Computing

Denis Kleyko, Christopher J. Kymn, E. Paxon Frady, Amy Loutfi, Friedrich T. Sommer · 2025

Read Towards a Comprehensive Theory of Reservoir Computing by Denis Kleyko, Christopher J. Kymn, E. Paxon Frady, Amy Loutfi, Friedrich T. Sommer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-14484.txt.

Introduction

Denis Kleyko and four co-authors build a fuller theory for reservoir computing with echo state networks. In these systems a fixed recurrent network transforms input over time. A simple readout then uses the resulting pattern for memory or prediction tasks. Many variants exist, and choosing among them has lacked a firm theory.

The team show that perceptron theory can predict memory and accuracy across thirty network variants. Experiments confirm those predictions in detail. They use the theory to tune a network over its full parameter space. They also propose a new model whose readout needs no training. That model beats earlier untrained designs on the tests reported.

Catalog record

CREATORDenis Kleyko, Christopher J. Kymn, E. Paxon Frady, Amy Loutfi, Friedrich T. Sommer
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE72 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-14484

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards a Comprehensive Theory of Reservoir Computing Thanks: DK acknowledges funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie Actions (Grant No. 839179), the Swedish Strategic Research Foundation under the Future Research Leaders program (Grant No. FFL24-0111), and the Swedish Research Council under the Starting Grant program (Grant No. 2025-05421). The work of AL and DK was supported by Knut and Alice Wallenberg Foundation under the Wallenberg Scholars program (Grant No. KAW2023.0327). The work of CJK was supported by the Center for the Co-Design of Cognitive Systems (CoCoSys), one of seven centers in JUMP 2.0, a Semiconductor Research Corporation (SRC) program sponsored by DARPA, in addition to the NDSEG Fellowship, Fernström Fellowship, Swartz Foundation, and NSF Grants 2147640 and 2313149. FTS was supported by NSF Grant IIS1718991, NIH Grant R01-EB026955, and by the Kavli Foundation. This work was supported in part by the AFOSR under award number FA8655-25-1-7007. Thanks: D. Kleyko is with the AI, Robotics and Cybersecurity Center (ARC), and with the Department of Computer Science at Örebro University, 70281 Örebro, Sweden and also with Intelligent Systems Lab at RISE Research Institutes of Sweden, 16440 Kista, Sweden ( e-mail : [email protected] ). Thanks: C. J. Kymn is with the Redwood Center for Theoretical Neuroscience at the University of California at Berkeley, CA 94720, USA ( e-mail : [email protected] ). Thanks: E. P. Frady is with the Neuromorphic Computing Lab, Intel Labs, Santa Clara, CA 95054, USA. e-mail : [email protected] Thanks: A. Loutfi is with the AI, Robotics and Cybersecurity Center (ARC), and with the Department of Computer Science at Örebro University, 70281 Örebro, Sweden and also with Department of Science and Technology at Linköping University, 58183 Linköping, Sweden ( e-mail : [email protected] ). Thanks: F. T. Sommer is with the Redwood Center for Theoretical Neuroscience at the University of California at Berkeley, CA 94720, USA and also with the Neuromorphic Computing Lab, Intel Labs, Santa Clara, CA 95054, USA ( e-mail : [email protected] ). Denis Kleyko, Christopher J. Kymn, E. Paxon Frady, Amy Loutfi, Friedrich T. Sommer Abstract In reservoir computing, an input sequence is processed by a recurrent neural network, the reservoir, which transforms it into a spatial pattern that a shallow readout network can then exploit for tasks such as memorization and time-series prediction or classification. Echo state networks (ESN) are a model class in which the reservoir is a traditional artificial neural network. This class contains many model types, each with sets of hyperparameters. Selecting models and parameter settings for particular applications requires a theory for predicting and comparing performances. Here, we demonstrate that recent developments of perceptron theory can be used to predict the memory capacity and accuracy of a wide variety of ESN models, including reservoirs with linear neurons, sigmoid nonlinear neurons, different types of recurrent matrices, and different types of readout networks. Across thirty variants of ESNs, we show that empirical results consistently confirm the theory’s predictions. As a practical demonstration, the theory is used to optimize memory capacity of an ESN in the entire joint parameter space. Further, guided by the theory, we propose a novel ESN model with a readout network that does not require training, and which outperforms earlier ESN models without training. Finally, we characterize the geometry of the readout networks in ESNs, which reveals that many ESN models exhibit a similar regular simplex geometry as has been observed in the output weights of deep neural networks. Figure 1: Standard architecture of an echo state network. The current input to the network \mathbf{u}(n) is projected to the reservoir with N neurons using a random matrix \mathbf{W}^{\mathrm{in}} . The neurons within the reservoir are interconnected through another random matrix \mathbf{W} . The trainable readout network with weight matrix \mathbf{W}^{\mathrm{out}} computes the output \mathbf{y}(n) from the current reservoir state \mathbf{x}(n) . I Introduction Reservoir computing [ 1 , 2 ] is a framework for processing spatio-temporal input patterns in a neural network that consists of three stages depicted in Fig. 1 . The central stage is a recurrent neural network with fixed dynamics – the reservoir – that receives input through a fixed feed-forward matrix, the input projection matrix . The reservoir feeds into the third stage, the readout network , consisting of the readout matrix and a layer of output neurons, a feedforward network which often is trained in a supervised fashion. The sustained dynamic state of the reservoir acts like a working memory, storing past and present inputs. The readout matrix can decode this information, to perform tasks such as memorization, time-series prediction or classification. The reservoir can consist of a dynamical system with fixed parameters of any type that retains information about past inputs. For example, the reservoir can be a network of spiking neurons like in liquid state machines [ 3 ] , consist of neurons with quantized states as in neural prediction machines [ 4 ] , or even have recurrent weights with short-term plasticity, like in state-dependent networks [ 5 ] . Here we focus on the most common model for reservoir computing, echo state networks (ESN) [ 6 ] , in which the reservoir is an artificial neural network with a fixed recurrent random matrix, and the projection matrix is also random. ESN models differ not only in the type of recurrent neural network used as reservoir, but also in how the weights of the readout network are formed. In many ESN models, the readout matrix is optimized by supervised learning (i.e., teacher forcing [ 7 ] ), which is feasible if sufficient training data are available. In other types of ESNs, the readout matrix can be directly computed from the weights of the reservoir and input projection matrices, without any training [ 8 ] . The critical performance measure for a reservoir computing network is its memory capacity [ 9 ] , which describes the maximal amount of information that can be contained within the current state. However, memory capacity has only been estimated for simple ESN models, such as those with linear neurons and orthogonal recurrent matrices [ 10 , 11 ] . The goal of this article is to develop a theory to estimate the memory capacity for a large variety of ESN models, (Section III-B ), in particular, including those with nonlinear reservoir neurons. Two earlier studies of ESNs [ 8 , 12 ] used the framework of hyperdimensional computing or vector symbolic architectures [ 13 , 14 ] to dissect the computations in the reservoir into two parts: adding time tags to individual inputs, and superimposing them into the reservoir state. Using this dissection, it is possible to precompute the readout matrices, thereby avoiding the need of training. Further, these studies proposed using the statistics of inner products between the reservoir state and readout matrices for estimating the accuracy of the ESN. This type of ESN theory has been called the perceptron theory because the ESN readout network is identical to the earliest and extensively studied neural network, the Perceptron [ 15 , 16 , 17 , 18 , 19 , 20 , 21 ] . Here we demonstrate experimentally that the most general form of the perceptron theory can accurately predict the

Read the complete text