Covariance spectrum in nonlinear recurrent neural networks
Xuanyu Shen, Yu Hu · 2025
Read Covariance spectrum in nonlinear recurrent neural networks by Xuanyu Shen, Yu Hu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-05288.txt.
Introduction
Xuanyu Shen and Yu Hu extended in 2025 the theory of how activity patterns spread across a network of model neurons. Recordings of many neurons at once have made the shape of shared activity a key clue to how networks are wired. A classic result describes that shape for networks with simple, linear units. Real neurons are nonlinear, though, and can even behave chaotically, leaving a gap in the theory.
The pair show the same formulas hold for nonlinear networks if one swaps in an effective link strength. That strength blends the wiring weights with how strongly each unit responds. Across calm and chaotic regimes, the effective value stays near its critical point. The result hints the brain may sit near criticality without fine tuning.
Catalog record
| CREATOR | Xuanyu Shen, Yu Hu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 106 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-05288
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Covariance spectrum in nonlinear recurrent neural networks Xuanyu Shen, Yu Hu Abstract Advances in simultaneous recordings of large numbers of neurons have driven significant interest in the structure of neural population activity such as dimension. A key question is how these dynamic features arise mechanistically and their relationship to circuit connectivity. It was previously proposed to use the covariance eigenvalue distribution, or spectrum, which can be analytically derived in random recurrent networks, as a robust measure to describe the shape of neural population activity beyond the dimension (Hu and Sompolinsky 2022). Applications of the theoretical spectrum have broadly found accurate matches to experimental data across brain areas providing mechanistic insights into the observed low dimensional population dynamics (Morales et al. 2023). However, the empirical success highlights a gap in theory, as the neural network model used to derive the spectrum was minimal with linear neurons. In this work, we aim to close this gap by studying the covariance spectrum in networks with nonlinear neurons and under broader dynamical regimes including chaos. Surprisingly, we found that the spectrum can be precisely understood by equations analogous to the linear theory substituted with an effective recurrent connection strength parameter, that reflects both the connection weights and the nonlinearity of neurons. Across dynamical regimes, this effective connection strength provides a unified interpretation for the spectrum and dimension changes, and stays near the critical value in the chaotic regime without fine-tuning. These results further our understanding of nonlinear neural population dynamics and provide additional theoretical support for applying the covariance spectrum analysis in biological circuits. Keywords: neural population dynamics; recurrent neural network; network connectivity; nonlinear dynamics and chaos 1 Introduction Understanding the relationship between neural dynamics and the underlying network connectivity is a central question in theoretical neuroscience and is also of common interest to other fields studying complex and extensively interacting systems. In recent years, there have been significant developments in large-scale simultaneous recordings of neural activity [ 2 , 15 , 25 , 27 ] , which not only increase the quantity of data but also allow for new ways of analyzing population activity structures at the multi-dimensional level, rather than averaging across neurons. Examples of such analyses include principal component analysis (PCA) [ 27 ] , dimension [ 22 , 5 , 6 ] , neural manifold [ 7 ] , communication subspace [ 25 ] among others. This, in turn, has driven intense interest in theoretical studies aimed at understanding how these properties of population dynamics arise mechanistically and relate to the network connectivity structure in neural circuit models [ 18 , 23 , 4 , 11 , 9 ] . In particular, the eigenvalues of the covariance matrix of neural activity, or the covariance spectrum [ 11 ] , provide information about the shape of population activity beyond its dimension. This description of population dynamics is useful and has been applied to whole-brain neural activity to reveal a scale-invariance phenomenon [ 30 ] . Hu and Sompolinsky developed a theory [ 11 ] predicting the covariance spectrum in recurrent neural circuits, using simple models with random connections [ 26 , 16 ] or with second-order motifs [ 31 , 10 ] , and linear neuronal dynamics. The theory has been found to match well with experimental data, including recordings from various brain regions, recording methods, and animal species [ 11 , 19 , 3 ] . By fitting the theoretical spectrum [ 11 ] to human Magnetoencephalography (MEG) data across frequency bands, Calvo et al. found that the fitted recurrent strength parameter can be used to distinguish Parkinson’s disease patients from healthy controls [ 3 ] . However, the empirical success of the model covariance spectrum highlights a gap in theory, because biological neural networks are complex, and the assumptions made in [ 11 ] may not be sufficient to describe more complex nonlinear dynamics such as chaos [ 26 , 16 , 24 ] . It is technically challenging to determine correlations in chaotic states. For example, in the classic mean-field theory (MFT) of randomly connected networks, correlations between neurons were ignored [ 26 ] , as they vanish in large networks as O(1/\sqrt{N}) , where N is the number of neurons. In a recent breakthrough, Clark et al. derived the leading-order correlations in terms of the four-point function by developing a two-site cavity method [ 4 ] . These results allow for computing the second moments of the covariance matrix eigenvalues and, in turn, for determining the participation ratio dimension [ 17 , 22 ] of these randomly connected recurrent neural networks (RNNs) in chaotic states. In this paper, we aim to close the aforementioned theoretical gap by determining the covariance spectrum in nonlinear randomly connected RNNs. This work builds on and extends the results of [ 11 ] and [ 4 ] to gain a more detailed understanding of population dynamics beyond its dimension in chaotic states and provide theoretical backing for applying covariance spectrum analysis to real data. A key step is proposing a closed-form ansatz for the covariance matrix ( Eq. 3.3 ), motivated and supported by theoretical and numerical evidence, particularly through comparisons with networks with linear neurons [ 11 ] . The (frequency-dependent) covariance spectrum, for both firing rates and currents, is then derived from these covariance matrix expressions. To obtain a full picture of how the dynamics, described by the spectrum and dimension, change across dynamical regimes, especially during the transition to chaos [ 28 ] , we have incorporated independent external inputs modeled as white noise into the network model [ 24 ] . Importantly, we found that the changes in covariance spectrum and dimension can be understood in a unified manner in terms of an effective recurrent connection strength parameter g_{\text{eff}} ( Eq. 3.2 ). Moreover, we study how the gap between the dimension of firing rates and that of currents [ 4 ] changes along the transition to chaos and reveal a subtle relationship. The rest of the paper is organized as follows. In Section 2 , we introduce the recurrent network model and define the covariance matrix, participation ratio dimension, and covariance spectrum. In Section 3 , we propose the ansatz for the covariance matrix and discuss its numerical and theoretical validations. In Section 4 , we use the ansatz and its corollary to derive the covariance spectrum for firing rates and currents. In Section 5 , we study the behavior of the effective recurrent connection strength g_{\text{eff}} across dynamical regimes and investigate the relationship between the dimension gap and the transition to chaos. 2 Neural network model We study the dynamics of a recurrent network of N neurons, where the total current of neuron i , x_{i}(t) , evolves as \frac{dx_{i}(t)}{dt}=-x_{i}+\sum_{j=1}^{N}J_{ij}\phi(x_{j})+\xi_{i}(t),\quad i=1,2,\cdots,N. (2.1) The neurons are coupled by a random network (fixed during the neural dynamics) [ 26 , 24 , 11 , 4 ] , specified by the connectivity matrix \mathbf{J}=(J_{ij})_{1\leq i,j\leq N} , whose entries are drawn i.i.d. from a Gaussian distribution with zero mean and variance g^{2}/N . In particular, such a network is balanced on average in excitation and inhibition, and the parameter g controls the recurrent connection strength. The nonlinear neuronal activation