How Long Does Infinite Width Last? Signal Propagation in Long-Range Linear Recurrences

Mariia Seleznova · 2026

Read How Long Does Infinite Width Last? Signal Propagation in Long-Range Linear Recurrences by Mariia Seleznova (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05113.txt.

Introduction

Mariia Seleznova asks how long the infinite-width approximation stays trustworthy in recurrent models as sequences grow long. Signal propagation theory usually assumes unlimited width, yet modern recurrent models run at finite width over very long inputs. Seleznova derives exact finite-width formulas for hidden-state signal energies in linear recurrences started from complex Gaussian weights.

Those formulas reveal three regimes set by how depth compares with the square root of width. Below that scale the infinite-width picture holds, near it a new joint scaling limit appears, and far above it finite-width effects dominate. The results pinpoint where standard initialisation schemes such as Glorot start to become unstable.

Catalog record

CREATORMariia Seleznova
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.05113

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

How Long Does Infinite Width Last? Signal Propagation in Long-Range Linear Recurrences Mariia Seleznova Abstract We study signal propagation in linear recurrent models at finite width. While existing signal propagation theory relies predominantly on the infinite-width limit, it remains unclear for how long that approximation remains accurate when recurrent depth t t grows jointly with width n n . This question is especially relevant for modern recurrent sequence models, whose natural operating regime involves long input sequences, i.e., large t t . We derive exact finite-width formulas for the hidden state signal energies in linear recurrences under complex Gaussian initialization. Using these formulas, we identify the joint depth–width scaling regimes that govern signal propagation: (i) a subcritical regime t = o ⁡ ( n ) t=o(\sqrt{n}) , in which the infinite-width approximation remains valid; (ii) a critical regime t ∼ c ​ n t\sim c\sqrt{n} , in which non-negligible deviations from infinite-width predictions appear and a nontrivial joint scaling limit emerges; and (iii) a supercritical regime t ≫ n t\gg\sqrt{n} , in which finite-width effects dominate. Thus, our results pinpoint the precise recurrent depth scale at which infinite-width theory breaks down in long-range linear recurrences. In turn, this shows when standard initialization schemes, such as Glorot, become unstable. More broadly, our results demonstrate that finite-width effects accumulate more rapidly with depth in recurrent models than in feedforward ones, leading to qualitatively different signal propagation behavior. 1 Introduction Understanding how signals propagate through the layers of a neural network (NN) is a foundational problem in deep learning (DL) theory. When signals are repeatedly transformed by many layers of a deep network, their magnitudes may progressively grow or decay, eventually leading to numerical failure. This phenomenon is widely known as the vanishing/exploding signals problem , and it poses a fundamental obstacle to training deep NNs ( 5 ; 30 ) . Ensuring stable signal propagation has therefore long been a central design principle in DL methods, motivating widely used initialization schemes ( 13 ; 23 ; 33 ) and architectural choices ( 3 ; 24 ; 25 ) . Despite the foundational nature and long history of this question, the theoretical understanding of signal propagation in DL models remains limited. Much of the existing literature relies on the infinite-width limit ( 33 ; 31 ; 42 ; 41 ) , where the network’s width n (i.e, the number of neurons per layer) tends to infinity. In this regime, mean-field-type approximations yield tractable descriptions of signal statistics and lead to elegant signal propagation theories for many DL architectures ( 42 ; 41 ) , including recurrent models ( 6 ; 12 ; 1 ) . Realistic networks, however, operate at finite width. In this setting, even small deviations from the infinite-width limit can accumulate over many layers or recurrent steps, eventually rendering infinite-width predictions inaccurate in sufficiently deep architectures or long-range recurrences ( 32 ; 21 ; 19 ; 35 ; 4 ) . This leads to the central question of this paper: How long does the infinite-width approximation remain accurate in recurrent models? For fully-connected feedforward networks at random initialization, the answer is relatively well understood. The deviations from infinite-width behavior become significant when depth scales proportionally with width, i.e., L\sim cn , where L is the number of layers and n is the layer width ( 21 ; 19 ; 35 ; 22 ) . That is, for fully connected feedforward networks, the critical depth scale at which finite-width effects begin to qualitatively alter signal propagation is by now known. By contrast, much less is known for recurrent networks. Although existing work suggests that finite-width corrections behave differently in recurrent and feedforward architectures ( 34 ; 14 ) , their precise effect for long-range recurrent signal propagation remains unclear. Intuitively, finite-width effects accumulate faster in recurrences, since the same weight matrix is applied repeatedly and its dominant spectral directions are therefore reinforced across recurrent steps t . Bar et al. (2025) ( 4 ) take a first step toward formalizing this intuition in linear recurrent sequence models with Gaussian initialization, showing that accumulated finite-width effects cause signal explosion when t\gg\sqrt{n} . However, their argument gives only a crude lower bound on the signal size. Thus, even for linear recurrences at random initialization, the sharp depth scale at which infinite-width predictions break down has remained unknown. This gap is especially striking from the perspective of modern long-range sequence modeling, where recurrence-based architectures—such as state-space models (SSMs) ( 18 ; 15 ; 8 ) and linear recurrent units (LRUs) ( 29 ) —are actively explored as computationally efficient alternatives to transformers for long-context tasks. These models have recently achieved competitive empirical performance while retaining the computational benefits of linear recurrence. Since they are designed for long-range tasks, their natural operating regime is that of long input sequences, or equivalently large recurrent depth t . Yet this is precisely the regime in which signal propagation theory remains weak. This paper helps close this gap by characterizing finite-width signal propagation in randomly-initialized linear recurrences and identifying the recurrent depth scaling regimes in which the infinite-width approximation remains accurate and those in which it fails. We first specify the considered setting in Section 1.1 , and then summarize the main contributions in Section 1.2 . 1.1 Problem Setting Our main object of interest is the linear recurrent unit (LRU), which has recently re-emerged as a building block for long-range sequence models ( 29 ) . On the way, our analysis also yields results for the simple linear recurrent network without sequential inputs. Thus, the considered models are: \displaystyle\text{{Linear recurrent unit (LRU):}}\qquad\mathbf{h}_{\textup{LRU}}^{(t)} \displaystyle:=\mathbf{W}\mathbf{h}_{\textup{LRU}}^{(t-1)}+\mathbf{x}^{(t)}=\sum_{k=0}^{t}\mathbf{W}^{k}\mathbf{x}^{(t-k)},\ t\geq 1, (1) \displaystyle\text{{Linear recurrent network (RNN):}}\qquad\mathbf{h}_{\textup{RNN}}^{(t)} \displaystyle:=\mathbf{W}\mathbf{h}_{\textup{RNN}}^{(t-1)}=\mathbf{W}^{t}\mathbf{x}^{(0)},\ t\geq 1. (2) Here \mathbf{h}^{(t)}_{\cdot}\in\mathbb{C}^{n} is the hidden state vector at step t\in\mathbb{N} , (\mathbf{x}^{(0)},\dots,\mathbf{x}^{(t)}) is the input sequence, where \mathbf{x}^{(i)}\in\mathbb{R}^{n} for all 0\leq i\leq t , and \mathbf{W}\in\mathbb{C}^{n\times n} is the weight matrix. We set \mathbf{h}_{\cdot}^{(0)}=\mathbf{x}^{(0)} in both models. The weight matrix \mathbf{W} is initialized with i.i.d. proper complex Gaussian entries of variance 1/n , namely \mathbf{W}_{ij}\sim\mathcal{N}_{\mathbb{C}}(0,1/n),\quad\text{i.i.d. across }i,j=1,\dots,n. (3) The main results in this work are formulated for complex weights. This choice is natural for two reasons. First, complex-valued linear recurrences are used in modern recurrent sequence models 29 ; 16 , since complex eigenvalues provide a convenient representation of oscillatory dynamics. Second, the complex Gaussian ensemble admits a particularly clean analysis, which allows us to derive sharp finite-width results for signal propagation. At the same time, our analysis also yields a sufficient condition under which linear recurrences with real Gaussian weights deviate from the infinite-width predictions. Our main quantitative objects are the normalized expected squared norms of the hidden states \mathbf{h}_{\mathrm{LRU}}^{(t)} and \mathbf{h}_{\mathrm{RNN}}^{(k)} , which we call signal energies . They are defined as follows: \textbf{Signal energies}:\quad S_{n,t}:=\frac{1}{n}\mathbb{E}\bigl[\|\mathbf{h}_{\mathrm{LRU}}^{(t)}\|_{2}^{2}\bigr],\qquad Q_{n,k}:=\frac{1}{n}\mathbb{E}\bigl[\|\mathbf{h}_{\mathrm{RNN}}^{(k)}\|_{2}^{2}\bigr], (4) where the expectation is taken jointly over the weights \mathbf{W} and the inputs (\mathbf{x}^{(0)},\dots,\mathbf{x}^{(t)}) . The same quantities are studied in classical works on signal propagation at infinite width ( 31 ; 33 ) . Notably, standard Gaussian initialization schemes—such as He ( 23 ) and Glorot ( 13 ) —are designed to stabilize these signal energies across layers at infinite width, so that Q_{\infty,k}\approx Q_{\infty,k+1} . However, in this work we explicitly

Read the complete text