Dynamics of the Transformer Residual Stream: Coupling Spectral Geometry to Network Topology

Jesseba Fernando, Grigori Guitchounts · 2026

Read Dynamics of the Transformer Residual Stream: Coupling Spectral Geometry to Network Topology by Jesseba Fernando, Grigori Guitchounts (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-14258.txt.

Introduction

Fernando and Guitchounts map how computation moves through layers of three large language models using full Jacobian eigendecomposition. Depth is treated as discrete time and the stream as a dynamic system, with each layer's nonlinear update locally linear. Prior work used scalar sums or rough linear fits, leaving the spectral shape of trained models unknown. Training installs a steady spectral slope through depth, from rotation-heavy early layers to near-symmetric late layers. A growing low-rank choke point funnels nudges into a small share of the stream's widths.

Catalog record

CREATORJesseba Fernando, Grigori Guitchounts
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62584
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.14258

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Dynamics of the Transformer Residual Stream: Coupling Spectral Geometry to Network Topology Jesseba Fernando, Grigori Guitchounts Abstract Large language models are remarkably capable, yet how computation propagates through their layers remains poorly understood. A growing line of work treats depth as discrete time and the residual stream as a dynamical system, where each layer’s nonlinear update has a local linear description. However, previous analyses have relied on scalar summaries or approximate linearizations, leaving the full spectral geometry of trained LLMs unknown. We perform full Jacobian eigendecomposition across three production–scale LLMs and show that training installs a monotonic spectral gradient through depth—from non-normal, rotation-dominated early layers to near–symmetric late layers—together with a cumulative low-rank bottleneck that funnels perturbations into a small fraction of the residual stream’s effective dimensions. Our experiments reveal that this gradient and the dimensional collapse are learned rather than architectural, and is largely dissolved when structured non-normality is removed. We further show that the topological positioning of graph communities predicts whether the Jacobian amplifies or suppresses them, with the sign of the coupling determined by the local operator type, a relationship absent at initialization. These results map a learned spectral geometry in LLMs that links perturbation propagation and compression to the network’s functional topology. 1 Introduction How computation propagates through a neural network remains largely unknown despite being one of the driving principles behind mechanistic interpretability. A growing toolkit addresses this question for transformers: circuit analysis traces computations through specific attention heads and MLP blocks ( 22 ; 24 ; 19 ) ; sparse autoencoders decompose activations into interpretable features ( 5 ; 4 ) ; probing classifiers and residual-stream readouts test what information is linearly accessible at each layer ( 1 ; 3 ) . These methods have been productive at identifying what individual components represent. But how these components come to be, or how a perturbation at one layer becomes amplified, compressed or redirected as it passes through subsequent layers, is an open question. Dynamical systems theory is a natural formalism for this question, and viewing layers as discrete time steps has precedent in work that treats transformers as ODE solvers or interacting-particle systems ( 20 ; 12 ) , and in empirical analyses of residual-stream trajectories through depth ( 15 ; 18 ; 9 ) . Each transformer block updates a shared residual stream, \mathbf{h}_{\ell+1}=\mathbf{h}_{\ell}+f_{\ell}(\mathbf{h}_{\ell}) , making depth a form of discrete time and the per-layer Jacobian J_{\ell}=\partial\mathbf{h}_{\ell+1}/\partial\mathbf{h}_{\ell} the local linear description of the dynamics. Theoretical literature on signal propagation has characterized these Jacobians at random initialization, establishing order-chaos phase boundaries ( 26 ; 30 ) and conditions for dynamical isometry ( 25 ) , but in most cases have been applied only at initialization or reduce the Jacobian to scalar summaries. Recent empirical work has begun to probe Jacobian-like objects in trained transformers—population-level linear maps ( 11 ) , detached Jacobians with frozen nonlinearities ( 14 ) , residual Jacobians correlated with benchmark performance ( 2 ) , Dynamic Mode Decomposition on averaged states ( 16 ) —and each independently finds signatures of progressive dimensional funneling through depth. These approaches use approximate or aggregate linearizations rather than the true input-space Jacobian. As a result, the full spectral geometry of trained production-scale Jacobians—the eigenvalue distribution, the non-normality structure, the balance of expanding and contracting modes, the rotational dynamics encoded in complex eigenvalues—remains uncharacterized. A separate question concerns the relationship between a network’s computational topology and its dynamics . Prior work on modularity in neural networks has focused on the weight graph—showing that trained networks are more “clusterable" than at initialization ( 10 ) or extracting communities from weight connectivity patterns ( 34 ) . In transformers, 7 framed computation as independent components “reading from” and “writing to” a shared residual stream. Whether this additive structure gives rise to mesoscale community organization in the residual stream’s activation correlations—and whether such structure is architectural or learned—has not been tested at production scale. We address these gaps by computing exact Jacobians of three production-scale LLMs—Llama 3.1 8B, OLMo 3 7B, and Gemma 4 E4B—performing full eigendecomposition and singular value analysis at every layer. For OLMo, we repeat this at three training checkpoints: step 0 (random initialization), step 471k, and step 1.41M (final pretraining step). Because step 0 shares the trained model’s architecture, depth, and wiring while varying only the weights, it serves as a null model: findings present at step 0 are architectural; findings absent at step 0 and emergent at trained checkpoints are attributed to training. We bridge these dynamics to activation-graph topology by constructing activation-correlation graphs and running community detection to test agreement with the Jacobian dynamics across the three checkpoints noted above. Our contributions are: 1. Full spectral characterization of trained production-scale Jacobians. We find that per-layer Jacobians organize into depth regimes with a monotonic non-normality gradient—from near-rotational early layers to near-symmetric late layers. Approximately, 98% of eigenvalues appear as complex conjugate pairs. Composing Jacobians across depth reveals that perturbations are funneled into a handful of effective channels ( \sim 4-40 ) across all three architectures. 2. Architecture-versus-training decomposition via OLMo’s step-0 null model. Depth regimes and adjacent-layer subspace decoupling are largely architectural, while the non-normality gradient and dimensional funneling are installed by training. Using Schur decomposition to zero the non-normality of the Jacobian reveals disruptions that track whether rank collapse is preserved, indicating that non-normal structure is central to the computations carried out by the residual stream. 3. Per-unit 1 1 1 We use “unit” throughout in the neuroscientific sense of a recording channel: the i -th coordinate of \mathbf{h}_{\ell}\in\mathbb{R}^{d} . We avoid “neuron” because residual stream coordinates do not correspond to discrete computational elements but are bookkeeping channels of a shared workspace ( 7 ) . coupling between community topology and Jacobian dynamics. Units that bridge multiple activation-graph communities (boundary nodes) are preferentially amplified or suppressed by the Jacobian, with the sign governed by operator type: boundary units are amplified at near-symmetric layers and de-amplified at non-normal ones. This coupling is absent at initialization and emerges monotonically over training ( 0\to 14\to 24 FDR-significant layers across OLMo checkpoints), establishing it as a learned property even though the mesoscale community structure itself is partly architectural. 2 Training reshapes per-layer Jacobians into a non-normality gradient and a dimensional bottleneck We treat each transformer block as one step of a discrete dynamical system on the residual stream and ask what its local linear description—the per-layer Jacobian J_{\ell}=\partial\mathbf{h}_{\ell+1}/\partial\mathbf{h}_{\ell} —looks like in trained transformers. We compute exact per-sample Jacobians at every sub-layer boundary on 1,000 WikiText-2 ( 21 ) samples for three architectures (Llama 3.1 8B, OLMo 3 7B, Gemma 4 E4B) and, for OLMo, at three pre-training checkpoints (step 0, 471k, 1.41M). Step 0 shares the trained model’s architecture, depth, and wiring while varying only the weights, so it serves as an architectural null. Computational and statistical details, and the formal definitions of the spectral quantities used below, are in Appendix A . A first basic observation cuts across every model and every layer: about 98% of the eigenvalues of the mean Jacobian come as complex conjugate pairs (visualized for Llama in Figure 1 ; full per-layer densities for all three models in Figure S1 ). Each

Read the complete text