Transformer Dynamics: A neuroscientific approach to interpretability of large language models
Jesseba Fernando, Grigori Guitchounts · 2025
Read Transformer Dynamics: A neuroscientific approach to interpretability of large language models by Jesseba Fernando, Grigori Guitchounts (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12131.txt.
Introduction
Jesseba Fernando and Grigori Guitchounts wrote a 2025 paper that studies large language models the way brain scientists study brains. They treat the model's residual stream, the channel running through its layers, as a moving system over time. Unit activity is then traced layer by layer.
Single units keep a steady identity across layers, a surprise in an unprivileged basis. Activity speeds up and thickens as layers pass. In squashed views, the stream follows a curved path with attractor like bends early on. The authors frame this as a start on a neuroscience of AI, joining hard theory to big data.
Catalog record
| CREATOR | Jesseba Fernando, Grigori Guitchounts |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-12131
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Transformer Dynamics: A neuroscientific approach to interpretability of large language models Jesseba Fernando, Grigori Guitchounts Abstract As artificial intelligence models have exploded in scale and capability, understanding of their internal mechanisms remains a critical challenge. Inspired by the success of dynamical systems approaches in neuroscience, here we propose a novel framework for studying computations in deep learning systems. We focus on the residual stream (RS) in transformer models, conceptualizing it as a dynamical system evolving across layers. We find that activations of individual RS units exhibit strong continuity across layers, despite the RS being a non-privileged basis. Activations in the RS accelerate and grow denser over layers, while individual units trace unstable periodic orbits. In reduced-dimensional spaces, the RS follows a curved trajectory with attractor-like dynamics in the lower layers. These insights bridge dynamical systems theory and mechanistic interpretability, establishing a foundation for a ”neuroscience of AI” that combines theoretical rigor with large-scale data analysis to advance our understanding of modern neural networks. 1 Network Science Institute, Northeastern University 2 Independent [email protected], [email protected] 1 Introduction Artificial intelligence models—particularly modern deep learning systems—have scaled in both size and capability at an astonishing rate ( Bahri et al., 2024 ) . Today’s large language models (LLMs), vision models, and other predictive models (e.g. recommender systems, weather prediction, navigation, etc) are operating in the real world. Yet, despite their ubiquity, we lack a comprehensive understanding of how these models work, where understanding is meant roughly as to be able to predict how a particular change to the input or to the model would affect its output in a wide range of cases. Our lack of understanding around such systems raises myriad concerns about their safety, fairness, and whether they might pose an existential risk to humanity ( Amodei et al., 2016 ; Bengio et al., 2024 ) . Like the brain, deep neural networks are complex systems composed of billions of parameters interacting in highly non-linear ways. Systems with even a small number of such interacting elements can give rise to emergent behaviors that are unpredictable from a strictly bottom-up perspective ( Lorenz, 1963 ) . Consequently, it is not surprising that existing methods for investigating the workings of these models have yielded only fragmentary insights. Current approaches in mechanistic interpretability often focus on identifying discrete circuits within neural networks—sub-networks or groups of neurons that implement particular functions ( Heimersheim & Nanda, 2024 ; Singh et al., 2024 ) . While these circuit-based approaches have provided some explanatory power, they tend to mirror pitfalls of early approaches to understanding the brain. One historical example in neuroscience was the concept of ”grandmother cells,” which posited that the unit of representation in the brain may be individual neurons that encode highly specific concepts (like a single neuron firing selectively for one’s grandmother) ( Plaut & McClelland, 2010 ) . This idea, together with sparse coding—the notion that only a small fraction of neurons are active at any one time, and that representations are distributed among populations of cells ( Olshausen & Field, 2004 ) —led to a wave of work around sparse distributed representations as a way to explain how the brain encodes information. Yet, while artificially sparsifying model activations with sparse autoencoders (SAEs) has yielded model-specific information about which sets of activations represent monosemantic concepts, many questions about how the models compute remain unanswered ( Wattenberg & Viégas, 2024 ) . Figure 1: Transformer residual stream (RS) activations grow dense over the layers, are highly correlated among successive layers, and exhibit nonstationary dynamics. A: Activations of the transformer RS were captured before layernorm and the attention operation (pre-Attn) and before the MLP at each layer of Llama 3.1 8B, resulting in \mathbf{64\times 4096} . ’layers’ by ’units’. Activations were analyzed at the last token position for data samples from the wikitext-2-raw-v1 dataset unless otherwise noted. B: Mean activations across N=1000 samples. C: Correlations of activations for unit u between layer l and l+1 over data samples. For most units, correlations among successive layers increase over the layers. D: Histogram of correlations across layers for each unit. Despite the residual stream not having privileged basis, activations of most units are highly correlated from layer to layer. E: Cosine similarity among pairs of RS vectors \mathbf{h}_{l}^{Attn}\rightarrow\mathbf{h}_{l}^{MLP} (green) and \mathbf{h}_{l}^{MLP}\rightarrow\mathbf{h}_{l+1}^{Attn} (blue). F: Velocity V of the RS vectors. G: Mutual information (MI) among pairs of activations for unit u between layer l and l+1 over data samples. H: MI over the layers, averaged across units in the RS. In neuroscience, a recent promising approach to interpret neural encoding of information comes from dynamical systems ( Shenoy et al., 2013 ; Barack & Krakauer, 2021 ; Vyas et al., 2020 ) . Treating the activity of populations of neurons as a time-evolving dynamical system has shed light on how such populations collectively implement sensory perception, compute cognitive variables, and produce behavior. Instead of aiming to explain the representations of single neurons—or even snapshots in time of the activities of populations—these dynamical approaches aim to understand how network-wide activity evolves over time to generate complex outputs. For example, in the motor system, preparatory activity appears to guide the population to an appropriate pre-movement state; and in some cases these states are attractors that are robust to noise ( Inagaki et al., 2019 ) . Dynamical approaches have yielded insights into the computations in recurrent artificial networks as well ( Maheswaranathan et al., 2019 ) . While transformers do not have inherent time-evolving dynamics like recurrent networks, some have examined their activations and treated them as dynamically-evolving systems ( Geshkovski et al., 2024 ; Lu et al., 2019 ; Hosseini & Fedorenko, 2023 ; Lawson et al., 2024 ) . Specifically, the residual stream, which is updated linearly after each layer’s attention and MLP operations, can be considered as a dynamic system that evolves over the layers. Lu et al. proposed that the transformer residual stream be considered as an ordinary differential equation (ODE) of multiple particles moving through space (i.e. across layers) and influenced by convection (external forces) and diffusion (internal forces among particles); their main contribution was proposing the Strang-Marchuk splitting scheme to replace Euler’s method in approximating the ODE. Geshkovski et al. similarly treat the transformer as interacting particle systems, with dynamically-interacting particles (i.e. tokens) described as flows of probability measures on the unit sphere. This work aims to re-envision the study of mechanistic interpretability (MechInterp) through the lens of dynamical systems, inspired by this approach’s success in neuroscience and driven by that field’s integration of theory and large-scale data analysis. As such, we would like to term this new subset of MechInterp ”the neuroscience of AI”. Our key contributions are as follows: 1. We demonstrate that individual units in the residual stream maintain strong correlations across layers, revealing an unexpected continuity despite the RS not being a privileged basis. 2. We characterize the evolution of the residual stream, showing that it systematically accelerates and grows denser as information progresses through the network’s layers. 3. We identify a sharp decrease in mutual information during early layers, suggesting a fundamental transformation in how the network processes information. 4. We discover that individual residual stream units trace unstable periodic