Recurrent Self-Attention Dynamics: An Energy-Agnostic Perspective from Jacobians
Akiyoshi Tomihari [email protected], Ryo Karakida [email protected] · 2025
Read Recurrent Self-Attention Dynamics: An Energy-Agnostic Perspective from Jacobians by Akiyoshi Tomihari [email protected], Ryo Karakida [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19458.txt.
Introduction
Akiyoshi Tomihari and Ryo Karakida wrote this 2025 theory paper on self-attention, the core part of modern language models. Much past work studies attention layers through an energy score that falls as the state updates. That view needs strong limits, such as symmetry, which real models may not meet. The authors instead study the dynamics with no energy score assumed.
Their tool is the Jacobian matrix, which tracks how small changes in the state grow or fade. They show that the normalization step keeps the dynamics tame, and that Lyapunov exponents place the system near a critical state. That near-critical state, they argue, tracks strong inference, and they build training methods on the idea.
Catalog record
| CREATOR | Akiyoshi Tomihari [email protected], Ryo Karakida [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-19458
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Recurrent Self-Attention Dynamics: An Energy-Agnostic Perspective from Jacobians Akiyoshi Tomihari [email protected], Ryo Karakida [email protected] Abstract The theoretical understanding of self-attention (SA) has been steadily progressing. A prominent line of work studies a class of SA layers that admit an energy function decreased by state updates. While it provides valuable insights into inherent biases in signal propagation, it often relies on idealized assumptions or additional constraints not necessarily present in standard SA. Thus, to broaden our understanding, this work aims to relax these energy constraints and provide an energy-agnostic characterization of inference dynamics by dynamical systems analysis. In more detail, we first consider relaxing the symmetry and single-head constraints traditionally required in energy-based formulations. Next, to investigate more general SA architectures capable of oscillatory dynamics without necessarily admitting an energy function, we analyze the Jacobian matrix of the state. We reveal that normalization layers effectively normalize the Jacobian’s complex eigenvalues, forcing the dynamics close to a critical state. This significantly enhances inference performance. Furthermore, we utilize the Jacobian perspective to develop regularization methods for training and a pseudo-energy for monitoring inference dynamics. \restylefloat figure 1 Introduction The theoretical understanding of self-attention (SA), a central component of Transformer architectures ( Vaswani 2017 ) , has deepened in recent years, including phenomena such as rank collapse ( Noci et al. 2022 ) and expressive capacity ( Yun et al. 2020 ; Kajitsuka and Sato 2024 ; Kingma and Ba 2025 ) . One major line of research formulates attention mechanisms as processes that minimize explicit or implicit energy functions ( Ramsauer et al. 2021 ; Yang et al. 2022 ; Hoover et al. 2023 ) . Since these energy functions serve as potential functions for gradient flows or as Lyapunov functions, they offer convergence guarantees and provide intuitive explanations for behaviors such as clustering and rank collapse in recurrent SA architectures ( Geshkovski et al. 2023a ; Geshkovski et al. 2023b ; Geshkovski et al. 2024 ; Bruno et al. 2025 ) . They restrict attention dynamics to a hypersphere, typically as a result of normalization. This facilitates theoretical analysis and can yield well-behaved functional properties ( Castin et al. 2024 ) . However, energy-based formulations often rely on idealized assumptions and require architectural modifications. These include imposing constraints on the weight matrices, limiting the number of attention heads to one, and modeling state updates in the continuum limit. In addition, some architectures inspired by Hopfield networks replace SA with cross-attention ( Ramsauer et al. 2021 ) or require double softmax passes ( Hoover et al. 2023 ; Hu et al. 2025 ) . Although they can be effective for exploring a frontier of new architectures, their utility still remains limited for quantitatively understanding or improving existing, realistic SA models. In this work, we deepen the understanding of SA by extending energy-based analysis and employing a more general stability analysis from a dynamical systems perspective. • First, we revisit the energy-based formulation and partially relax traditional architectural constraints, such as symmetric weights and single-head assumptions, to better approximate realistic SA settings ( Section 4 ). These relaxed constraints provide insights into designing regularization methods, which we experimentally explore later in Section 6.2 . • To study a broader class of SAs more flexibly, we next analyze the Jacobian matrix of SA dynamics ( Section 5 ). The Jacobian approach is more general than the energy-based analysis (a.k.a. Lyapunov’s direct method) in the sense that it characterizes linear stability (a.k.a. Lyapunov’s indirect method) and enables us to detect non-stationary dynamics, including oscillations. We demonstrate that normalization layers, unique to discrete updates, play a critical role in stabilizing dynamics. Specifically, they effectively suppress the Jacobian’s spectral norm ( Proposition 5.1 ) and control oscillatory behaviors by normalizing the complex eigenvalues of the Jacobian ( Section 5.2 ). Empirically, we confirm that high-performance SA models exhibit a maximum Lyapunov exponent close to zero, suggesting that rich non-stationary inference dynamics emerge at the boundary between convergence and instability. • Finally, we investigate test-time scaling of inference through the lens of Jacobians. We show that regularizing the spectral norm of weight matrices in SA improves performance ( Section 6.2 ), and that the Jacobian offers an interpretation of the pseudo-energy proposed in prior work, linking it to large eigenvalues ( Section 6.3 ). Thus, our work broadens the dynamical understanding of SA through dynamical systems analysis, laying a foundation for further exploration of realistic SA architectures. 2 Related work Energy-based understanding. Geshkovski et al. 2023a ; Geshkovski et al. 2023b ; Geshkovski et al. 2024 ; Karagodin et al. 2024 ; Bruno et al. 2025 formulated recurrent SA dynamics as interactions among tokens (“particles”), enabling theoretical analysis of phenomena such as meta-stable clustering and rank-one collapse. Their continuous-time dynamics monotonically decrease an energy (Lyapunov) function, typically requiring constraints like single-head attention, hyperspherical token states, and symmetric weights. Yang et al. 2022 similarly interpreted Transformers as alternating minimization of energy functions, though with stricter conditions on step sizes and fixed-point proximity. Ramsauer et al. 2021 formalized the attention mechanism as modern Hopfield networks, and Hoover et al. 2023 ; Hu et al. 2025 further developed energy functions for Transformers. We do not address approaches based on the Hopfield networks, as they require architectural modifications distinct from standard self-attention. Jacobian-based analysis. The Jacobian of state updates is fundamental for characterizing neural network dynamics. For example, it has been used to analyze edge-of-chaos behavior for stable signal propagation and gradient control ( Boedecker et al. 2012 ; Poole et al. 2016 ; Pennington et al. 2017 ) , and to investigate discrete-time stability in dynamics with anti-symmetric matrices ( Haber and Ruthotto 2017 ; Chang et al. 2019 ) . Several studies have explored Jacobian-based regularization, including for generalization ( Yoshida and Miyato 2017 ) and for continual learning ( Lewandowski et al. 2025 ) . Regarding SA specifically, Noci et al. 2022 analyzed Jacobians to explain rank collapse, while Castin et al. 2024 evaluated their spectral properties mathematically. In this work, we use Jacobian analysis to understand inference dynamics in realistic SAs and also employ them as regularizers and performance indicators. Looped architectures. Looped architectures in Transformers have been studied since their introduction by Dehghani et al. 2018 . Yang et al. 2024 ; Giannou et al. 2023 showed that looped Transformers can learn algorithmic tasks, and Saunshi et al. 2025 further demonstrated their effectiveness in enhancing reasoning via strong inductive bias. Geiping et al. 2025 increased the number of loop iterations to improve performance on reasoning benchmarks, and Bansal et al. 2022 showed that looped architectures generalize to harder problems. Miyato et al. 2025 proposed artificial Kuramoto oscillatory neurons (AKOrN), a looped architecture that successfully solves tasks in a neuroscience-inspired manner, demonstrating strong empirical results in unsupervised object discovery, adversarial robustness, calibrated uncertainty quantification, and reasoning. Weight tying in ALBERT ( Lan et al. 2020 ) and fixed-point computation in equilibrium models ( Bai et al. 2019 ) are also interpreted as looped architectures. For a more detailed overview of previous work, see the extended related work in Section C . 3 Preliminaries Notations. For a matrix \bm{A} , we use the subscripts A_{[i,j]} , \bm{A}_{[i,:]} , and \bm{A}_{[:,j]}