The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
Matteo Saponati, Pascal Sager, Pau Vilimelis Aceituno, Thilo Stadelmann, Benjamin Grewe · 2025
Read The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training by Matteo Saponati, Pascal Sager, Pau Vilimelis Aceituno, Thilo Stadelmann, Benjamin Grewe (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-10927.txt.
Introduction
Matteo Saponati and four co-authors analyze the structure of self-attention in Transformer models in this 2025 paper. Self-attention is the core mechanism of these models, yet how information settles into its matrices during training was unclear. The team built a mathematical frame that derives the rules behind the weight updates. That frame let them predict how the training style shapes the matrices.
Training that reads both ways, they show, induces symmetry in the weights. Training that predicts the next token instead creates direction and column dominance. Tests confirmed both patterns across several models, among them GPT and LLaMA3, and across text, vision, and audio inputs. A practical payoff followed: starting encoder models with symmetric weights improved their language task scores. The authors offer the frame as a clearer view of how these models store what they learn.
Catalog record
| CREATOR | Matteo Saponati, Pascal Sager, Pau Vilimelis Aceituno, Thilo Stadelmann, Benjamin Grewe |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 125 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-10927
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training Matteo Saponati, Pascal Sager, Pau Vilimelis Aceituno, Thilo Stadelmann, Benjamin Grewe Abstract Self-attention is essential to Transformer architectures, yet how information is embedded in the self-attention matrices and how different objective functions impact this process remains unclear. We present a mathematical framework to analyze self-attention matrices by deriving the structures governing their weight updates. Using this framework, we demonstrate that bidirectional training induces symmetry in the weight matrices, while autoregressive training results in directionality and column dominance. Our theoretical findings are validated across multiple Transformer models — including ModernBERT, GPT, LLaMA3, and Mistral — and input modalities like text, vision, and audio. Finally, we apply these insights by showing that symmetric initialization improves the performance of encoder-only models on language tasks. This mathematical analysis offers a novel theoretical perspective on how information is embedded through self-attention, thereby improving the interpretability of Transformer models. 1. Introduction † † Correspondence: [email protected] , [email protected] Transformer models now achieve state-of-the-art performance across a wide range of tasks and domains [ Radford et al., 2019 , Dosovitskiy et al., 2021 , Radford et al., 2023 ] . Despite their success, the internal mechanisms governing their decision-making processes remain poorly understood, raising concerns regarding model alignment, reliability, and safety [ Wang et al., 2023 , Yao et al., 2024 ] . A key challenge in understanding these models is unraveling the structures of self-attention, which is essential to Transformer architectures. Current literature largely overlooks the nature of the self-attention weight matrices during autoregressive training, where the model predicts the next token in a sequence given previous ones [ Radford et al., 2019 , Black et al., 2021 , Touvron et al., 2023 ] and bidirectional training, where the model predicts a missing token given the full sequence [ Devlin et al., 2019 , Bao et al., 2022 , Warner et al., 2024 ] . Understanding self-attention requires answering two fundamental questions: How can we interpret the structures learned in the self-attention matrices? What is the impact of different objective functions on these matrices? Previous work used sparse auto-encoders to identify interpretable features [ Huben et al., 2024 , Bricken et al., 2023 ] , circuit analysis to interpret Transformer components [ Olah et al., 2020 , Elhage et al., 2021 , Olah, 2022 ] , and techniques like the logit lens to analyze self-attention mechanisms [ Geva et al., 2021 , Dar et al., 2023 ] (for a detailed discussion, see Section 5 ). However, these methods do not reveal the structural patterns in self-attention matrices or the transformations they encode. Crucially, how autoregressive and bidirectional training shape specific weight structures remains unclear. To address this gap, we introduce a novel framework for analyzing self-attention matrices and understanding how different objective functions define their weight updates. We then use this framework to derive understandable mathematical structures that should emerge from such updates. Finally, we verify these interpretable structures numerically on many pre-trained and custom models across different modalities, supporting the universality of our results. Identifying these universal structures is fundamental not only for improving the performance of Transformer models, but also for their safety, alignment, and interpretability [ Olah, 2022 ] . Specifically, we connect the matrix \mathbf{W}_{qk}=\mathbf{W}_{q}\mathbf{W}_{k}^{\top} of self-attention with bilinear forms, offering novel insights compared to studying query and key matrices alone. We reveal structured patterns in the implicit weight updates of \mathbf{W}_{qk} , uncovering key differences between encoder-only and decoder-only models: 1. Decoder-only models : Training with an autoregressive objective produces a few columns with disproportionately high norms, introducing directionality in \mathbf{W}_{qk} . 2. Encoder-only models : Bidirectional optimization induces symmetric structures in \mathbf{W}_{qk} , reflecting the balanced nature of the training objective. 3. We validate these theoretical findings across diverse Transformer architectures and input modalities, showing that they generalize across models and tasks. 4. Empirically, we find that symmetric structures in \mathbf{W}_{qk} enhance training efficiency for encoder-only models, leading to higher accuracy and faster convergence in language tasks. 2. Autoregressive and bidirectional training leads to directional and symmetric weight updates In this section, we introduce a novel framework that links the self-attention matrices \mathbf{W}_{qk} to bilinear forms, enabling us to analyze how an objective function influences their structure. This approach reveals fundamental patterns in \mathbf{W}_{qk} that are not apparent when examining \mathbf{W}_{q} and \mathbf{W}_{k} separately. For example, we prove that autoregressive and bidirectional training induce directional and symmetric structures in the \mathbf{W}_{qk} matrices. In the following sections, we define and formalize these concepts. 2.1. Interpreting self-attention with bilinear forms Self-attention [ Vaswani et al., 2017 , Radford et al., 2019 ] is a type of score function A:\mathbf{R}^{N,d}\times\mathbf{R}^{N,d}\rightarrow\mathbf{R}^{N,N} that maps a sequence of N token embeddings with dimension d into a matrix of attention scores. Except for the row-wise softmax function \sigma(\cdot) , self-attention is a linear transformation of the embedded tokens. In particular, \begin{split}\mathbf{A}(\mathbf{X})&=\sigma\left(\frac{1}{\sqrt{d}}\,\hat{\mathbf{A}}(\mathbf{X})\right)\\ &=\sigma\left(\frac{1}{\sqrt{d}}\,\mathbf{Q}\mathbf{K}^{T}\right)\\ &=\sigma\left(\frac{1}{\sqrt{d}}\,\mathbf{X}\mathbf{W}_{qk}\mathbf{X}^{T}\right)\,,\end{split} (1) where \hat{\mathbf{A}}(\mathbf{X}) is the linear part of self-attention (raw unscaled attention scores), \mathbf{X}=[\mathbf{x}^{\top}_{1},\dots,\mathbf{x}^{\top}_{N}]\in\mathbb{R}^{N,d} is the sequence of N token embeddings \mathbf{x}_{i}\in\mathbb{R}^{d} , and \mathbf{W}_{qk}=\mathbf{W}_{q}\mathbf{W}_{k}^{\top}\in\mathbb{R}^{d,d} . This equation shows that the linear transformation \mathbf{W}_{q} and \mathbf{W}_{k} are always combined to compute attention scores with one single matrix \mathbf{W}_{qk} . While the matrices \mathbf{W}_{q} and \mathbf{W}_{k} are defined separately for computational efficiency, this formulation remains mathematically equivalent [ Elhage et al., 2021 , Olsson et al., 2022 , Dar et al., 2023 , see also] . We observe that \mathbf{X}\mathbf{W}_{qk}\mathbf{X}^{\top} corresponds to a bilinear form (see Definition S1.2 ). Specifically, the entry \hat{\alpha}_{ij}=[\hat{\mathbf{A}}]_{ij} can be formulated in two equivalent ways: (1) as the canonical dot product between a query \mathbf{q}_{i} and a key \mathbf{k}_{j} (like in standard Transformer models), or (2) as the dot product between tokens \mathbf{x}_{i} and \mathbf{x}_{j} under the bilinear form \mathbf{W}_{qk} , \hat{\alpha}_{ij}=\langle\mathbf{q}_{i},\,\mathbf{k}_{j}\rangle=\langle\mathbf{x}_{i},\mathbf{W}_{qk}\mathbf{x}_{j}\rangle=\langle\mathbf{x}_{i},\mathbf{x}_{j}\rangle_{\mathbf{W}_{qk}}\,. (2) Intuitively, this equivalence shows that the matrices \mathbf{W}_{q} and \mathbf{W}_{k} together define an alternative metric in the embedding space, which quantifies the score of \mathbf{x}_{i} and \mathbf{x}_{j} without explicitly constructing the query and key vectors. Let us consider a specific input token \mathbf{x}_{i}\in\mathbb{R}^{d} . The self-attention layer maps it to an updated token \hat{\mathbf{x}}_{i}\in\mathbb{R}^{d} as follows, \hat{\mathbf{x}}^{\top}_{i}=\mathbf{x}^{\top}_{i}+\sum_{j=1}^{N}\hat{\alpha}_{ij}\,\mathbf{x}_{j}\mathbf{W}_{v}\,, (3) where the coefficients \{\hat{\alpha}_{ij}\} in the second term represent the projection of \mathbf{x}_{i} onto \text{span}\{\mathbf{X}\} (the subspace spanned by \mathbf{X}=\{\mathbf{x}_{0}^{\top},\mathbf{x}_{1}^{\top},\dots,\mathbf{x}_{N}^{\top}\}) in the transformed embedding space defined by \mathbf{W}_{qk} , \sum_{j}\hat{\alpha}_{ij}\,\mathbf{x}_{j}=\sum_{j}\langle\mathbf{x}_{i},\mathbf{x}_{j}\rangle_{\mathbf{W}_{qk}}\,\mathbf{x}_{j}\,. (4) Because the softmax function preserves the order of its input values, the ranking of the raw attention scores \hat{\alpha}_{ij} is maintained in the final normalized scores \alpha_{ij} , \alpha_{ij}<\alpha_{ij^{\prime}}\,\Leftrightarrow\,\hat{\alpha}_{ij}<\hat{\alpha}_{ij^{\prime}}\quad\forall i,j,j^{\prime}\,. (5) As a result, the attention weights {\alpha_{ij}} define a convex combination of the token vectors {\mathbf{x}_{j}} , meaning the output lies within the convex hull of the input sequence \text{Conv}(\mathbf{X})\subset\text{span}\{\mathbf{X}\} (see also Appendix S1.2 ). We note that \mathbf{W}_{v} is a linear transformation that is applied independently to each projection in the sum and thus does not influence our derivation. In the following sections, we demonstrate how this equivalent formulation of self-attention provides a useful framework for analyzing the training of Transformer models. Specifically, we show that the choice of the objective function such as autoregressive prediction [ Radford et al., 2019 ] or bidirectional training [ Devlin et al., 2019 ] produces distinct structural