Self-Attention as Distributional Projection: A Unified Interpretation of Transformer Architecture
Nihal Mehta · 2025
Read Self-Attention as Distributional Projection: A Unified Interpretation of Transformer Architecture by Nihal Mehta (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-13780.txt.
Introduction
Nihal Mehta offers a unified reading of the transformer design through distributional semantics. Self-attention, he argues, arises from mapping corpus-level word co-occurrence statistics into the context of a single sequence. Starting from the same matrix behind GloVe embeddings, that mapping naturally yields contextual influence between tokens.
The query, key, and value mechanism then appears as the natural asymmetric extension for directional relations. Position marks and multi-head designs follow as structured refinements of the same idea. On this account, the transformer's basic shape is not an arbitrary invention but a consequence of those principles, a viewpoint that may guide how future architectures are derived rather than guessed.
Catalog record
| CREATOR | Nihal Mehta |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-13780
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Self-Attention as Distributional Projection: A Unified Interpretation of Transformer Architecture Nihal Mehta Abstract This paper presents a mathematical interpretation of self-attention by connecting it to distributional semantics principles. We show that self-attention emerges from projecting corpus-level co-occurrence statistics into sequence context. Starting from the co-occurrence matrix underlying GloVe embeddings, we demonstrate how the projection naturally captures contextual influence, with the query-key-value mechanism arising as the natural asymmetric extension for modeling directional relationships. Positional encodings and multi-head attention then follow as structured refinements of this same projection principle. Our analysis demonstrates that the Transformer architecture’s particular algebraic form follows from these projection principles rather than being an arbitrary design choice. Keywords: self-attention, transformers, distributional semantics, co-occurrence matrix, representation learning, architectural interpretation, linear algebra, context projection. 1 Introduction Transformer models rely on self-attention layers to construct contextual token representations [ 9 ] . These layers are typically introduced procedurally: given learned projections W_{Q} , W_{K} , W_{V} , a sequence of token embeddings H is mapped to queries, keys, and values; pairwise dot products between queries and keys are normalized by row-wise softmax to produce attention weights. Despite their empirical success, this mechanism is rarely analyzed from mathematical principles, leaving open why this particular algebraic form so effectively models contextual interaction. The standard presentation borrows terminology from information retrieval—queries search for matching keys to retrieve values—but this analogy obscures rather than illuminates. The equations \mathrm{softmax}(QK^{\top}/\sqrt{d_{k}})V contain no search procedure; the Q , K , V matrices are simply learned projections whose relationship to “queries” and “keys” is purely metaphorical. This borrowed language has left the architectural choices appearing arbitrary and the mechanism’s effectiveness mysterious. This paper presents a mathematical analysis that addresses this gap by extending the distributional semantics foundation of GloVe [ 7 ] . We begin with the symmetric co-occurrence operator S\in\mathbb{R}^{n\times n} that underlies GloVe embeddings, projecting it into sequence-specific contexts via selector matrix Q\in\mathbb{R}^{R\times n} to capture how words interact in context. This bridges the gap between classical distributional semantics and modern neural architectures, offering a principled explanation for the architectural choices in Transformers. This work provides a mathematical clarification of the self-attention mechanism, showing that the Transformer architecture emerges naturally from extending distributional semantics principles to handle the core challenges of sequence modeling. This framework reveals three key insights that connect distributional semantics to Transformer architecture: • Projection Characterization: The projection of global co-occurrence statistics into local sequence context is uniquely characterized by natural mathematical properties, providing a principled foundation for understanding the algebraic structure of self-attention. • Asymmetric Extension: The query-key split emerges as the natural extension when generalizing the symmetric projection to model directional linguistic relationships, explaining why Transformers use separate transformations rather than a single symmetric operator. • Unified Framework: Positional encodings and multi-head attention arise as structured refinements of the same projection principle, revealing how the complete architecture follows from systematically addressing the requirements of sequence modeling. This progression—from symmetric projection through asymmetric generalization to multi-head factorization—demonstrates that the Transformer architecture is not merely an empirically successful design, but rather the mathematical consequence of systematically extending distributional semantics to handle the core challenges of sequence modeling: context specificity, directional relationships, and positional order. Our analysis proceeds systematically: we begin with the symmetric co-occurrence projection M=QSQ^{\top} (Section 2 ), extend it to vocabulary-level prediction (Section 3 ), incorporate positional sensitivity (Section 4 ), then introduce directional attention through learned transformations (Section 5 ), and finally connect to the full Transformer architecture including multi-head attention (Section 6 ). Throughout, we emphasize the mathematical intuition behind each component, with illustrative examples included solely to clarify the analysis. 2 Projection of Corpus-Level Statistics into Sequence Context The standard presentation of self-attention begins with learned projections W_{Q},W_{K},W_{V} applied to token embeddings. This starting point, however, already assumes the architectural form we wish to understand. We instead begin with the most fundamental statistical regularities in language: distributional structure as captured through co-occurrence statistics. 2.1 Global co-occurrence as a representation operator Let the corpus contain n distinct token types. Define the global co-occurrence operator S\in\mathbb{R}^{n\times n} , where each entry S_{ij} measures the empirical association between tokens i and j . Each row of S represents the empirical neighborhood of a token—the distribution of other tokens that tend to appear near it. Multiplying a one-hot basis vector e_{i} by S yields a vector e_{i}S whose components indicate the relative frequencies of words co-occurring with token i . Thus, S defines a linear map from token identity to its observed contextual distribution. 2.2 Restricting to the input subspace Consider an input sequence of R tokens. Define a selection matrix Q\in\mathbb{R}^{R\times n} whose rows identify the tokens that appear in the input. The local projection M=QSQ^{\top}\in\mathbb{R}^{R\times R}, restricts the global operator S to the subspace spanned by the input tokens. Left-multiplying by Q retains only associations that originate from input tokens; right-multiplying by Q^{\top} retains only associations that terminate on input tokens. The resulting matrix M extracts from the global co-occurrence statistics the specific tokens that are contextually related to the input sequence while filtering out tokens that don’t align with the emergent semantic context. Formally, this projection extracts the subnetwork of relationships most relevant to the current sequence , amplifying associations that are mutually reinforced by the input tokens and suppressing those that are contextually incompatible: M_{rs}=q_{r}Sq_{s}^{\top}, which measures the compatibility between tokens r and s according to the global distribution. Each row of M therefore defines how one token’s representation should be adjusted in light of the others that co-occur with it in data. 2.3 Algebraic Analysis of Contextual Weighting The contextual weighting mechanism is defined as follows: • q_{r}\in\mathbb{R}^{n} is the r -th row of Q . This row vector is a one-hot vector selecting the token at position r . • M_{rs}=q_{r}Sq_{s}^{\top} be the contextual weight between tokens r and s The matrix M encodes how much tokens agree about the semantic context. The amplification and suppression properties emerge from the co-occurrence patterns in M : Amplification occurs when: • Multiple tokens have strong mutual agreement ( M_{rs}\gg 0 ) due to high co-occurrence between the tokens • These tokens share common associations with the same vocabulary items • Result: Constructive interference in the contextual representation Suppression occurs when: • Tokens have weak contextual agreement ( M_{rs}\approx 0 ) due to low co-occurrence • Vocabulary associations are isolated to single or few tokens • Result: Dilution and weak evidence due to lack of reinforcement This algebraic structure automatically amplifies semantically coherent associations while suppressing contextually isolated ones through the agreement patterns encoded in M . 2.4 Row Normalization and Emergent Attention The matrix M=QSQ^{\top} contains raw compatibility scores that require normalization to function as proper attention weights. We apply row-wise normalization to convert these scores into probability distributions: \mathrm{Norm}(M)_{rs}=\frac{M_{rs}}{\sum_{t=1}^{R}M_{rt}} This normalization serves two crucial purposes: 1. It converts the compatibility scores into probability-like distributions over contextual partners 2. It ensures numerical stability by controlling the magnitude of the scores The normalized matrix \mathrm{Norm}(M) already functions as a primitive attention mechanism: each token redistributes its representation according to corpus-consistent co-occurrence weights. This indicates that the core attention mechanism emerges naturally from normalized projections of corpus statistics, with the learned