The Laplacian Mechanism Improves Transformers by Reshaping Token Geometry

Yuchong Zhang, Vardan Papyan · 2026

Read The Laplacian Mechanism Improves Transformers by Reshaping Token Geometry by Yuchong Zhang, Vardan Papyan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-09297.txt.

Introduction

Yuchong Zhang and one co-author propose Laplacian heads for transformers, a small tweak with wide effects. Standard heads refresh token pictures through softmax focus; Laplacian heads instead use the matrix I minus focus, letting plain heads steer sequence means while Laplacian heads steer within-sequence spread straight. Seen as a web over tokens, the Laplacian step is one move of heat spread.

The tweak lifts scores across teacher-led learning, language modeling, and self-taught learning. Study shows why. In teacher-led settings, Laplacian heads fold same-sequence tokens and line up sequence means with neural-collapse shape. In language modeling, they sharpen split-ability among tokens sharing a next-token guess, and in self-taught learning, their main parts fit splitting better.

Catalog record

CREATORYuchong Zhang, Vardan Papyan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-09297

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Laplacian Mechanism Improves Transformers by Reshaping Token Geometry Yuchong Zhang, Vardan Papyan Abstract Transformers leverage attention, the residual connection, and layer normalization to control the variance of token representations. We propose to modify attention into a Laplacian mechanism that gives the model more direct control over token variance. We conjecture that this helps transformers achieve the ideal token geometry. To investigate our conjecture, we first show that incorporating the Laplacian mechanism into transformers induces consistent improvements across benchmarks in computer vision and language. Next, we study how the Laplacian mechanism impacts the geometry of token representations using various tools: 1) principal component analysis, 2) cosine similarity metric, 3) analysis of variance, and 4) Neural Collapse metrics. Our investigation shows that the Laplacian mechanism reshapes token embeddings toward a geometry of maximal separability: tokens collapse according to their classes, and the class means exhibit Neural Collapse. (a) Baseline (b) Proposed Figure 1 : The proposed modification cleanly separates tokens into clusters according to class. Each colour corresponds to one class. 1 Introduction 1.1 Motivation The transformer architecture has achieved remarkable success across a wide range of tasks since its introduction by 46 . A transformer tokenizes its input into a sequence of tokens and processes these tokens using a sequence of blocks. The core components of each block are attention, the residual connection ( 19 ) , and layer normalization ( 4 ) . Together, they shape how tokens move in the representation space. Softmax self-attention can be viewed as a learned averaging operator. For each token, attention computes a contextual mean of tokens, where the weights are determined by token similarities. The residual connection then adds this mean back to each token. This update changes the ratio between the magnitude of the mean and the variance. Layer normalization then projects the tokens back onto a sphere, thereby controlling the variance of the tokens. This process occurs in all blocks, driving token representations toward some geometric configuration at the final layer. What geometry should token representations follow? For standard classifiers, 34 showed that as training approaches zero error, the final-layer representations exhibit a distinctive geometry—termed Neural Collapse (NC) —with four defining properties: NC1 Final-layer features collapse to their class means. NC2 The class means converge to a simplex equiangular tight frame (ETF). NC3 The linear classifier weights align with the corresponding class means. NC4 The classifier predicts each class via nearest-class-mean decision boundaries. While NC has been extensively studied in convolutional networks for image classification ( 34 ; 18 ) , transformers present a markedly different setting: inputs are tokenized. A related line of work studies rank collapse , where token embeddings become low-rank and can collapse toward a single point. In particular, 16 proved theoretically that self-attention collapses tokens into various geometries depending on the initial conditions. These works do not identify an optimal geometry, but they suggest that the rank and variance of tokens are central to the geometry induced by attention dynamics. Standard transformers control token variance in a seemingly indirect manner: by first changing the mean’s magnitude relative to variance and then normalizing. Thus, we are motivated to conjecture: giving the transformer more direct control over token variance can help it attain the ideal token geometry. 1.2 Contributions To investigate our conjecture, we first introduce the Laplacian mechanism, a simple modification to attention that computes the difference between each token and its corresponding contextual mean. This allows the model to directly decrease or increase the variance of tokens via the residual connection (Section 2 ). Next, we incorporate the Laplacian mechanism into standard transformers and evaluate on both vision and language datasets. Our experiments show consistent performance improvements (Section 3 ). We then study the geometry of token representations (Section 4 ) in vision transformers trained for image classification using PCA, cosine similarity metric, analysis of variance, and Neural Collapse metrics. We find that the Laplacian mechanism promotes a geometry that admits maximal separability between classes. We term this geometry Neural Token Collapse (NTC), which is characterized by: NTC0 Tokens in the same sequence collapse. NTC1 Tokens in the same class collapse to the class mean. NTC2–4 The class means and classifier weights satisfy NC2, NC3, and NC4. We formally define NTC in Section 5 . 2 Proposed Architecture Modification 2.1 Standard Attention Mechanism Consider a sequence of T token embeddings (x_{1},\dots,x_{T}) with each x_{i}\in\mathbb{R}^{d} , collected into a matrix X\in\mathbb{R}^{T\times d} . Attention projects X into queries, keys, and values using trainable matrices W_{Q},W_{K},W_{V}\in\mathbb{R}^{d\times d_{k}} : Q=XW_{Q},\qquad K=XW_{K},\qquad V=XW_{V}. It then computes scaled dot-product weights and outputs a weighted sum of the values: \displaystyle P=\operatorname{softmax}\!\Bigl(\frac{QK^{\top}}{\sqrt{d_{k}}}\Bigr),\quad\text{Attention}(X)=PV. (1) Building on this definition, multi-head attention ( MHA ) computes h attention heads in parallel, concatenates their outputs, then applies a trainable projection matrix W_{o}\in\mathbb{R}^{hd_{k}\times d} . The result is added back to the input via a residual connection ( 19 ) : \tilde{X}=\text{LayerNorm}(X),X^{\prime}=X+\text{MHA}(\tilde{X}). (2) After this residual update, the token embeddings are passed through layer normalization, which projects the tokens onto a sphere with radius \sqrt{d} . This process repeats at each layer. (a) Standard attention mechanism (b) Proposed Laplacian mechanism Figure 2 : The Laplacian mechanism changes the variance of token embeddings more directly. Interpretation. Equation 2 changes how large the tokens’ mean is relative to their variance, and the subsequent normalization regulates the variance of tokens. Figure 2(a) illustrates an example of decreasing token variance: token embeddings \vec{x}_{1},\vec{x}_{2} (black arrows) are shifted by their attention-weighted means (PX)_{1},(PX)_{2} (blue arrows) along the orange arrows. After projection, the tokens become closer and the variance decreases. 2.2 The Laplacian Mechanism We modify the standard attention mechanism into a Laplacian mechanism , denoted by L : \displaystyle P=\operatorname{softmax}\!\Bigl(\frac{QK^{\top}}{\sqrt{d_{k}}}\Bigr),\qquad\textbf{L}(X)=V-PV. (3) The name “Laplacian” will be explained in Section 6 . Interpretation. L computes the difference between tokens and their average. If every attention head is replaced with L , then Equation 2 allows the model to directly decrease (or increase, since W_{o} can change the sign of the update) the variance of tokens without relying on the redundant radial movement later canceled by projection. Figure 2(b) illustrates an example of decreasing token variance: token embeddings \vec{x}_{1},\vec{x}_{2} (black arrows) move directly towards the means (PV)_{1},(PV)_{2} (blue arrows) along the variance direction (orange arrows). The green points are the projected tokens, which end up much closer together with normalization playing only a minor role. We provide empirical evidence for this interpretation in Appendix D . 2.3 Mixing Both Mechanisms We propose incorporating the Laplacian mechanism into transformers by the following simple strategy: in every layer, use the Laplacian for k heads and standard attention for the remaining heads for some fixed k<h , where h is the total number of heads. This strategy is straightforward to implement and introduces no additional trainable parameters. Moreover, it allows us to study the impact of the Laplacian heads by varying k in each layer. We describe some alternative strategies we have tried in Appendix E . Interpretation. This setup allows the standard attention heads to update the mean of tokens, while the Laplacian heads gives the model more direct control over the variance of tokens, thereby giving the model greater flexibility to steer tokens toward the

Read the complete text