The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks

Shangwen Sun, Alfredo Canziani, Yann LeCun, Jiachen Zhu · 2026

Read The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks by Shangwen Sun, Alfredo Canziani, Yann LeCun, Jiachen Zhu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-05498.txt.

Introduction

Sun and team study two Transformer patterns: massive activations and attention sinks. Massive activations are extreme values in a few channels for a few tokens, while attention sinks draw far more attention than they merit. Tests trace the pairing largely to modern Transformer design, with the pre-norm setup making it happen, and removing it pulls the two apart. The roles differ. Massive activations act across the model as near-constant hidden representations across layers, while attention sinks act locally, shaping attention outputs and biasing heads toward short-range links.

Catalog record

CREATORShangwen Sun, Alfredo Canziani, Yann LeCun, Jiachen Zhu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69784
LICENSEhttp://creativecommons.org/publicdomain/zero/1.0/
RIGHTSCC0
SOURCEarXiv

Source: https://arxiv.org/abs/2603.05498

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks Shangwen Sun, Alfredo Canziani, Yann LeCun, Jiachen Zhu Abstract We study two recurring phenomena in Transformer language models: massive activations , in which a small number of tokens exhibit extreme outliers in a few channels, and attention sinks , in which certain tokens attract disproportionate attention mass regardless of semantic relevance. Prior work observes that these phenomena frequently co-occur and often involve the same tokens, but their functional roles and causal relationship remain unclear. Through systematic experiments, we show that the co-occurrence is largely an architectural artifact of modern Transformer design, and that the two phenomena serve related but distinct functions. Massive activations operate globally : they induce near-constant hidden representations that persist across layers, effectively functioning as implicit parameters of the model. Attention sinks operate locally : they modulate attention outputs across heads and bias individual heads toward short-range dependencies. We identify the pre-norm configuration as the key choice that enables the co-occurrence, and show that ablating it causes the two phenomena to decouple. 1 Introduction Transformer-based ( 62 ) Large language models (LLMs) have achieved unprecedented success across a wide range of tasks ( 46 ; 47 ; 7 ; 39 ; 2 ; 60 ; 61 ; 17 ; 45 ; 76 ) , yet many aspects of their internal computations remain poorly understood. In this paper, we study two phenomena that reliably co-occur in decoder-only, pre-norm Transformers ( 46 ; 74 ) : massive activations , in which a handful of tokens exhibit extreme outliers in a few hidden channels ( 59 ; 79 ) , and attention sinks , in which a small number of tokens attract disproportionate attention mass across many heads and layers ( 72 ) . Both phenomena have significant practical implications for quantization ( 70 ; 79 ; 56 ) , pruning ( 33 ; 50 ; 53 ) , KV-cache management ( 16 ; 58 ; 68 ) , and long-context inference ( 22 ; 15 ; 71 ) , among others. Understanding how these two phenomena relate is therefore both theoretically and practically important. Prior work ( 59 ; 23 ; 44 ) has suggested that the co-occurrence is driven by the overlap of involved tokens, but existing explanations remain largely descriptive. Here, we move beyond description to provide a mechanistic account of how and why this overlap emerges in pretrained LLMs. Our core finding is that the co-occurrence is not an inherent property of Transformers, but a predictable consequence of specific architectural and training choices. We advance three central claims: First , normalization is a key architectural component bridging the relationship between massive activations and attention sinks. Changing the normalization configuration can suppress massive activations while preserving attention sinks. Mechanistically, massive activations interact with normalization to produce near-constant hidden representations within a forward pass, effectively serving as implicit parameters that can be exploited to generate attention sinks. Second , attention sinks are primarily driven by the dimensionality of the attention space and by the training context-length distribution. We further show that sinks provide a mechanism to dynamically modulate attention output across heads, biasing certain heads toward short-range dependencies that capture local sentence structure. Third , each phenomenon can be independently suppressed without degrading language-modeling performance, suggesting that their overlap reflects incidental architectural interactions rather than a functional necessity. Together, our results clarify the casual relationship between massive activations and attention sinks and show how alternative design choices can mitigate either of the phenomena. For readability, we refer to the tokens and channels that exhibit massive activations as spike tokens and spike channels , and to the tokens and attention heads affected by attention sinks as sink tokens and sink heads . 2 Preliminaries Modern LLMs are typically trained on the next-token prediction task ( 4 ; 46 ) over large text corpora, using decoder-only Transformers ( 62 ) with the pre-norm configuration ( 74 ) . Despite variations in model size and training data, these choices have remained remarkably consistent across most major models. This section formalizes these core components and establishes the notation used throughout subsequent discussion. 2.1 Next-Token Prediction Next-token prediction is a self-supervised learning objective that leverages the sequential structure of natural language. By treating token order as a natural supervisory signal, models can be trained on vast unlabeled corpora. Formally, let \mathbf{x}\coloneqq(x_{1},\ldots,x_{\mathrm{T}}) be a sequence of \mathrm{T} tokens, where each token x_{i} takes values in a finite vocabulary \mathcal{V} . A language model parameterized by \theta defines a joint distribution: \displaystyle\mathbb{P}_{\theta}(\mathbf{x})=\mathbb{P}_{\theta}(x_{1},\ldots,x_{\mathrm{T}}). (1) Direct modeling of this joint distribution is computationally intractable due to the exponential growth of the sample space. Autoregressive models address this by factorizing the joint distribution into a product of conditional probabilities: \displaystyle\mathbb{P}_{\theta}(x_{1},\ldots,x_{\mathrm{T}})=\prod_{i}\mathbb{P}_{\theta}(x_{i}\mid\mathbf{x}_{<i}), (2) where \mathbf{x}_{<i}\coloneqq(x_{1},\dots,x_{i-1}) represents the prefix (context) preceding index i . In decoder-only Transformers, each conditional is computed by mapping the prefix \mathbf{x}_{<i} to a distribution over \mathcal{V} . During training, all conditionals are produced in parallel by supplying the ground-truth prefix at every position via teacher forcing ( 67 ) . Given a training corpus \mathcal{D} , parameters \theta are learned by minimizing the expected negative log-likelihood: \displaystyle\mathcal{L}(\theta)\coloneqq-\mathbb{E}_{\mathbf{x}\sim\mathcal{D}}\left[\sum_{i}\log\mathbb{P}_{\theta}(x_{i}\mid\mathbf{x}_{<i})\right]. (3) This objective reduces language modeling to a sequence of conditional classification problems over \mathcal{V} , with the conditioning context growing with i . 2.2 Transformer Architecture Since the introduction of the Transformer model, many architectural variants have been proposed, and modern LLMs differ in numerous details. In this section, we describe the specific architecture used by the Llama family of LLMs ( 60 ; 61 ; 17 ) . We focus on Llama because it is among the most widely used open-weight models, and its design choices have strongly influenced subsequent open models such as Qwen ( 45 ; 76 ) and Mistral ( 29 ) . Token embedding. A natural language sentence is first decomposed into a sequence of discrete tokens by a tokenizer, then mapped to continuous vectors via an embedding table. Specifically, each token is mapped to a \mathrm{d}_{\mathrm{model}} -dimensional vector. For a sequence of \mathrm{T} tokens, we denote the resulting hidden representation by \mathbf{H}_{1}\in\mathbb{R}^{\mathrm{T}\times\mathrm{d}_{\mathrm{model}}} . Transformer layers. Starting from \mathbf{H}_{1} , a stack of \mathrm{L} Transformer layers transforms the hidden representation while preserving its dimensionality. Each layer consists of two blocks—an attention block and a feed-forward block—yielding 2\mathrm{L} blocks in total. Let \mathbf{H}_{i}\in\mathbb{R}^{\mathrm{T}\times\mathrm{d}_{\mathrm{model}}} denote the input to block i , and let \mathcal{F}_{i}(\cdot) denote its transformation. Every block employs a residual connection with pre-norm configuration: \displaystyle\mathbf{H}_{i+1}=\mathbf{H}_{i}+\mathcal{F}_{i}(\operatorname{RMSNorm}(\mathbf{H}_{i})), (4) where \mathcal{F}_{i} is the attention block when i is odd and the feed-forward block when i is even. The function \operatorname{RMSNorm}(\cdot) ( 81 ) is applied row-wise: \displaystyle\operatorname{RMSNorm}(\mathbf{h})\coloneqq\sqrt{\mathrm{d}_{\mathrm{model}}}\frac{\mathbf{h}}{\|\mathbf{h}\|}, (5) where \mathbf{h}\in\mathbb{R}^{\mathrm{d}_{\mathrm{model}}} is a single row of \mathbf{H}_{i} . We omit the learnable scale parameter from the \operatorname{RMSNorm} formulation here, since every \operatorname{RMSNorm} is immediately followed by a linear layer and the scale parameter can be absorbed into the subsequent weight matrix during the forward pass. Attention block. The attention mechanism is implemented as multi-head attention with \mathrm{N}_{\mathrm{head}} heads, each of dimension \mathrm{d}_{\mathrm{head}} . For each head i , the normalized input \tilde{\mathbf{H}}\coloneqq\operatorname{RMSNorm}(\mathbf{H}) is projected using head-specific weight matrices \mathbf{W}_{\mathrm{Q}}^{(i)},\mathbf{W}_{\mathrm{K}}^{(i)},\mathbf{W}_{\mathrm{V}}^{(i)}\in\mathbb{R}^{\mathrm{d}_{\mathrm{model}}\times\mathrm{d}_{\mathrm{head}}} : \displaystyle\mathbf{Q}^{(i)} \displaystyle\coloneqq\tilde{\mathbf{H}}\mathbf{W}_{\mathrm{Q}}^{(i)}, (6)

Read the complete text