Why do LLMs attend to the first token?

Federico Barbero, Álvaro Arroyo, Xiangming Gu, Christos Perivolaropoulos, Michael Bronstein, Petar Veličković, Razvan Pascanu · 2025

Read Why do LLMs attend to the first token? by Federico Barbero, Álvaro Arroyo, Xiangming Gu, Christos Perivolaropoulos, Michael Bronstein, Petar Veličković, Razvan Pascanu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-02732.txt.

Introduction

Barbero and co-authors ask why language models pour attention onto the very first token in this 2025 paper. This habit, called an attention sink, affects quantization, security, and streaming systems. Many papers describe when sinks appear, but few explain what they are for.

The authors argue the sink stops information from over-mixing as it flows through the transformer. Their theory connects the sink to known math on signal propagation in deep networks. Experiments then test how context length, model depth, and data packing change sink behavior, matching the theory. The reading reframes the sink as a useful learned tool rather than a glitch, which may guide future model design.

Catalog record

CREATORFederico Barbero, Álvaro Arroyo, Xiangming Gu, Christos Perivolaropoulos, Michael Bronstein, Petar Veličković, Razvan Pascanu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-02732

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Why do LLMs attend to the first token? Federico Barbero, Álvaro Arroyo, Xiangming Gu, Christos Perivolaropoulos, Michael Bronstein, Petar Veličković, Razvan Pascanu Abstract Large Language Models (LLMs) tend to attend heavily to the first token in the sequence – creating a so-called attention sink . Many works have studied this phenomenon in detail, proposing various ways to either leverage or alleviate it. Attention sinks have been connected to quantisation difficulties, security issues, and streaming attention. Yet, while many works have provided conditions in which they occur or not, a critical question remains shallowly answered: Why do LLMs learn such patterns and how are they being used? In this work, we argue theoretically and empirically that this mechanism provides a method for LLMs to avoid over-mixing, connecting this to existing lines of work that study mathematically how information propagates in Transformers. We conduct experiments to validate our theoretical intuitions and show how choices such as context length, depth, and data packing influence the sink behaviour. We hope that this study provides a new practical perspective on why attention sinks are useful in LLMs, leading to a better understanding of the attention patterns that form during training. 1 Introduction Large Language Models (LLMs) are powered by hundreds or even thousands of attention heads that are orchestrated to update the values of tokens within a sequence. As their attention patterns are the only mechanism that allows for the mixing of information between tokens within a Transformer architecture, it is natural to study them to understand how information is being processed. A peculiar and interesting phenomenon that has been spotted across frontier language models is that attention heads often exhibit ‘attention sinks’, where seemingly meaningless tokens – often the first one in the sequence – tend to capture most of the attention. Attention sinks have been linked to a number of important topics such as quantisation ( Liu et al., 2024 ) , improved KV-caching ( Ge et al., 2024 ) , streaming attention ( Xiao et al., 2024 ) , and even security vulnerabilities ( Yona et al., 2025 ) , making them an important artifact that is not yet well-understood in frontier LLMs. While many works have studied attention sinks in order to mitigate them, in this work we take a different angle and aim to understand why they are useful. We do this for a simple reason: as sinks are widespread and appear as a byproduct of gradient descent rather than any explicit priors, they must provide an important mechanism for processing the context. We are therefore interested in understanding and explaining when and why the attention sink mechanism is useful. We tackle the issue from various angles: theoretically studying why such an effect is useful from a ‘mixing’ perspective, and performing measurements in both frontier LLMs and models we train from scratch to support our theory. We focus specifically on understanding why attention sinks occur mostly at the first position of the context. In this position, there commonly lives a special token often denoted as \langle\text{bos}\rangle (beginning of sequence). We find, for example, that in a typical prompt in Llama 405B, almost 80\% of the attention is concentrated on the \langle\text{bos}\rangle token (see Section 4.2 for details). This type of attention allocation is in many ways ‘wasted’, and we find it intriguing to understand exactly why this type of learned behavior is useful. Our main contribution is showing that this type of pattern is a way for deep Transformers to avoid ‘over-mixing’ , a phenomenon related to a number of theoretical and empirical works that study rank collapse ( Dong et al., 2021 ) , representational collapse ( Barbero et al., 2024 ) , signal propagation ( Noci et al., 2022 ; Arroyo et al., 2025 ) , and over-smoothing ( Di Giovanni et al., 2023 ) . In particular, the depth and large context of modern LLMs seems to be sufficient to cause representational collapse, which can be slowed down by making certain heads inactive : a behaviour which attention sinks directly promote (see Figure 1 ). The deeper the architecture, the more inactive heads have to become to ensure that representations stay sufficiently separated throughout the entire architecture to avoid collapse. Figure 1 : Our key result is to illustrate how attention sinks are usefully leveraged by decoder-only Transformers. The presence of attention sinks slows down the mixing of information between tokens and hence makes Transformers more robust to perturbations of prompts. To illustrate this, we demonstrate how sharply a perturbation in the second token’s input representation (in red ) affects the embeddings of other tokens throughout the model, both without ( left ) and with ( right ) a sink token (e.g. \langle\text{bos}\rangle ). The presence of a sink draws attention away from the rest of the tokens, limiting the spread of perturbed information and resulting in more stable embeddings. See Figure 2 for a direct measurement in Gemma 7B. Contributions. We summarise our main contributions. • In Section 3 , we argue that attention sinks are useful to control over-mixing. We connect this to existing theoretical phenomena such as rank collapse, representational collapse, and over-squashing. We show how our mathematical intuitions manifest in Gemma 7B. • In Section 4 , we further support our over-mixing hypothesis. Our refined over-squashing analysis suggests that bigger models and models trained over longer contexts should have stronger sinks. We verify both hypotheses using the LLaMa 3.1 family of models and through our own pre-training runs. We find that a staggering 80% of attention heads form strong sinks in LLaMa 3.1 405B. • In Section 5 , we show that, as expected from our hypothesis, attention sinks form regardless of how \langle\text{bos}\rangle is included during pre-training. Fixing \langle\text{bos}\rangle in pre-training as the first token, however, does impact how the model constructs the sinks. 2 Background In this work, we focus on decoder-only Transformer models ( Radford et al., 2018 ) , that apply a causal mask to the attention mechanism. These are by far the most common types of Transformer models that are used in modern LLMs to date ( Gemma Team et al., 2024 ; Dubey et al., 2024 ) . We follow the notation of Barbero et al. (2024) , but we importantly also consider a model with H\geq 1 attention heads: \displaystyle\mathbf{z}^{(\ell,h)}_{i} \displaystyle=\sum_{j\leq i}\alpha_{ij}^{(\ell,h)}\mathbf{W}^{(\ell,h)}\mathbf{v}^{(\ell)}_{j},\text{with }\alpha_{ij}^{(\ell,h)}=\frac{\exp\left(k\left(\mathbf{q}_{i}^{(\ell,h)},\mathbf{k}_{j}^{(\ell,h)},\mathbf{p}_{ij}\right)\right)}{\sum_{w\leq i}\exp\left(k\left(\mathbf{q}_{i}^{(\ell,h)},\mathbf{k}_{w}^{(\ell,h)},\mathbf{p}_{iw}\right)\right)} \displaystyle\mathbf{z}^{(\ell)}_{i} \displaystyle=\mathbf{W}^{(\ell)}\bigoplus_{h\in H}\mathbf{z}_{i}^{(\ell,h)}+\mathbf{v}^{(\ell)}_{i}, \displaystyle\mathbf{v}_{i}^{(\ell+1)} \displaystyle=\bm{\psi}^{(\ell)}\left(\mathbf{z}_{i}^{(\ell)}\right)+\mathbf{z}_{i}^{(\ell)},\vskip-15.0pt where \bm{\psi}^{(\ell)} is a non-linearity, k takes queries \mathbf{q} , keys \mathbf{k} , and positional encodings \mathbf{p}_{ij} to produce an activation, \mathbf{W}^{(\ell,h)}\in\mathbb{R}^{d\times d} and \mathbf{W}^{(\ell)}\in\mathbb{R}^{Hd\times d} are learnable matrices, and \bigoplus represents a direct sum (concatenation). To simplify notation, we ignore the layer normalisations without loss of generality. The sum ranging over j such that j\leq i is due to the causal mask. If we represent the attention coefficients via a matrix \mathbf{A}^{(\ell,h)} such that \mathbf{A}^{(\ell,h)}_{ij}=\alpha^{(\ell,h)}_{ij} , this condition is equivalent to enforcing that \mathbf{A}^{(\ell,h)} is lower triangular. An LLM consists of L such blocks, with L often called the depth . New tokens are generated autoregressively by considering the final layer representation \mathbf{v}^{(L)}_{n} of the last token and mapping it to a distribution over the token vocabulary. A new token is sampled from this distribution, and

Read the complete text