Sink vs. diagonal patterns as mechanisms for attention switch and oversmoothing prevention
Peter Súkeník, Cristina López Amado, Christoph H. Lampert, Marco Mondelli · 2026
Read Sink vs. diagonal patterns as mechanisms for attention switch and oversmoothing prevention by Peter Súkeník, Cristina López Amado, Christoph H. Lampert, Marco Mondelli (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08453.txt.
Introduction
Peter Súkeník, Cristina López Amado, Christoph Lampert, and Marco Mondelli compare two patterns inside transformer attention. One pattern is sinks, where attention piles onto one special token. The other is diagonal patterns, where tokens attend to themselves. Both have been cast as switches. They stop attention from oversmoothing every token into sameness.
The authors pin down when a sink can be represented at all. Its embedding must align with every other embedding. They sharpen when dense attention provably smooths more than sparse attention. Sinks turn out to be equal to a hard attention switch whose output is zero. Relaxing that switch lets them price sinks against diagonal patterns. It also explains why trained transformers favor sinks.
Catalog record
| CREATOR | Peter Súkeník, Cristina López Amado, Christoph H. Lampert, Marco Mondelli |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 111 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.08453
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Sink vs. diagonal patterns as mechanisms for attention switch and oversmoothing prevention Peter Súkeník, Cristina López Amado, Christoph H. Lampert, Marco Mondelli Abstract This paper studies the role of sinks and diagonal patterns as attention switch and anti-oversmoothing mechanisms. We analyze geometric conditions under which sinks can be represented, showing a necessary alignment between the embedding of the sink and all other embeddings. Next, we refine the current understanding of the role of sinks in oversmoothing prevention: we specify the conditions under which dense attention provably smooths more than sparse attention, and empirically verify that such conditions are often satisfied in practice. We further prove an equivalence between sinks and hard attention switch, in which the output of the attention is identically 0. Finally, we relax the hard attention switch by allowing token self-communication: we provide a quantitative comparison of the costs of representing sinks vs. diagonal patterns, showing why sinks are favored in pretrained transformers. The introduction and analysis of diagonal patterns and the generalization of the attention switch close the gap between what oversmoothing prevention requires and what sinks provide, while also establishing when and why attention layers act like MLPs if token communication is not necessary. 1 Introduction Softmax self-attention is the key component of transformers [ 38 ] that enables communication between tokens and, hence, makes it possible for non-recurrent architectures to solve sequential modeling tasks. Unlike in GNNs [ 35 ] , the full-graph structure of attention allows for much denser and long-range communication patterns, a core feature that made transformers outperform vanilla RNN-like architectures [ 3 ] in language modeling. Thus, it came as a surprise when [ 43 ] revealed that many tokens in many attention heads do not communicate at all. More precisely, [ 43 ] showed heads in which all tokens in a sequence idly attended to the first, beginning-of-sequence (BOS) token. This pattern was termed attention sinks due to the resemblance of visualized attention maps (a vertical line) to sinks. Sinks have since been reproduced in different language settings [ 12 ] , vision transformers [ 6 ] , vision-language models [ 22 ] , multi-modal models [ 17 ] and diffusion-based models [ 41 , 31 ] . Sinks have received significant attention from both practitioners and theoreticians. Practitioners primarily studied how to take practical advantage of their existence or how to mitigate their emergence. [ 43 ] used sinks explicitly to generate long-form content without any fine-tuning and suggested to assign a dedicated sink token during pre-training. Similar concepts were discussed in the follow-ups [ 21 , 5 ] . [ 44 ] presented mixed results where not all sinks improve the accuracy. On the negative side, sinks have been a target of blame in the context of resource effectiveness and interpretability [ 17 ] , hallucination [ 37 , 40 ] , quantization issues [ 4 ] , and training stability [ 27 ] . Thus, some works [ 27 , 12 ] suggested methods to mitigate them, mostly by relaxing the softmax-normalization constraint. Theoreticians focused on explaining why attention sinks emerge (see Section 2 for an overview). Some works [ 2 , 1 ] claimed that sinks help transformers reduce oversmoothing and oversquashing. Others [ 13 , 30 ] connected sinks with the active-dormant mechanism (which we call attention switch), where the head is inactive on some tokens and outputs zero. However, existing oversmoothing results do not cover all architectural components of transformers, and they do not exclude other sparse attention patterns, such as diagonal attention. Similarly, [ 13 ] neither proves that sinks are the only mechanism for attention switch, nor it argues why switching off attention means that it outputs 0, instead of e.g. acting as an MLP on idle tokens. Finally, it remains unclear how heads exhibit the sink from a geometric point of view. Our work closes the outlined gaps via the following contributions: • Section 4 studies how sinks are represented. We show that, for the attention layer to be able to represent the sink, it suffices that all token embeddings are aligned (either positively or negatively) with the sink embedding. We empirically verify this is the case in pretrained transformers. • Section 5 provides the first quantitative analysis of oversmoothing considering all architectural components. We analyze a single attention step and derive conditions under which increasing attention density increases or decreases token cosine similarity. Our analysis shows that, in principle, skip connections and value-output transformations may counteract the mixing induced by attention, even under uniform attention. However, measurements on trained LLMs show that this is rare in practice: in most heads, token similarity increases as attention becomes more uniform. • While earlier work [ 13 , 30 ] defines an attention switch when the output on the residual stream is 0, we regard this as a hard attention switch and define a relaxation, dubbed the soft attention switch , which only prohibits inter-token communication, but doesn’t force zero output. Soft switches allow for diagonal attention, which we empirically verify to be common in transformers. • Section 6 shows that, under mild conditions (which we verify in pretrained LLMs), a head is a hard attention switch if and only if it is a sink. This equivalence goes beyond the results of [ 13 ] , which suggest sinks only as a possible mechanism for hard attention switch and show the emergence of sinks close to convergence (i.e., when attention already exhibits some amount of sink behavior). • Section 7 focuses on the soft attention switch, and it establishes a trade-off between sink and diagonal attention in terms of the \ell_{2} cost necessary to represent them. Because sinks are provably cheaper in a wide range of hyperparameters and distributions, this supports their larger relative prevalence w.r.t. diagonal attention in pretrained transformers. 2 Related work Attention sinks and diagonal patterns. [ 13 ] argues that sinks implement the active-dormant mechanism and justifies the emergence of sinks in a near-convergence regime on a bigram-backcopy task. [ 46 ] claims that sinks play a more active role in computation by tagging future tokens with a shared embedding which can be exploited geometrically in subsequent layers, e.g., for averaging. [ 2 , 1 ] study sinks from the representational viewpoint and show that they help prevent oversmoothing and oversquashing, while [ 28 ] connects sinks with compression valleys arguing they are both manifestations of massive activations in the residual stream. [ 32 ] proposes a geometric interpretation in which sinks act as reference-frame anchors, simplifying the geometry and subsequent attention allocation. [ 26 ] argues that attention and residual sinks rescale and refine non-outlier features and attention weights. The contemporaneouswork [ 30 ] proves that sinks are necessary in an adaptation of the bigram-backcopy task from [ 13 ] . We note that this paper relies on the following assumption: if a pair of token embeddings x_{t},x_{q} appears at positions t,q in a sequence with non-vanishing probability, then the same pair must appear with non-vanishing probability precisely at positions 2,3 . Our theory in Section 6 explicitly takes into account positional embeddings, and it does not require such a strong assumption. Moreover, the main theorem of