Unveiling Simplicities of Attention: Adaptive Long-Context Head Identification
Konstantin Donhauser, Charles Arnal, Mohammad Pezeshki, Vivien Cabannes, David Lopez-Paz, Kartik Ahuja · 2025
Read Unveiling Simplicities of Attention: Adaptive Long-Context Head Identification by Konstantin Donhauser, Charles Arnal, Mohammad Pezeshki, Vivien Cabannes, David Lopez-Paz, Kartik Ahuja (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-09647.txt.
Introduction
Long documents strain transformer models, yet only some of their attention heads truly need the far context. Konstantin Donhauser, Charles Arnal, Mohammad Pezeshki, Vivien Cabannes, David Lopez-Paz, and Kartik Ahuja find a way to spot those heads in a 2025 paper. Some heads stick to nearby words, while others swing between near and far as the query demands. The team predicts the far-needing heads from local keys alone.
A simple second-moment model of the long-range scores does the predicting. Spotting the hungry heads early could save heavy computing on long texts. The results expose plain structure inside attention that was easy to miss. Efficiency gains are the promised payoff.
Catalog record
| CREATOR | Konstantin Donhauser, Charles Arnal, Mohammad Pezeshki, Vivien Cabannes, David Lopez-Paz, Kartik Ahuja |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-09647
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Unveiling Simplicities of Attention: Adaptive Long-Context Head Identification Konstantin Donhauser, Charles Arnal, Mohammad Pezeshki, Vivien Cabannes, David Lopez-Paz, Kartik Ahuja Abstract The ability to process long contexts is crucial for many natural language processing tasks, yet it remains a significant challenge. While substantial progress has been made in enhancing the efficiency of attention mechanisms, there is still a gap in understanding how attention heads function in long-context settings. In this paper, we observe that while certain heads consistently attend to local information only, others swing between attending to local and long-context information depending on the query. This raises the question: can we identify which heads require long-context information to predict the next token accurately? We demonstrate that it’s possible to predict which heads are crucial for long-context processing using only local keys. The core idea here is to exploit a simple model for the long-context scores via second moment approximations. These findings unveil simple properties of attention in the context of long sequences, and open the door to potentially significant gains in efficiency. † † date: August 24, 2026 † † correspondence: First Author at [email protected] 1 Introduction The landscape of large language models (LLMs) is rapidly evolving, with modern architectures capable of generating text from vast contexts. Recent advances have led to a significant increase in context window sizes, with Llama 3 ( Dubey et al., 2024 ) , DeepSeekv3 ( Liu et al., 2024 ) , and Gemini ( Team et al., 2024a ) supporting windows of at least 128k. However, long context modeling still poses significant challenges ( Hsieh et al., 2024 ) in terms of both accuracy and the substantial cost of processing long contexts in terms of memory and run-time compute. In spite of their importance, our current comprehension of the attention mechanism in long-context tasks remains incomplete. This work aims to address some of these knowledge gaps. Despite the overwhelming complexity of state-of-the-art models, certain simple behaviors in the attention mechanism are strikingly consistent. In particular, many forms of sparse behaviors have been consistently observed, and exploited by numerous methods for efficient inference (see Section 6 ). Among them, Xiao et al. (2023) showed that even when computing the attention only using tokens close to the current token plus initial “sink” tokens, as illustrated in Figure 2 , the model is still capable of generating fluent text. We refer to these tokens as local window, and always implicitly include the initial tokens as they play a crucial role as an attention “sink” (see also Chen et al. (2024) ; Gu et al. (2024) ; Sun et al. (2024b) ). However, such a local window approximation, if applied to every attention head simultaneously, necessarily harms the capabilities of LLMs to retrieve and process long-context information (see e.g., Xiao et al. (2024) ). Instead, to overcome such limitations, we aim to identify the heads whose output can be well-approximated via a local window attention, and apply the approximation to those only. If a head can be approximated via a local approximation, we call it a local head , and otherwise it is a long-context head . In particular, we ask: Which heads can be approximated using a local window with minimal impact on downstream task performance? Figure 2 : Attention sparsity and its impact on efficiency. Left: Attention scores are split into bulk ( A^{\text{bulk}} ) for distant tokens and local window ( A^{\text{local}} ) for nearby ones. A head is considered local if most of its attention mass falls within the local window. The static criterion pre-assigns local heads, while the adaptive oracle query-dependently compares bulk and local contributions but is computationally expensive. Our approximation models A^{\text{bulk}} using a Gaussian distribution for efficiency. Middle: Oracle-based classification with \tau=0.6 (see Figure 4 for the threshold) reveals three types of heads: consistently local (heads labeled more than 95\% of the times as local), often long-context (less than 50\% ), and varying, which switch behavior dynamically. Right: Comparison of three methods: Static (green) removes a fixed fraction of heads, the adaptive oracle (blue) dynamically selects heads but is costly, and our adaptive method (purple) achieves near-oracle performance with significantly lower cost. As sparsity increases, static pruning degrades performance, while our adaptive method remains robust. These results show that most attention heads do not need to attend to the entire context , enabling significant efficiency gains with query-adaptive head classification. Two approaches to this problem can be distinguished: Static criteria label the heads – local vs long-context – once for all queries, while query-adaptive criteria change the labels from query to query. Static criteria, as used by Xiao et al. (2024) ; Tang et al. (2024a) , have the advantage that all key-value pairs (except for the few in the local window) of local heads can be discarded, thus saving memory. While recent works ( Wu et al., 2024 ; Tang et al., 2024a ; Hong et al., 2024 ) provide some evidence that a fixed small subset of the heads are particularly relevant for processing long-context information, the following question remains unclear: How much sparsity (measured as the average percentage of local heads) can we gain using query-adaptive criteria compared to static criteria? Contribution 1. We present an extensive analysis comparing a query-adaptive oracle criterion, which selects local heads independently for each token, with static criteria. We make two observations: first, we find that static criteria can label up to 60% of the heads as local heads without impacting downstream task evaluations, which confirms the intuition from ( Wu et al., 2024 ) . Nevertheless, we find that a query-adaptive oracle criterion allows to label a substantially higher percentage of heads as local heads (up to 90%) without sacrificing performance (see Figure 2 ). Unfortunately, the oracle requires the computation of the full attention scores. This leads to the following question: For each query, can we determine which heads are long-context and which are local without computing the full attention scores? The relevance of this question is twofold: on one hand, answering it helps guide further research toward developing more compute-efficient attention mechanisms. On the other hand, it advances our understanding of the inner workings of attention mechanisms, which is central to mechanistic interpretability (see, e.g., Olsson et al. (2022) ). Contribution 2. We address this question by proposing a novel query-adaptive attention criterion (QAdA) based on second-order statistics of the attention scores (briefly summarized in Figure 2 ). Our experiments on three families of LLMs, Llama ( Dubey et al., 2024 ) , Qwen ( Bai et al., 2023 ) and Mistral ( Jiang et al., 2023 ) applied to a variety of standard long-context benchmarks, as well as hard reasoning tasks embedded in long-context prompts, show that this relatively simple criterion allows to efficiently identify long-context heads: our method increased sparsity at a smaller loss in downstream performance than oracle static approaches. Along with our other experiments, it sheds light onto certain simple behaviors of attention heads in long-context settings. Figure 3 : Examples of attention score distributions for each possible outcome with \tau_{\text{approx}}=\tau_{\text{oracle}}=0.6 with the oracle criterion as ground truth. We show histograms of scores from the local window \mathcal{I} ( brown ) and the