XAttention: Block Sparse Attention with Antidiagonal Scoring

Ruyi Xu, Guangxuan Xiao, Haofeng Huang, Junxian Guo, Song Han · 2025

Read XAttention: Block Sparse Attention with Antidiagonal Scoring by Ruyi Xu, Guangxuan Xiao, Haofeng Huang, Junxian Guo, Song Han (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16428.txt.

Introduction

Ruyi Xu and co-authors present XAttention, a method that speeds up long-context Transformer models. Attention cost grows with the square of context length, so long inputs get pricey fast. Block sparse attention helps by skipping dull regions, but judging which blocks matter can itself be costly. Their 2025 paper finds a cheap signal for that judgment.

The key trick sums values along the antidiagonals of the attention matrix. That sum turns out to flag which blocks hold the real content, so the rest can be pruned. Across language and video benchmarks, XAttention kept accuracy close to full attention while cutting compute sharply. Model engineers get a plug-in speedup for long inputs.

Catalog record

CREATORRuyi Xu, Guangxuan Xiao, Haofeng Huang, Junxian Guo, Song Han
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-16428

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

XAttention: Block Sparse Attention with Antidiagonal Scoring Ruyi Xu, Guangxuan Xiao, Haofeng Huang, Junxian Guo, Song Han Abstract Long-Context Transformer Models (LCTMs) are vital for real-world applications but suffer high computational costs due to attention’s quadratic complexity. Block-sparse attention mitigates this by focusing computation on critical regions, yet existing methods struggle with balancing accuracy and efficiency due to costly block importance measurements. In this paper, we introduce XAttention, a plug-and-play framework that dramatically accelerates long-context inference in Transformers models using sparse attention. XAttention’s key innovation is the insight that the sum of antidiagonal values (i.e., from the lower-left to upper-right) in the attention matrix provides a powerful proxy for block importance. This allows for precise identification and pruning of non-essential blocks, resulting in high sparsity and dramatically accelerated inference. Across comprehensive evaluations on demanding long-context benchmarks—including RULER and LongBench for language, VideoMME for video understanding, and VBench for video generation—XAttention achieves accuracy comparable to full attention while delivering substantial computational gains. We demonstrate up to 13.5 × \times acceleration in attention computation. These results underscore XAttention’s ability to unlock the practical potential of block sparse attention, paving the way for scalable and efficient deployment of LCTMs in real-world applications. † † affiliationnotice: Equal contribution https://github.com/mit-han-lab/x-attention 1 Introduction The transformative impact of Large Language Models (LLMs) ( Dubey et al. 2024 ; OpenAI 2023 ) is expanding beyond natural language processing, steering in a new era of multimodal capabilities. Long-Context Transformer Models (LCTMs) are emerging as essential tools in this evolution, particularly for tasks like video understanding ( Lin et al. 2023 ; Wang et al. 2024 ) and video generation ( Kong et al. 2025 ) that demand processing and generating exceptionally long sequences of information. These models hold the key to unlocking brilliant systems capable of interacting with the world in a human-like way, understanding and generating not just text, but also visual information over extended periods. Imagine AI agents engaging in seamless, multimodal, day-long interactions, or powerful world simulators generating hours of coherent video—tasks that hinge on processing a tremendous number of tokens. However, realizing this vision requires overcoming a significant challenge: the computational burden of the attention mechanism ( Vaswani et al. 2017 ) . While crucial for capturing relationships within sequences, attention’s cost scales quadratically with sequence length. This quadratic scaling creates a substantial bottleneck during the pre-filling stage, hindering the practical deployment of LCTMs for complex, real-world applications. In the pursuit of more efficient Transformers, block-sparse attention ( Zaheer et al. 2020 ; Guo et al. 2024 ) has emerged as a promising avenue. The core idea is appealing: instead of computing attention between all token pairs, focus resources on the most crucial regions of the attention map, creating ”blocks” of relevant information. This selective computation promises to drastically reduce computational burden while preserving the model’s ability to capture essential long-range dependencies. Yet, existing block-sparse methods have struggled to deliver on their full potential, often grappling with a trade-off between accuracy and efficiency. This stems from the lack of lightweight yet effective mechanisms for identifying and prioritizing truly important attention blocks. The overhead involved in determining block importance can negate the gains achieved through sparsity, rendering these methods impractical for real-world deployment. This leads us to a question: Can we design a block-sparse attention mechanism that dramatically accelerates long-context Transformers without compromising accuracy, truly unlocking their potential for real-world applications? Figure 1: Illustration of XAttention: XAttention optimizes attention through a three-step process: (Left) Strided Antidiagonal Scoring: Each block (8 \times 8 in this example) is scored by summing values along its strided antidiagonals (stride = 4), with red lines generally indicating higher summed values and blue lines lower values. (Middle) Block Selection: High-scoring blocks are selected based on these evaluations. (Right) Block Sparse Attention: Attention is computed only on the selected blocks (red blocks on the right), achieving substantial computational savings. This example uses a sequence length of 24. We answer this question by introducing XAttention, a novel plug-and-play framework designed to significantly improve the efficiency of block-sparse attention in long-context Transformers. XAttention is based on the key insight that the sum of antidiagonal values within the attention matrix can serve as a powerful, yet computationally efficient, indicator of block importance. Unlike existing methods that primarily rely on computationally intensive and lossy solutions like token pooling to identify important blocks, XAttention leverages this simple score to offer a potentially more streamlined and direct approach for rapidly and accurately identifying critical attention blocks. This antidiagonal scoring algorithm allows XAttention to aggressively find and prune non-essential computations, achieving substantial sparsity without sacrificing accuracy. We extensively evaluate XAttention on challenging long-context benchmarks, including RULER and LongBench for natural language processing, VideoMME for video understanding, and VBench for video generation. Across these benchmarks, XAttention achieves accuracy comparable to full attention while delivering substantial computational gains, demonstrating up to 13.5 \times acceleration in attention computation during pre-filling. These results underscore XAttention’s ability to unlock the practical potential of block-sparse attention, paving the way for scalable and efficient deployment of long-context Transformers in demanding applications, especially in the expanding field of multimodal AI. 2 Method In this section, we introduce our method, XAttention . The XAttention algorithm comprises three primary components: (1) importance prediction of attention map blocks, (2) selection of important attention blocks, and (3) prediction of the minimum threshold for attention heads. Figure 2: XAttention’s antidiagonal pattern intersects both vertical and slash patterns within a block, enabling efficient detection of these patterns and guiding effective sparse attention computation. 2.1 Importance Prediction The inherent sparsity of attention maps necessitates a robust strategy for predicting the importance of attention blocks. While methods like MInference ( Jiang et al. 2024 ) and FlexPrefill ( Anonymous 2025 ) utilize a combination of pooling and ”vertical slash detection,” our ablation study reveals that relying solely on average or sum pooling yields inaccurate predictions. Pooling methods are particularly ineffective when only a few significant vertical or slash patterns exist within a block, failing to capture these crucial indicators of importance. MInference and FlexPrefill attempt to overcome this limitation by analyzing the last segment of the input query to identify important ”vertical and slash indices.” However, this approach faces two key challenges: firstly, important attention patterns may not persist in the final query segment; secondly, the search algorithm itself introduces substantial computational overhead (demonstrated in Figure 5 ). Fundamentally, an effective block importance prediction method should automatically and robustly identify significant patterns, including crucial vertical and slash patterns. To achieve this, we propose the antidiagonal selection method . Within each block of size B , we select elements along the antidiagonal using a stride S (visualized in Figure 1 ). The sum of these selected elements serves as a proxy for the overall importance of the corresponding attention block. The effectiveness of this method can be understood from two perspectives: (1) Information Preservation : This selection strategy ensures that information from all tokens is considered, as each token contributes to at least one antidiagonal sum. (2) Pattern Detection: As illustrated in Figure 2 , the antidiagonal intersects every possible vertical and slash pattern within a block. XAttention’s antidiagonal pattern intersects both vertical and slash patterns within

Read the complete text