DashAttention: Differentiable and Adaptive Sparse Hierarchical Attention
Yuxiang Huang, Nuno M. T. Gonçalves, Federico Alvetreti, Lei Li, Xu Han, Edoardo M. Ponti, André F. T. Martins, Marcos V. Treviso · 2026
Read DashAttention: Differentiable and Adaptive Sparse Hierarchical Attention by Yuxiang Huang, Nuno M. T. Gonçalves, Federico Alvetreti, Lei Li, Xu Han, Edoardo M. Ponti, André F. T. Martins, Marcos V. Treviso (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18753.txt.
Introduction
Yuxiang Huang and seven colleagues offer DashAttention, a 2026 method for long-context language models. Older layered attention picks a fixed number of key-value blocks, which blocks learning between sparse and dense stages. Their design lets each query choose a varying number of blocks, then guides full softmax attention with that choice. The whole chain stays learnable end to end, and tests show it matches full attention while skipping three quarters of the work.
Catalog record
| CREATOR | Yuxiang Huang, Nuno M. T. Gonçalves, Federico Alvetreti, Lei Li, Xu Han, Edoardo M. Ponti, André F. T. Martins, Marcos V. Treviso |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 81446 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.18753
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DashAttention: Differentiable and Adaptive Sparse Hierarchical Attention Yuxiang Huang, Nuno M. T. Gonçalves, Federico Alvetreti, Lei Li, Xu Han, Edoardo M. Ponti, André F. T. Martins, Marcos V. Treviso Abstract Current hierarchical attention methods, such as NSA and InfLLMv2, select the top- k k relevant key-value (KV) blocks based on coarse attention scores and subsequently apply fine-grained softmax attention on the selected tokens. However, the top- k k operation assumes the number of relevant tokens for any query is fixed and it precludes the gradient flow between the sparse and dense stages. In this work, we propose DashAttention (Differentiable and Adaptive Sparse Hierarchical Attention), which leverages the adaptively sparse α \alpha -entmax transformation to select a variable number of blocks according to the current query in the first stage. This in turn provides a prior for the second-stage softmax attention, keeping the entire hierarchy fully differentiable . Contrary to other hierarchical attention methods, we show that DashAttention is non-dispersive, translating to better long-context modeling ability. Experiments with large language models (LLMs) show that DashAttention achieves comparable accuracy as full attention with 75% sparsity and a better Pareto frontier than NSA and InfLLMv2, especially in high-sparsity regimes. We also provide an efficient, GPU-aware implementation of DashAttention in Triton, which achieves a speedup of up to 3.3 × 3.3\times over FlashAttention-3 at inference time. Overall, DashAttention offers a cost-effective strategy to model long contexts. 1 1 1 Code available at https://github.com/fasa-org/dash-attention . 1 Introduction The complexity of long-context tasks is impacted by the quantity of information to be retrieved, its obfuscation (i.e., whether it is distinguishable from noise), and its distribution throughout the context (e.g., scattered or concentrated) [ 19 , 43 ] . Hence, to achieve good performance in such tasks, models must remain selective enough to ignore irrelevant tokens, but also sufficiently flexible to recover the positions that matter for a given query, irrespective of their number, position, or similarity to the rest of the context. Dense softmax attention [ 56 ] handles the second requirement but fails the first, since every visible token receives nonzero probability mass, which can cause dispersion [ 58 ] . Hard sparse routing methods such as top- k block selection [ 66 , 73 ] address the first requirement; however, they do so by imposing a fixed budget, sacrificing the second requirement, and by severing the differentiable path between coarse routing decisions and fine token-level attention. Therefore, achieving both query-dependent flexibility and stringent token-level selectivity remains an open challenge. This paper proposes DashAttention (Differentiable and Adaptive Sparse Hierarchical Attention), a multi-stage attention mechanism designed to meet both requirements. Instead of routing with a hard top- k operator over block scores, we route with \alpha -entmax [ 47 ] , an adaptive sparse distribution whose support is learned from the input itself and whose nonzero masses remain differentiable. The chunk-level probabilities are then consumed by a second online-softmax refinement stage that operates at full resolution, but only on tokens inside the routed chunks. This way, the model learns where and how much to look at coarsely, and what to read precisely, while keeping the entire hierarchy differentiable end-to-end. This design is motivated by both theory and practice. On the theory side, recent work has shown that softmax full attention can be dispersive in long contexts, whereas sparse alternatives such as \alpha\text{-entmax } can preserve concentration and improve long-context capabilities [ 58 , 57 ] . On the systems side, modern sparse attention methods almost always rely on a hierarchical decomposition, since selecting tokens based directly on attention scores is too expensive: the query-key multiplications must still be computed, leaving little opportunity to reduce computation [ 66 , 73 , 41 ] . Our key intuition is to combine these two ideas into a single hierarchical attention mechanism that is inherently adaptive, fully differentiable, and efficient. Our main contributions are as follows: • We analyze the limitations of top- k -based sparse attentions and introduce DashAttention, an end-to-end differentiable method that adaptively allocates sparsity across attention heads. • We integrate DashAttention into long-context continual pretraining and show that it outperforms existing hierarchical sparse attention methods while matching full-attention accuracy. • We present an efficient GPU-aware implementation of DashAttention that achieves speedups of 3.36 \times and 1.35 \times over FlashAttention-3 and InfLLMv2, respectively. Figure 1 : High-level overview of DashAttention. Stage 0 builds chunk summaries by local SDPA; Stage 1 routes with \alpha\text{-entmax } to obtain an adaptively sparse support; Stage 2 refines at token resolution with a softmax whose logits are biased by d_{i,j} , derived from the router weights, maintaining full differentiability and FlashAttention compatibility. See Section 3 for details. 2 Background Standard scale dot-product attention. Given a set of matrices \bm{Q},\bm{K},\bm{V}\in\mathbb{R}^{n\times d} containing d -dimensional representations for n queries, keys, and values, the scale dot-product self-attention (SDPA) at a single head is computed as follows [ 56 ] : \bm{Z}=\frac{\bm{Q}\bm{K}^{\top}}{\sqrt{d}}\in\mathbb{R}^{n\times n},\quad\bm{P}=\pi\left(\bm{Z}+\bm{M}\right),\quad\bm{O}=\bm{P}\bm{V}\in\mathbb{R}^{n\times d}, (1) where \bm{M} is the attention mask and \pi:\mathbb{R}^{n}\to\triangle_{n} is a transformation applied row-wise that maps logits to vectors in the n -dimensional probability simplex \triangle_{n}:=\{\bm{p}\in\mathbb{R}^{n}\,:\,\mathbf{1}^{\top}\bm{p}=1,\,\bm{p}\geq\mathbf{0}\} , with softmax being the most common choice. FlashAttention [ 11 ] efficiently implements Eq. ( 1 ) by reducing memory to \mathcal{O}(n) while preserving softmax semantics. Adaptively sparse attention: \alpha\text{-entmax } . Softmax attention is inherently dense, as every key receives a positive weight. A principled differentiable sparse alternative is the \alpha\text{-entmax } function [ 47 ] , \alpha\text{-entmax }(\bm{s})=[(\alpha-1)\bm{s}-\tau\bm{1}]_{+}^{\frac{1}{\alpha-1}}, (2) where [\cdot]_{+} is the ReLU function, and \tau\in\mathbb{R} is the (unique) normalizing constant which ensures the output is a valid probability distribution, \sum_{i}\alpha\text{-entmax }(\bm{s})_{i}=1 . Importantly, for \alpha>1 , \alpha\text{-entmax } can produce sparse probability vectors. When \alpha\to 1 , \alpha\text{-entmax } recovers softmax, whereas \alpha=2 leads to sparsemax [ 42 ] ; the sparsity increases monotonically with \alpha . This means that coordinates with (\alpha-1)s_{i}\leq\tau become exactly zero. As such, \alpha\text{-entmax } yields dynamic sparsity, where the pattern of zeros (as well as its number) depends on the input \bm{s} . Efficient GPU implementations for \alpha\text{-entmax } attention have been developed in AdaSplash [ 21 , 20 ] . More information on the \alpha\text{-entmax } transformation, including the distinction to top- k softmax, is provided in App. A . Hierarchical trainable sparse attention. Recently proposed pretraining-compatible sparse attention methods [ 66 , 73 , 41 ] typically adopt a coarse-to-fine hierarchical design: the context is divided into blocks or pages, summarized by lightweight representations, routed via top- k coarse relevance scores, and followed by exact token-level SDPA within the selected KV segments. However, their fixed top- k selection overlooks the heterogeneous information demands across tokens, attention heads, and layers. As shown in Sec. 5.4 , DashAttention dynamically allocates sparsity across tokens and heads, which is crucial for improving performance under a fixed sparsity budget. 3 DashAttention Motivation. The key innovation of DashAttention is that the coarse router is itself a sparse attention mechanism—specifically, \alpha -entmax from Eq. ( 2 )—rather than a top- k truncation applied to dense scores. Since \alpha -entmax is fully differentiable, the hierarchical attention remains end-to-end trainable, while ensuring a level of sparsity adaptive to each specific token sequence. Hence, DashAttention is built around two complementary inductive biases: \alpha