Scaling Attention via Feature Sparsity

Yan Xie, Tiansheng Wen, Tangda Huang, Bo Chen, Chenyu You, Stefanie Jegelka, Yifei Wang · 2026

Read Scaling Attention via Feature Sparsity by Yan Xie, Tiansheng Wen, Tangda Huang, Bo Chen, Chenyu You, Stefanie Jegelka, Yifei Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-22300.txt.

Introduction

Xie and co-authors cut the cost of attention by sparsifying features, not tokens. Long contexts strain transformers because self-attention grows with sequence length squared. Most fixes trim along the sequence axis and lose accuracy. Sparse Feature Attention instead writes queries and keys as sparse codes. A custom kernel, FlashSFA, runs the sparse overlaps without building dense score matrices. In GPT-2 and Qwen3 training, the method matches dense baselines. It runs up to 2.5 times faster and halves KV cache use.

Catalog record

CREATORYan Xie, Tiansheng Wen, Tangda Huang, Bo Chen, Chenyu You, Stefanie Jegelka, Yifei Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69817
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.22300

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Scaling Attention via Feature Sparsity Thanks: This work was done at MIT prior to Yifei Wang joining Amazon. Yan Xie, Tiansheng Wen, Tangda Huang, Bo Chen, Chenyu You, Stefanie Jegelka, Yifei Wang Abstract Scaling Transformers to ultra-long contexts is bottlenecked by the O ⁡ ( n 2 ​ d ) O(n^{2}d) cost of self-attention. Existing methods reduce this cost along the sequence axis through local windows, kernel approximations, or token-level sparsity, but these approaches consistently degrade accuracy. In this paper, we instead explore an orthogonal axis: feature sparsity . We propose Sparse Feature Attention (SFA) , where queries and keys are represented as k k -sparse codes that preserve high-dimensional expressivity while reducing the cost of attention from Θ ⁡ ( n 2 ​ d ) \Theta(n^{2}d) to Θ ⁡ ( n 2 ​ k 2 / d ) \Theta(n^{2}k^{2}/d) . To make this efficient at scale, we introduce FlashSFA , an IO-aware kernel that extends FlashAttention to operate directly on sparse overlaps without materializing dense score matrices. Across GPT-2 and Qwen3 pretraining, SFA matches dense baselines while improving speed by up to 2.5 × 2.5\times and reducing FLOPs and KV-cache by nearly 50%. On synthetic and downstream benchmarks, SFA preserves retrieval accuracy and robustness at long contexts, outperforming short-embedding baselines that collapse feature diversity. These results establish feature-level sparsity as a complementary and underexplored axis for efficient attention, enabling Transformers to scale to orders-of-magnitude longer contexts with minimal quality loss. Code is available at https://github.com/YannX1e/Sparse-Feature-Attention . Figure 1: Overview of our proposed method. (a) Trade-off between performance and speed. Compared to directly reducing dimensionality with short embeddings , our method achieves a more favorable balance, delivering a 259% speedup over the original dimensionality while improving performance by 21.4% relative to the short-embedding baseline. (b) Computational and memory efficiency comparison. Our method reduces KV-cache memory usage by 41% and FLOPs by 49%. 1 Introduction Scaling language models to ever longer contexts is fundamentally limited by the O(n^{2}d) cost of self-attention, where n is the sequence length and d the feature dimension. Most existing approaches attempt to reduce this cost along the sequence axis. Windowed or low-rank attention variants constrain interactions to achieve linear complexity, while token-level sparsity prunes which tokens interact ( 5 ; 1 ; 43 ; 6 ; 34 ; 38 ) . Yet large-scale benchmarks consistently show that these approximations sacrifice accuracy, leaving dense attention the most reliable option at long ranges. This raises a natural question: rather than reducing the set of tokens, can we explore feature diversity as an orthogonal axis for scaling attention? This question is motivated by findings in representation learning, where sparse embeddings ( 15 ; 35 ; 18 ; 41 ; 14 ; 37 ) show that high-dimensional spaces encode rich features and that selective activation can preserve expressivity while yielding large efficiency gains. If attention itself can be viewed as retrieval over feature coordinates, then sparsifying queries and keys by activating only their most salient dimensions could reduce computation without collapsing representational capacity. The challenge is to realize this idea in practice: how to preserve expressivity while sparsifying, how to implement kernels that benefit from sparsity without materializing the n\times n score matrix, and how to adapt pretrained dense models without eroding their quality. We address these challenges with Sparse Feature Attention (SFA) . Instead of dense d -dimensional queries and keys, SFA learns k -sparse codes in which each token activates only a handful of coordinates. Attention scores are computed solely from overlaps between these supports, reducing the arithmetic of QK^{\top} from \Theta(n^{2}d) to \Theta(n^{2}k^{2}/d) – a fraction (k/d)^{2} of the dense cost – while storing only O(nk) nonzeros. To make this efficient at scale, we introduce FlashSFA , a new IO-aware kernel that extends FlashAttention by operating directly on sparse overlaps with online softmax. This design avoids materializing any dense n{\times}n scores, retains exactness, and brings compute and memory scaling in line with feature sparsity. The benefits of this shift are demonstrated in Figure 1 . Compared to simply shrinking hidden size (“short embeddings”), SFA achieves a much better trade-off: it improves perplexity by more than 20% while delivering over 2.5 \times speedup, and reduces FLOPs by nearly half together with a 41% drop in KV-cache memory. Experiments confirm that these benefits extend broadly. On GPT-2 and Qwen3 pretraining, SFA matches dense baselines in perplexity and downstream accuracy. On synthetic long-context benchmarks such as Needle-in-a-Haystack, it sustains retrieval accuracy across unseen lengths, while providing consistent latency gains. Crucially, the method is orthogonal to token-level sparsity and paging, multiplying their benefits by lowering per-interaction cost. This work thus establishes feature-level sparsity as a powerful and previously underexplored axis for efficient attention. By leveraging feature diversity rather than compressing it away, SFA preserves high-dimensional expressivity while unlocking substantial efficiency gains. Together with FlashSFA, it makes exact long-context attention practical at scale, and paves the way for context windows extended by orders of magnitude without compromising model quality. 2 Preliminaries Transformers and multi-head attention. Let a sequence of n tokens be represented by hidden states X\in\mathbb{R}^{n\times d_{\text{model}}} . For each head h\in\{1,\dots,H\} with head dimension d , standard scaled dot-product attention computes: \displaystyle Q_{h}=XW^{Q}_{h}\in\mathbb{R}^{n\times d},\qquad K_{h}=XW^{K}_{h}\in\mathbb{R}^{n\times d},\qquad V_{h}=XW^{V}_{h}\in\mathbb{R}^{n\times d}, (1) \displaystyle S_{h}=\frac{Q_{h}K_{h}^{\top}}{\sqrt{d}}\in\mathbb{R}^{n\times n},\qquad P_{h}=\mathrm{softmax}(S_{h}\odot M)\in\mathbb{R}^{n\times n},\qquad O_{h}=P_{h}V_{h}\in\mathbb{R}^{n\times d}, (2) where M encodes causal or padding masks, and the head outputs are concatenated and projected. The principal cost arises from the dense Q_{h}K_{h}^{\top} and materialization of P_{h} ; IO-aware kernels (e.g., FlashAttention) compute O_{h} in tiles without forming P_{h} explicitly, minimizing HBM traffic while remaining exact ( 10 ; 11 ; 31 ) . Sparse formats for efficient storage. Sparse matrices that contain only a few non-zero elements can be stored efficiently in sparse formats. Consider a matrix A\in\mathbb{R}^{n\times d} with \mathrm{nnz}(A) nonzero elements. In the Compressed Sparse Row (CSR) format, we store three arrays: (i) data \in\mathbb{R}^{\mathrm{nnz}(A)} , containing the values of all nonzero entries, (ii) indices \in\{0,\dots,d-1\}^{\mathrm{nnz}(A)} , recording the column index of each nonzero, and (iii) indptr \in\{0,\dots,\mathrm{nnz}(A)\}^{n+1} , where indptr[i] marks the offset in data/indices where row i begins. Thus, the nonzeros of row i can be read quickly from data[indptr[i]:indptr[i+1]]. The Compressed Sparse Column (CSC) format is analogous, but compresses by columns instead of rows, with an indptr array of length d+1 ( 30 ; 12 ) . Efficient multiplication with spare formats. When multiplying two sparse matrices, the cost is not proportional to the dense size n\times d but rather to the number of structural intersections between the nonzero patterns of rows and columns. This operation, called Sparse General Matrix Multiplication (SpGEMM), is typically implemented by Gustavson’s row-wise accumulation algorithm ( 19 ) or by hash-based methods ( 4 ) . The efficiency of SpGEMM therefore depends on how many row–column index sets overlap, making CSR and CSC natural formats for storing query and key matrices in our method. 3 Sparse Feature Attention Figure 2: Three paradigms of attention. Left: Standard attention computes all N\times N query–key interactions in the full feature dimension d . Middle: Sparse attention reduces cost by selecting, for each query i , a small subset of keys \Omega_{i} and masking the remaining logits before softmax, but each retained interaction still spans

Read the complete text