Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps
Yanke Zhou 1,♯ Yiduo Li 2 Hanlin Tang 2,† Maohua Li 1,♯ Kan Liu 2 Lan Tao 2 Lin Qu 2 Yuan Yao 1,§ Xiaoxing Ma 1 · 2026
Read Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps by Yanke Zhou 1,♯ Yiduo Li 2 Hanlin Tang 2,† Maohua Li 1,♯ Kan Liu 2 Lan Tao 2 Lin Qu 2 Yuan Yao 1,§ Xiaoxing Ma 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-16928.txt.
Introduction
Zhou's team shows full-attention language models are already sparse inside and can be turned sparse with only a few hundred training steps. Only a small set of attention heads needs full long-context work, long-range retrieval rests on a low-dimensional subspace, and the token budget varies by query. Their system, RTPurbo, keeps the full key-value cache only for retrieval heads and adds a light token indexer for sparse attention. Long-context tests keep near-lossless accuracy while reaching ninefold faster prefill at one-million context.
Catalog record
| CREATOR | Yanke Zhou 1,♯ Yiduo Li 2 Hanlin Tang 2,† Maohua Li 1,♯ Kan Liu 2 Lan Tao 2 Lin Qu 2 Yuan Yao 1,§ Xiaoxing Ma 1 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46634 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.16928
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps Yanke Zhou 1,♯ Yiduo Li 2 Hanlin Tang 2,† Maohua Li 1,♯ Kan Liu 2 Lan Tao 2 Lin Qu 2 Yuan Yao 1,§ Xiaoxing Ma 1 Abstract Long-context inference in large language models is bottlenecked by the quadratic cost of full attention. Existing efficient alternatives often rely either on native sparse training or on heuristic token eviction, creating an undesirable trade-off among efficiency, training cost, and accuracy. In this work, we show that full-attention LLMs are already intrinsically sparse and can be transformed into highly sparse models with only minimal adaptation. Our approach is built on three observations: (1) only a small subset of attention heads truly requires full long-context processing; (2) long-range retrieval is governed primarily by a low-dimensional subspace, allowing relevant tokens to be retrieved efficiently with a 16-dimensional indexer; and (3) the useful token budget is strongly query-dependent, making dynamic top- p p selection more suitable than fixed top- k k sparsification. Based on these insights, we propose RTPurbo , which retains the full KV cache only for retrieval heads and introduces a lightweight token indexer for sparse attention. By exploiting the model’s intrinsic sparsity, RTPurbo achieves sparsification with only a few hundred training steps. Experiments on long-context benchmarks and reasoning tasks show that RTPurbo preserves near-lossless accuracy while delivering substantial efficiency gains, including up to a 9.36 × \times prefill speedup at 1M context and about a 2.01 × \times decode speedup. These results suggest that strong sparse inference can be obtained from standard full-attention training without expensive native sparse pretraining. \checkdata [E-mail] † † footnotetext: \dagger Project lead § Corresponding author \sharp Work done during internship at Alibaba Figure 1 : A teaser view of the efficiency and accuracy gains of RTPurbo . 1 Introduction Long-context capability has become a core requirement for modern large language models (LLMs), especially for applications such as multi-turn dialogue, long-horizon reasoning, and document understanding [ 8 , 21 , 30 , 3 ] . However, the cost of full attention grows rapidly with context length, making long-context inference a major efficiency bottleneck. Sparse attention thus emerges as a natural direction for reducing inference cost [ 28 , 32 , 33 ] . Although many recent advances in this area replace standard full attention with more efficient alternatives, such as Kimi Delta Attention [ 22 ] and DeepSeek Sparse Attention [ 5 ] , our study suggests that models trained with full attention already exhibit substantial intrinsic sparsity. Prior work has partially revealed this phenomenon. Specifically, sparsity arises at both the head level and the token level: most heads rely primarily on local information [ 27 , 19 , 26 ] , whereas for each query only a small subset of tokens receives substantial attention mass [ 24 , 20 , 12 ] . This observation naturally raises a key question: What is the minimal surgery required to transform a full-attention model into a highly sparse one while preserving its capabilities? We identify three challenges: • Head selection: a robust metric is needed to identify the heads that genuinely require full-context access. • Efficient token indexing: a lightweight selector is needed to identify the necessary tokens efficiently. • Adaptive sparsity: because different queries require different numbers of attended tokens, a static sparsity budget can lead to information loss. Our method, RTPurbo , is designed to address these challenges with minimal adaptation. The design of RTPurbo is grounded in both LLM interpretability and theoretical analysis. Prior work on inductive heads shows that some heads implement a retrieval mechanism by attending to previously similar tokens [ 16 ] . Follow-up work further shows that, in long-context settings, these heads are primarily responsible for remote retrieval, whereas the remaining heads focus on local context [ 19 ] . This observation motivates our head-wise design: we retain the full KV cache only for retrieval heads and discard remote tokens for local heads. For retrieval heads, the key challenge is to identify relevant tokens efficiently. Our analysis shows that high-frequency components contribute little to long-range retrieval and can even interfere with it, suggesting that the retrieval process is governed largely by a low-dimensional subspace. This hypothesis is strongly supported by experiments: with our trained low-dimensional projector, we achieve over 90\% recall using only 16 dimensions. Moreover, our analysis suggests that a static Top- k selector can fail in certain cases, whereas a Top- p selector better adapts to the attention distribution and yields substantially better accuracy on both reasoning and long-context tasks. Finally, we find that self-distillation is particularly effective for recovering the performance of the sparsified model. Aligning the sparse model’s outputs with those of the original model substantially reduces the risk of overfitting, and only a few hundred training steps (about 1M label tokens) are required for this alignment stage. This result further supports our claim that RTPurbo performs only minimal surgery on the original model. To the best of our knowledge, RTPurbo is the first method to achieve such near-lossless compression with lightweight continual training. Coupled with our custom sparse kernels, RTPurbo delivers up to a 9.36 \times speedup in prefill and a 2.01 \times speedup in decoding (Figure 1 ). Importantly, the sparsification paradigm of RTPurbo remains highly interpretable. More broadly, our results highlight an overlooked point for full-attention models: even without native sparse training, a fully trained model can be sparsified with very small additional cost while preserving strong performance . This finding suggests that full-attention training remains a highly competitive and practical choice. 2 Insight Behind RTPurbo Figure 2 : Unlike most attention heads that mainly focus on local information, retrieval heads attend to regions that are semantically related to the current query token (e.g. similar pattern), even when those regions are far away in the context. 2.1 Head Specialization as a Natural Prior for Sparse Attention Recent studies suggest that attention heads in pretrained LLMs are not homogeneous, but instead specialize into distinct functional roles. In particular, prior work has shown that only a small subset of heads is responsible for retrieving distant relevant content, while many others mainly process local information [ 26 , 19 ] . We refer to this subset as retrieval heads . Their characteristic behavior is to place strong attention on earlier context surrounding semantically related content, thereby exhibiting an information-retrieval pattern, as illustrated in Figure 2 . This observation provides an important design motivation for our method: we can naturally exploit the sparsity structure that the model has already formed. Concretely, we retain the full KV cache only for retrieval heads, while for the remaining heads, which are already intrinsically sparse, we can safely discard remote tokens. 2.2 RoPE Induces a Compressible Geometry for Retrieval Heads Retrieval heads should assign high attention to semantically related tokens even when they are far apart. However, this property of retrieval heads appears, at first glance, to be in tension with RoPE [ 18 ] . For a query token at position m and a key token at position n with dimension d=2D , RoPE injects position through a rotation matrix: R_{i}(m)=\begin{pmatrix}\cos(m\theta_{i})&-\sin(m\theta_{i})\\ \sin(m\theta_{i})&\cos(m\theta_{i})\end{pmatrix},\qquad q_{m}=R(m)q,\;k_{n}=R(n)k, (1) where R(m)=\mathrm{diag}(R_{1}(m),\dots,R_{D}(m)) , and \theta_{i} decreases with