Optimizing Native Sparse Attention with Latent Attention and Local Global Alternating Strategies
Yuxuan Hu, Jianchao Tan, Jiaqi Zhang, Wen Zan, Pingwei Sun, Yifan Lu, Yerui Sun, Yuchen Xie, Xunliang Cai, Jing Zhang · 2025
Read Optimizing Native Sparse Attention with Latent Attention and Local Global Alternating Strategies by Yuxuan Hu, Jianchao Tan, Jiaqi Zhang, Wen Zan, Pingwei Sun, Yifan Lu, Yerui Sun, Yuchen Xie, Xunliang Cai, Jing Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-00819.txt.
Introduction
Yuxuan Hu and co-authors analyze Native Sparse Attention, a design that lets large language models handle long texts by attending to only part of the context at each step. They look for structural changes that help long range information move through the network.
Their main change alternates attention styles across layers: some layers use local sliding windows, while others use compressed or selective global views. They also add latent attention variants that shrink the memory cache by half compared with the original design. On models up to about a billion parameters, the revised system matches or beats full attention on reasoning and long text tasks, suggesting a practical route to cheaper long context models.
Catalog record
| CREATOR | Yuxuan Hu, Jianchao Tan, Jiaqi Zhang, Wen Zan, Pingwei Sun, Yifan Lu, Yerui Sun, Yuchen Xie, Xunliang Cai, Jing Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-00819
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Optimizing Native Sparse Attention with Latent Attention and Local Global Alternating Strategies Yuxuan Hu, Jianchao Tan, Jiaqi Zhang, Wen Zan, Pingwei Sun, Yifan Lu, Yerui Sun, Yuchen Xie, Xunliang Cai, Jing Zhang Abstract In this work, we conduct a systematic analysis of Native Sparse Attention (NSA) and propose targeted improvements that enhance long-context modeling. A key insight is that alternating between local (sliding-window) and global (compression/selective) attention across layers, rather than using fixed patterns, enables more effective propagation of long-range dependencies and substantially boosts performance on long-sequence tasks. Meanwhile, we further refine NSA’s branches with Latent Attention that the sliding-window branch is enhanced with Multi-head Latent Attention (MLA) while compression and selective branches adopt Group-head Latent Attention (GLA). These changes reduce KV-cache memory by 50% versus NSA while improving the model’s common-sense reasoning and long-text understanding capabilities. Experiments on models from 340M to 1.3B parameters (trained on 15B and 100B tokens) show our method matches or exceeds full attention and native sparse attention in both common-sense reasoning and long-context understanding tasks. 1 Introduction Benefiting from the application of transformer-based large language models DeepSeek-AI et al. 2025b ; Team et al. 2025a (LLMs) in deep reasoning DeepSeek-AI et al. 2025a ; Team et al. 2025b , multi-turn agent systems, and codebase-level code comprehension, the research community has been increasingly focusing on the capabilities of LLMs in handling long-context inputs and test-time computation. As sequence lengths increase, the high computational complexity of the attention module has emerged as a significant efficiency bottleneck, constraining the further development of LLMs and necessitating the design of more efficient model architectures. Given the inherent sparsity of softmax attention Song et al. , sparse attention has emerged as a promising strategy to accelerate attention computations, especially under long-context scenarios. In such cases, only a subset of critical key-value pairs interacts with the query at each generation step. Existing research has proposed a variety of training-free and post-training methods for selecting these key-value pairs, including KV-cache eviction techniques Xiao et al. , as well as index-based Tang et al. 2024 ; Gao et al. 2025b , sampling-based Chen et al. , clustering-based Liu et al. 2025 , and hash-based approaches Desai et al. for KV-cache selection. Despite their significant potential, these training-free and post-training techniques fail to fully explore model sparsity. To this end, recent research has introduced natively trainable sparse attention mechanisms Lu et al. 2025 ; Yuan et al. 2025 , among which native sparse attention Yuan et al. 2025 (NSA) stands out as the most widely recognized and promising approach. Figure 1: Overview of Llama-like Models Incorporating Full Attention, Native Sparse Attention, and Alternating Sparse Attention. Figure 2: Overview of ASA’s architecture. In ASA, consecutive attention layers alternate between compressed selective attention and sliding window attention. Furthermore, GLA and MLA replace the GQA mechanism used in NSA to enhance model expressiveness. To improve training efficiency, every fourth token in the compressed selective attention module inherits the block index of the first token in its block. In the NSA framework, dense attention is decomposed into three components: sliding-window attention, compressed attention, and selective attention. Through experimental analysis of these branches, we observe that sliding-window attention plays the dominant role in common sense reasoning, while compression and selective attention primarily serve to enrich the model with global contextual information. Additionally, compared to using the same sparsity level across all layers, we found that an imbalanced sparsity distribution yields better performance on long retrieval tasks. Building on this insight, we improve NSA by introducing targeted enhancements tailored to the distinct functional roles of each attention branch. The outcome is our proposed method, Alternating Sparse Attention (ASA), a sparse attention architecture explicitly designed for efficient and effective language modeling. As shown in Figure 1 , ASA structures attention layers at the individual layer level into two complementary types: sliding-window attention, which effectively models local contextual information, and compress/selective attention, which efficiently captures long-range global context. These two mechanisms are alternated in a strict one-to-one pattern across layers, ensuring a balanced and synergistic representation of both local and global information throughout the model. On the other hand, NSA is originally implemented based on grouped query attention (GQA). While GQA demonstrates strong effectiveness, certain attention mechanisms such as multi-head latent attention (MLA) have shown even better performance. To further enhance NSA, we replaced GQA with MLA in ASA. However, MLA is equivalent to multi head attention (MHA) during training, which presents challenges when integrating MLA with sparse attention mechanisms. To address this issue, we introduce a grouping mechanism into MLA, enabling it to better adapt to sparse attention mechanisms. We conduct a thorough evaluation of ASA using transformer models with 340M and 1.3B parameters, trained on 15B and 100B tokens sampled from the SlimPajama dataset. The trained models are evaluated across multiple task categories, including general common sense reasoning, long-context retrieval and long-context understanding. Experimental results show that ASA achieves performance on par with, or even exceeds, that of the full attention baseline, while outperforming existing sparse attention approaches. Moreover, compared to the full attention baseline, ASA reduces the KV-cache storage overhead by 50%, offering a significant gain in memory efficiency without compromising model quality. 2 Related Work With the increasing demand for processing long sequences in large language models, the quadratic complexity of vanilla attention has become a significant bottleneck. To address this challenge, a wide range of efficient attention mechanisms have been proposed Sun et al. 2025 , where sparse attention is considered a promising approach. Sparse attention mechanisms aim to enhance computational efficiency by selectively computing attention scores over a strategically chosen subset of key-value pairs, rather than exhaustively attending to all possible token interactions. By exploiting the intrinsic sparsity observed in attention distributions, these approaches substantially reduce computational complexity, often achieving sub-quadratic scaling, while preserving the model’s capacity to capture long-range contextual dependencies. Common sparse attention paradigms include attention sinks Xiao et al. , sliding window attention Fu et al. 2025 , selective attention, and hybrid architectures that combine multiple sparsity patterns. A salient feature of many sparsity-based methods is their training-free design, which allows seamless integration with pre-trained dense models without requiring additional fine-tuning. For instance, techniques such as Streaming LLM Xiao et al. , SnapKV Li et al. 2024 , and PyramidKV Cai et al. 2025 enforce a fixed-size cache during autoregressive decoding; as new key-value states are generated, they dynamically evict less salient states based on learned or heuristic attention scores. In contrast, methods like Quest Tang et al. 2024 , infLLM Xiao et al. 2024 , ClusterKV Liu et al. 2025 , and RetroInfer Chen et al. 2025 retain the full sequence of key-value states and construct auxiliary indices to enable efficient retrieval. At each decoding step, only the most contextually relevant states are retrieved and attended to, thereby maintaining high fidelity while reducing computational overhead. Further advances include approaches such as SeerAttention Gao et al. 2025b and SeerAttention-R Gao et al. 2025a , which introduce sparsity during the fine-tuning phase by incorporating distillation objectives that align dense softmax attention with sparse approximations. This regularization encourages the model to learn to prioritize semantically critical key-value pairs, thereby improving both efficiency and retrieval accuracy.