OmniSparse: Training-Aware Fine-Grained Sparse Attention for Long-Video MLLMs
Feng Chen, Yefei He, Shaoxuan He, Yuanyu He, Jing Liu, Lequan Lin, Akide Liu, Zhaoyang Li, Jiyuan Zhang, Zhenbang Sun, Bohan Zhuang, Qi Wu · 2025
Read OmniSparse: Training-Aware Fine-Grained Sparse Attention for Long-Video MLLMs by Feng Chen, Yefei He, Shaoxuan He, Yuanyu He, Jing Liu, Lequan Lin, Akide Liu, Zhaoyang Li, Jiyuan Zhang, Zhenbang Sun, Bohan Zhuang, Qi Wu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-12201.txt.
Introduction
Feng Chen and eleven co-authors introduce OmniSparse, a sparse attention method for language models that process long videos. Attention over long video is costly, and most sparse methods act only at inference, picking key tokens by fixed patterns. That leaves a gap between training and use, and token choice is often coarse.
OmniSparse works in both training and inference, with budgets that adapt. It keeps active queries that carry broad meaning, drops most lazy local ones, and selects key-value pairs with a shared budget set at head level. It also trims the key-value cache to cut redundancy. Together these steps aim for faster long-video models without losing key detail.
Catalog record
| CREATOR | Feng Chen, Yefei He, Shaoxuan He, Yuanyu He, Jing Liu, Lequan Lin, Akide Liu, Zhaoyang Li, Jiyuan Zhang, Zhenbang Sun, Bohan Zhuang, Qi Wu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-12201
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OmniSparse: Training-Aware Fine-Grained Sparse Attention for Long-Video MLLMs Feng Chen, Yefei He, Shaoxuan He, Yuanyu He, Jing Liu, Lequan Lin, Akide Liu, Zhaoyang Li, Jiyuan Zhang, Zhenbang Sun, Bohan Zhuang, Qi Wu Abstract Existing sparse attention methods primarily target inference-time acceleration by selecting critical tokens under predefined sparsity patterns. However, they often fail to bridge the training–inference gap and lack the capacity for fine-grained token selection across multiple dimensions—such as queries, key-values (KV), and heads—leading to suboptimal performance and acceleration gains. In this paper, we introduce OmniSparse , a training-aware fine-grained sparse attention of long-video MLLMs, which is applied in both training and inference with dynamic token budget allocation. Specifically, OmniSparse contains three adaptive and complementary mechanisms: (1) query selection as lazy-active classification, aiming to retain active queries that capture broader semantic similarity, while discarding most of lazy ones that focus on limited local context and exhibit high functional redundancy with their neighbors, (2) KV selection with head-level dynamic budget allocation, where a shared budget is determined based on the flattest head and applied uniformly across all heads to ensure attention recall after selection, and (3) KV cache slimming to alleviate head-level redundancy, which selectively fetches visual KV cache according to the head-level decoding query pattern. Experimental results demonstrate that OmniSparse can achieve comparable performance with full attention, achieving 2.7 × \times speedup during prefill and 2.4 × \times memory reduction for decoding. Introduction Long-video multimodal large language models (MLLMs) ( Li et al. 2025a ; Weng et al. 2024 ; Chen et al. 2024b ) are crucial for understanding complex temporal interactions, but their application is limited by the quadratic computational cost of attention mechanism ( Vaswani et al. 2017 ) . Recent studies ( Chen et al. 2024a ; Li et al. 2025b ) primarily focus on training-free sparse attention, leveraging the inherent sparsity of attention to predefine sparsity patterns, to significantly reduce computational overhead during inference. For instance, FastV ( Chen et al. 2024a ) reduces half of the visual tokens after layer 2 based on the attention scores. MMInference ( Li et al. 2025b ) introduces a modality-aware dynamic sparse attention mechanism that performs predefined pattern search to accelerate the prefill stage. These methods achieve significant hardware efficiency but still struggle with training-inference inconsistency and fine-grained token selection across multiple dimensions, resulting in suboptimal performance and limited acceleration gains. The training-inference gap stems from the use of sparse attention exclusively at inference time, while models are trained under full attention ( Chen et al. 2025a ) . This mismatch leads to inconsistent attention patterns between training and deployment, ultimately degrading generalization and performance. In addition, fine-grained token selection aims to eliminate redundant computations by identifying and removing token-level redundancies across multiple dimensions—namely queries, key-values, and attention heads. However, existing methods typically focus on only one or two of these dimensions, limiting their ability to fully exploit the potential for computational savings and model efficiency. (a) (b) (c) Figure 1: (a) The majority of queries focus on fewer than 100 out of 2,600 tokens and can be pruned from attention with minimal performance degradation. (b) The dynamic sparsity across layers and heads suggests that head-wise budget allocation could improve efficiency, but determining an optimal budget for each head is computationally expensive. (c) Heterogeneous token focus across heads (selected keys are highlighted in red) necessitates the head-level KV selection. Data collected from LLaVA-Video-7b ( Zhang et al. 2024c ) with VideoMME ( Fu et al. 2024 ) . In this paper, we propose OmniSparse , a training-aware fine-grained sparse attention for long-video MLLMs. OmniSparse dynamically determines token selection budgets across queries, key-values, and attention heads, and applies the same sparse attention during both training and inference to ensure consistency. It adopts a Top- p token-wise sparsification strategy and primarily consists of query selection to remove most functionality-overlapped queries, key-value (KV) selection to determine the minimum token budget across heads, and KV cache slimming to reduce head-wise redundancy. The first two components are designed to alleviate the computation cost of the prefill phase, while the last component focuses on eliminating the memory redundancy of the decoding phase. Query selection as binary lazy-active classification. Although the query-key angle is designed to capture semantic similarity between tokens ( Zhang et al. 2024b ) , not all queries contribute equally. As shown in Figure 1(a) , we observe that the majority of queries focus on fewer than 100 out of 2600 tokens ( i.e. , less than 3%), typically concentrating on the attention sink ( Xiao et al. 2023 ) or spatial-temporal adjacent tokens. These “lazy” queries capture limited semantic similarity and can be safely removed from attention with minimal performance degradation, as their functionality is often overlapped with nearby queries or those from other heads (we will demonstrate in Sec. Query Selection as Binary Lazy-Active Classification ). To this end, we aim to retain “active” queries that capture broader context, while pruning most of the lazy ones that primarily focus on fewer tokens. We formulate query selection as a binary lazy-active classification task. Specifically, we construct an active probe key by aggregating all visual keys, and take the key of attention sink as the lazy probe key, which is used to absorb unnecessary weights from other keys ( Xiao et al. 2023 ) . For each query, we compute its similarity to these probe keys, where queries exhibiting high similarity with the attention sink are identified as lazy and excluded from subsequent attention computation. To avoid over-pruning of lazy queries, we preserve the queries in the first attention head, which serves as a reliable channel to maintain information integrity. KV selection with head-level dynamic budget allocation. As illustrated in Figure 1(b) and 1(c) , we observe heterogeneous attention patterns across heads: each head attends to distinct KV pairs and exhibits distinct sparsity due to varying attention distribution ( Jiang et al. 2024 ) . This disparity causes suboptimal key-value selection under the predefined Top- k strategy ( Chen et al. 2024a ; Gao et al. 2024 ) , leading to under-selection of flatter heads ( Lin et al. 2025 ) —which have relatively uniform attention weights across tokens—and over-selection of sharp heads. A straightforward solution is to dynamically allocate KV budgets per head based on their individual attention distributions. However, this fine-grained allocation incurs substantial computational overhead. To address this, we first identify the flattest attention head using kurtosis, as it typically requires a larger token budget to preserve attention recall. Then we allocate the token budget in this head and then uniformly apply it across all heads, allowing each head to select its most salient KV pairs while ensuring the overall attention recall remains above a predefined threshold. KV cache slimming to alleviate head-wise redundancy. Apart from head-wise selection of visual KV pairs, we further reduce memory overhead by conditionally fetching KV caches based on query classification during decoding. Specifically, when a decoding query is identified as lazy, we skip fetching the corresponding head’s visual KV cache entirely. This selective fetching strategy significantly decreases the memory access cost during decoding, as it avoids unnecessary KV cache retrievals. In summary, our contributions are as follows: (1) We propose OmniSparse , a