Spava : Accelerating Long-Video Understanding via Sequence-Parallelism-aware Approximate Attention
Yuxiang Huang, Mingye Li, Xu Han, Chaojun Xiao, Weilin Zhao, Ao Sun, Ziqi Yuan, Hao Zhou, Fandong Meng, Zhiyuan Liu · 2026
Read Spava : Accelerating Long-Video Understanding via Sequence-Parallelism-aware Approximate Attention by Yuxiang Huang, Mingye Li, Xu Han, Chaojun Xiao, Weilin Zhao, Ao Sun, Ziqi Yuan, Hao Zhou, Fandong Meng, Zhiyuan Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21444.txt.
Introduction
Yuxiang Huang and nine co-authors speed up how large multimodal models watch long videos. The slow part is the first pass, where the model digests every frame. Dense attention over thousands of frames makes that pass very costly.
Older fixes either squeezed the visual input or thinned attention on a single chip. Both gave limited speed gains or hurt quality, and neither scaled to very long clips. Their method, Spava, pairs approximate attention with awareness of how work is split across chips. Splitting the load well lets models handle longer, richer videos without a matching rise in cost.
Catalog record
| CREATOR | Yuxiang Huang, Mingye Li, Xu Han, Chaojun Xiao, Weilin Zhao, Ao Sun, Ziqi Yuan, Hao Zhou, Fandong Meng, Zhiyuan Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-21444
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Spava : Accelerating Long-Video Understanding via Sequence-Parallelism-aware Approximate Attention Yuxiang Huang, Mingye Li, Xu Han, Chaojun Xiao, Weilin Zhao, Ao Sun, Ziqi Yuan, Hao Zhou, Fandong Meng, Zhiyuan Liu Abstract The efficiency of long-video inference remains a critical bottleneck, mainly due to the dense computation in the prefill stage of Large Multimodal Models (LMMs). Existing methods either compress visual embeddings or apply sparse attention on a single GPU, yielding limited acceleration or degraded performance and restricting LMMs from handling longer, more complex videos. To overcome these issues, we propose Spava , a sequence-parallel framework with optimized attention that accelerates long-video inference across multiple GPUs. By distributing approximate attention, Spava reduces computation and increases parallelism, enabling efficient processing of more visual embeddings without compression and thereby improving task performance. System-level optimizations, such as load balancing and fused forward passes, further unleash the potential of Spava , delivering speedups of 12.72 × \times , 1.70 × \times , and 1.18 × \times over FlashAttn , ZigZagRing , and APB , without notable performance loss. Code available at https://github.com/thunlp/APB . 1 Introduction The rapid advances of Large Language Models (LLMs) ( 2 ; 3 ; 9 ; 12 ) in long-context inference have catalyzed the concurrent development of Large Multimodal Models (LMMs) ( 36 ; 32 ; 18 ) , endowing LMMs with the capacity to understand extended video sequences. This capability is typically realized through the synergistic integration of visual encoders ( 10 ) with long-context LLM backbones, yielding remarkable performance on long-video benchmarks such as LongVideoBench ( 39 ) and VNBench ( 53 ) . Despite the potential in advancing long-video processing, LMMs encounter severe efficiency bottlenecks when handling ultra-long videos. An increased number of video frames results in a larger batch size for the visual encoder, leading to a rise in both compute cost and the number of generated video embeddings. Given the quadratic complexity of the attention layers employed by LLM backbones, these additional video embeddings extend the input length and slow down the inference speed. Figure 1: Spava ’s performance and inference speed using Qwen2.5VL-3B as the base LMM on VNBench (processing 64-frame 1440p videos). To alleviate the above efficiency issue, existing efforts can be categorized into two main approaches: intrinsic attention optimization and explicit input reduction . Intrinsic attention optimization ( 45 ; 25 ; 41 ) methods aim to reduce the computational burden of the LLM backbone within LMMs by optimizing the attention mechanism or minimizing the key-value (KV) cache size, but cannot address the computational cost of the feed-forward networks (FFNs) and the visual encoder. Explicit input reduction methods ( 43 ; 7 ; 46 ) aim to reduce the number of video embeddings by applying adaptive selection or pooling operations within or after the visual encoder, thereby lowering the overall computational cost. Since long-video processing typically involves computation-intensive inference, neither approach can achieve satisfactory efficiency improvements while preserving fine-grained video details. Consequently, existing approaches inevitably face a trade-off between efficiency and performance (e.g., in our experiments, the typical method SlowFast achieves less than 3\times speedup but suffers around 25\% accuracy degradation in Figure 7 and Table 1 ). As single-GPU methods usually suffer from excessive compute reduction and cannot well overcome the efficiency-performance trade-off, a third scaling dimension is necessary. Scaling computational power while suppressing quadratic complexity, i.e., by running approximate attention with reduced compute across more GPUs (referred to as hosts ), offers a promising pathway, directly addressing the core challenges and providing the following advantages. Advantage 1: Supporting strong compute density through well designed parallelism. Long-video inference lends itself naturally to parallelization when guided by effective design. The video encoding process is inherently independent across frames, allowing distribution across multiple hosts. Meanwhile, the LLM backbone can be parallelized via sequence parallelism, with context segments placed and processed on different hosts. However, the dense computation and communication in exact sequence parallelism algorithms hinder scalability to larger host counts, resulting in limited efficiency gain ( ZZRing in Figure 1 ). In contrast, approximate attention improves scalability through optimized computation and compressed communication, enabling faster inference. Advantage 2: Preserving task performance. Conventional optimizations often sacrifice fine-grained details in long-video inputs, especially when token pruning is applied for explicit input reduction. By replacing input reduction with approximate attention compute reduction, the full set of visual embeddings is preserved, thereby safeguarding task performance. Building on this perspective, we propose Spava . Our main contributions are summarized as follows: (I) We introduce Spava , a sequence-parallelism-aware approximate attention framework for accelerating long-video understanding. Unlike embedding compression or single-GPU attention optimization, Spava adopts local KV cache compression with a distributed approximate attention mechanism, well striking an efficiency-performance balance. (II) We optimize Spava from a system-level perspective by designing a load-balancing strategy across hosts, along with a two-stage attention mechanism where communication perfectly overlapped with computation. (III) We evaluate Spava on extensive benchmarks. As shown in Table 1 , the experimental results show that Spava exhibits the best tradeoff between performance and efficiency. More specifically, Spava achieves speedup of 12.72 \times , 1.70 \times , and 1.18 \times over FlashAttn , ZigZagRing , and APB , without significant performance degredation. Figure 2: The framework of Spava . The anchor block and passing block are denoted as \mathbf{B}_{a} and \mathbf{B}_{p} , while \mathbf{B}^{(h)} denotes the context block on virtual host h . \mathbf{B}_{qr} represents the query block. The video input is first encoded into embeddings using frame parallelism across hosts. After context splitting, each physical host (containing two virtual hosts) holds the anchor block, query block, and corresponding context blocks. In approximate attention, each context block is first compressed then communicated. Attention is then computed over \mathbf{B}_{a} , \mathbf{B}^{(h)} , \mathbf{B}^{(2H-1-h)} , \mathbf{B}_{p}^{(h)} , \mathbf{B}_{p}^{(2H-1-h)} , and \mathbf{B}_{qr} . The passing blocks are discarded immediately after attention. 2 Preliminaries We introduce the basic notations of Transformers and long-video inference in this section. Transformers. For an L -layer Transformer-based model, each layer consists of an attention block and an FFN block ( 37 ) . The model takes a sequence of d -dimensional embeddings \mathbf{E}\in\mathbb{R}^{n\times d} as the input, where n is the input sequence length. For each layer, given the input hidden \mathbf{H} , the query, key, and value matrices are first computed as \mathbf{Q}=\mathbf{H}\mathbf{W}_{Q},\mathbf{K}=\mathbf{H}\mathbf{W}_{K},\mathbf{V}=\mathbf{H}\mathbf{W}_{V}, and then the attention is applied as \displaystyle\mathbf{A} \displaystyle=\texttt{Attn}\left(\mathbf{Q},\mathbf{K},\mathbf{V}\right) (1) \displaystyle=\texttt{Softmax}\left(\frac{\mathbf{Q}\mathbf{K}^{\top}\odot\mathbf{M}}{\sqrt{d}}\right)\cdot\mathbf{V}, where \mathbf{M} is the attention mask. Based on the attention block, the FFN block computation is \displaystyle\mathbf{H}^{\prime}=\texttt{FFN}\left(\mathbf{A}\right). (2) We omit the notations for attention heads, position embeddings, and normalizations for simplicity. Long-Video Inference. An LMM \mathcal{M} typically consists of a visual encoder \mathcal{M}_{v} , a connector \mathcal{M}_{c} , an embedding layer \mathcal{M}_{e} , and an LLM backbone \mathcal{M}_{l} . During the long-video inference, the video is first divided into F frames, each containing H\times W d_{v} -dimensional patches. We denote the patchified input as a fourth-order tensor \mathbf{I}\in\mathbb{R}^{F\times H\times W\times d_{v}} . For the i -th frame \mathbf{I}_{i} , it is first reshaped into a second-order embedding sequence \tilde{\mathbf{I}}_{i}\in\mathbb{R}^{HW\times d_{v}} , and then encoded by the visual encoder, producing the output \mathbf{O}_{i}=\mathcal{M}_{v}(\tilde{\mathbf{I}}_{i})\in\mathbb{R}^{HW\times d_{v}} . By encoding each frame output \mathbf{O}_{i} to n_{v} d -dimensional embeddings through the connector \mathcal{M}_{c} , we get the whole video