Sparse-to-Dense: A Free Lunch for Lossless Acceleration of Video Understanding in LLMs

Xuan Zhang, Cunxiao Du, Sicheng Yu, Jiawei Wu, Fengzhuo Zhang, Wei Gao, Qian Liu · 2025

Read Sparse-to-Dense: A Free Lunch for Lossless Acceleration of Video Understanding in LLMs by Xuan Zhang, Cunxiao Du, Sicheng Yu, Jiawei Wu, Fengzhuo Zhang, Wei Gao, Qian Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19155.txt.

Introduction

Xuan Zhang and co-authors wrote a paper from 2025 on speeding up video-aware language models. These models answer questions about video, but they write one word at a time. Long videos make the word chain long, so replies come slowly. The team noticed that most words in a reply need only a glance at a few key past words. Only some words need a full look at all that came before.

Their method pairs a fast, sparse reader with a slow, full one. The fast reader drafts several words at once, and the full reader checks them in one pass. The scheme needs no retraining and plugs into a working model. In tests it cut answer time by close to half at best, with no loss in quality. That is why the team calls the gain a free lunch.

Catalog record

CREATORXuan Zhang, Cunxiao Du, Sicheng Yu, Jiawei Wu, Fengzhuo Zhang, Wei Gao, Qian Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-19155

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Sparse-to-Dense: A Free Lunch for Lossless Acceleration of Video Understanding in LLMs Xuan Zhang, Cunxiao Du, Sicheng Yu, Jiawei Wu, Fengzhuo Zhang, Wei Gao, Qian Liu Abstract Video Large Language Models (Video-LLMs) suffer from high inference latency in long video processing due to their auto-regressive decoding mechanism, posing challenges for the efficient processing of video sequences that are usually very long. We observe that attention scores in Video-LLMs during decoding exhibit pronounced sparsity, with computational focus concentrated on a small subset of critical tokens. Motivated by this insight, we introduce Sparse-to-Dense ( StD ), a novel decoding strategy that integrates two distinct modules: a sparse module that rapidly generates speculative tokens using efficient top- K K attention, and a dense module that verifies these tokens in parallel via full self-attention. This collaborative approach accelerates Video-LLMs losslessly, effectively offering a free lunch for video understanding. StD is a plug-and-play solution requiring no fine-tuning or architectural changes and achieves up to a 1.94 × \times wall time speedup while preserving model performance. It enables a seamless conversion of standard Video-LLMs into sparse counterparts, unlocking efficient long-video processing without sacrificing accuracy. 1 Introduction Recent advances in Video Large Language Models (Video-LLMs), which combine large language models with video understanding, have achieved exceptional performance on tasks like video question answering and captioning Lin et al. 2024a ; Cao et al. 2024 ; Zhang et al. 2025a . A common practice in Video-LLMs is representing a video as a sequence of image frames, which results in extremely long token sequences that can strain computational resources. For instance, a 1-hour video sampled at 5-second intervals produces 720 frames, which translates to 141,120 visual tokens in VILA Lin et al. 2024a . These extremely long token sequences cause Video-LLMs to suffer from high inference latency when processing lengthy videos, making real-time applications challenging. This latency is primarily introduced by the auto-regressive nature of current Video-LLMs, where each new token must attend to all preceding tokens, creating substantial memory and computational challenges.While mechanisms like key-value (KV) caching are employed to store pre-computed key and value tensors and reduce redundant re-computation, frequent access to the cache imposes heavy demands on memory bandwidth due to the growing amount of KV cache with the increasing sequence length. This significantly reduces the throughput of Video-LLMs. A common approach to addressing this problem is KV cache compression Du et al. 2024b ; Chen et al. 2024b ; Lin et al. 2024b ; Zhang et al. 2025b or quantization Su et al. 2025 ; Hooper et al. 2024 ; Liu et al. 2024 at test time. However, these methods introduce discrepancies between training and inference, degrading the performance of LLMs. In this paper, we aim to build a lossless acceleration method designed specifically for Video-LLMs that preserves the exact output distribution of the original model. Although speculative decoding Leviathan et al. 2023 ; Chen et al. 2023 ; Hou et al. 2025 meets this requirement, it usually requires an extra draft model, which is expensive for Video-LLMs. In contrast, we observe that Video-LLMs exhibit a unique structural property, attention sparsity, which can serve as a training-free and plug-and-play draft model. Specifically, retaining only the top- K KV caches in the attention layers preserves the original predictions for approximately 95% of tokens (empirically verified), suggesting that most attention heads contribute minimally to the final output. Motivated by this observation, we introduce a novel decoding method called Sparse-to-Dense ( StD ), which leverages the sparse structure of Video-LLMs as its draft model. This design eliminates the need for an extra trained draft model, making StD a plug-and-play solution. We refer to the original Video-LLM as the dense model because it decodes using the full KV cache, whereas the model with top- K attention is termed the sparse model. Both models share identical architectures, differing only in how they compute attention. Therefore, we do not need additional GPU memory to store the sparse model, nor does it require any extra training. The top - K attention in the sparse model boosts decoding speed while sacrificing some token quality, whereas the dense model is slower but guarantees accuracy. We use the sparse model to auto-regressively draft the next \gamma tokens, while the dense model verifies them in parallel. This approach avoids redundant full KV cache memory and ensures the outputs exactly match those of the original Video-LLM. We conduct experiments on representative Video-LLMs including LLaVA-OneVision Li et al. 2024a and Qwen2-VL Wang et al. 2024 , evaluating them on video understanding benchmarks like MLVU Zhou et al. 2024 and VideoMME Fu et al. 2024 . Experiment results show that our StD , serving as a tuning-free, plug-and-play solution, achieves up to a 1.94 \times of video input processing without any performance degradation. It is immediately deployable, requiring only 20 lines of code to transform an original Video-LLM into a sparse Video-LLM, and it does not require any extra training to deploy the draft model. 2 Observation In this section, we investigate the disparity in decoded tokens between two configurations of Video-LLMs: 1) sparse top- K KV cache: utilizing only the top- K KV caches based on the highest attention weights; and 2) dense full KV cache: employing the complete set of KV caches. We conduct experiments using the Qwen2-VL-7B Wang et al. 2024 model on randomly selected samples from MLVU Zhou et al. 2024 , and Video-MME Fu et al. 2024 datasets. We evaluate the next-token prediction accuracy of the model when employing sparse attention with top- K KV caches. Our findings indicate that the model with sparse attention maintains an average token prediction accuracy exceeding 95%. This high accuracy suggests that for the majority of decoded tokens, only the top- K KV caches are necessary. However, it is important to note that the 95% accuracy is measured per individual token and does not accumulate across multiple tokens. For instance, the accuracy of correctly predicting five consecutive tokens drops to approximately (95\%)^{5}\approx 77\% . 3 Method In this section, we present Sparse-to-Dense ( StD ), a method designed to achieve lossless acceleration for Video-LLMs. We refer to the original model \mathcal{M} as the dense model, as it requires the full KV cache during decoding, while the sparse model \mathcal{M}_{s} uses sparse attention. Although \mathcal{M}_{s} is faster, it is somewhat less accurate. Unlike traditional speculative decoding, which relies on an additional draft model, our approach leverages \mathcal{M}_{s} with the same parameters as \mathcal{M} . The only difference is that \mathcal{M}_{s} loads a reduced KV cache to perform sparse attention, eliminating the need for extra GPU memory to store another model’s parameters. In the following subsections, we will detail the decoding procedure and the design of the sparse model. 3.1 Decoding Procedures In our StD , the sparse model \mathcal{M}_{s} functions as a draft model to propose potential next \gamma tokens, while the dense model \mathcal{M} verifies them to derive the final output sequence. Given an input sequence \{x_{0},\cdots,x_{m-1}\} , consisting of visual and textual tokens , the sparse model \mathcal{M}_{s} auto-regressively generates \gamma subsequent token candidates \{x_{m},\cdots,x_{m+\gamma-1}\} . Because the tokens proposed by the sparse model \mathcal{M}_{s} might not align with those predicted by the dense model \mathcal{M} ,

Read the complete text