FastAV: Efficient Token Pruning for Audio-Visual Large Language Model Inference
Chaeyoung Jung, Youngjoon Jang, Seungwoo Lee, Joon Son Chung · 2026
Read FastAV: Efficient Token Pruning for Audio-Visual Large Language Model Inference by Chaeyoung Jung, Youngjoon Jang, Seungwoo Lee, Joon Son Chung (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-13143.txt.
Introduction
Chaeyoung Jung, Youngjoon Jang, Seungwoo Lee, and Joon Son Chung speed up models that read both sound and sight. Such audio visual models join a large language core with ears and eyes. The join makes their token load swell, so runs cost more time and power.
Token prune work has cut load for plain text and for sight plus text models. It had not been fit to audio visual models. Their method, FastAV, is the first prune frame built for that case. It drops the least need tokens while it keeps the key sound and sight clues. For live apps such as video chat aids, cuts like this can make big models fit to run.
Catalog record
| CREATOR | Chaeyoung Jung, Youngjoon Jang, Seungwoo Lee, Joon Son Chung |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-13143
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FastAV: Efficient Token Pruning for Audio-Visual Large Language Model Inference Chaeyoung Jung, Youngjoon Jang, Seungwoo Lee, Joon Son Chung Abstract In this work, we present FastAV, the first token pruning framework tailored for audio-visual large language models (AV-LLMs). While token pruning has been actively explored in standard large language models (LLMs) and vision-language models (LVLMs), its application to AV-LLMs has received little attention, even though multimodal integration substantially increases their token demands. To address this gap, we introduce a pruning strategy that utilizes attention weights to identify tokens emphasized at different stages and estimates their importance. Building on this analysis, FastAV applies a two-stage pruning strategy: (1) global pruning in intermediate layers to remove broadly less influential tokens, and (2) fine pruning in later layers considering the impact on next token generation. Notably, our method does not rely on full attention maps, which makes it fully compatible with efficient attention mechanisms such as FlashAttention. Extensive experiments demonstrate that FastAV reduces FLOPs by more than 40% on two representative AV-LLMs, while preserving or even improving model performance. † † address: Korea Advanced Institute of Science and Technology, South Korea 1 Introduction Recent advances in large language models (LLMs) [ 2 , 3 , 19 , 29 ] have driven growing interest in extending them to multimodal settings by incorporating vision, audio, and other modalities to tackle more complex tasks. However, as model sizes and the number of processed tokens grow substantially, both memory consumption and computational cost increase sharply, leading to degraded inference efficiency. To address these challenges, recent studies have explored various strategies to improve model efficiency, ranging from structure-level pruning [ 22 , 10 , 8 , 26 , 36 ] to token-level pruning [ 9 , 4 , 18 , 30 , 16 ] , all aiming to reduce overhead while preserving model performance. Several pruning strategies for LLMs have been proposed and can be categorized as post-training, training-free, and inference-time methods. Post-training methods compress pretrained models via structured pruning, including gradient-guided pruning with LoRA recovery [ 22 ] and refined importance metrics [ 35 , 10 ] . Training-free methods enhance model efficiency without fine-tuning, such as Hessian-based reconstruction [ 8 ] , activation-preserving criteria [ 26 ] , and dynamic sparse masks [ 36 ] . For inference, LazyLLM [ 9 ] dynamically prunes tokens during generation by selectively computing key–value pairs for tokens deemed important, even if they are discarded in earlier layers. Extending inference-time token pruning to large vision-language models (LVLMs), recent studies focus on efficient visual token selection. FastV [ 4 ] prunes visual tokens in later layers based on adaptive attention patterns learned in early layers. VTW [ 18 ] removes mid-to-late visual tokens using attention-sink and information-migration signals. TopV [ 30 ] prunes less important visual tokens using a visual-aware cost function. Unlike the extensive study of token pruning in LLMs and LVLMs, its application to audio-visual LLMs (AV-LLMs) remains underexplored. AV-LLMs [ 7 , 25 , 28 , 6 , 32 , 33 , 20 , 12 , 21 , 34 , 5 , 24 , 37 , 11 , 14 ] handle rich multimodal streams of audio, video, and text, leading to an inevitable increase in token counts. However, it is still unclear how many of these tokens are essential for downstream reasoning. To bridge this gap, we present FastAV, a framework that analyzes AV-LLMs to identify key audio-visual tokens and prune less informative ones at inference, enabling efficient computation without compromising performance. To gain deeper insights into the behavior of AV-LLMs, we analyze attention rollout [ 1 ] , which accumulates attention across layers to trace token influence. Higher rollout values indicate stronger influence, highlighting critical tokens. We apply this to two representative AV-LLMs, VideoLLaMA2 [ 6 ] and video-SALMONN2 [ 28 ] . As shown in Fig. 1 , our rollout analysis reveals that after tokens pass through the middle layer (layer 14), attention increasingly concentrates on earlier tokens, particularly on the left side of the red line, forming anchor-like patterns [ 13 ] in both models. In the first row of Fig. 2 , we observe that rollout in early layers such as layer 4 remains uniform, whereas by layer 14 it clearly shifts toward earlier tokens. This rollout pattern then persists through deeper layers, including layer 24. Motivated by this observation, we apply global pruning at the middle layer to roughly remove less influential tokens. Since this stage already eliminates more than half of the tokens, the remaining ones must be pruned with caution to avoid harming model performance. To further increase inference efficiency while preserving important information, we introduce a fine pruning stage. Following prior studies [ 23 , 15 ] , we estimate token importance using the attention weights of the last query token, and progressively remove the least important tokens at each subsequent layer based on their contribution to the final answer, further reducing computational costs. Figure 1: Attention rollout at the 14th layer in VideoLLaMA2 [ 6 ] and video-SALMONN2 [ 28 ] . Accumulated attention concentrates on earlier tokens, highlighting their pivotal role in carrying the most essential information. Experimental results show that FastAV reduces FLOPs by over 40% in two representative AV-LLMs, VideoLLaMA2 and video-SALMONN2, while maintaining or even improving performances across three datasets. Furthermore, applying FastAV to both AV-LLMs drastically reduces audio tokens (e.g., from 1,496 to 10 in VideoLLaMA2) without degrading performance, highlighting their essential yet compact role and underscoring the need for efficient strategies to leverage multimodal information, particularly audio. Our contributions are threefold. First, we present FastAV, the first systematic analysis of the role and impact of audio and video tokens in AV-LLMs. Second, we introduce a two-stage pruning strategy that combines global pruning guided by attention rollout with fine grained pruning based on last query token importance. Third, we show that FastAV reduces FLOPs by more than 40% while maintaining or even improving model performance, enabling efficient inference. 2 Method 2.1 AV-LLM Inference For video data, a visual encoder extracts frame-level features into a sequence of M visual tokens \mathbf{X}^{vis}=[\mathbf{x}_{1}^{vis},...,\mathbf{x}_{M}^{vis}] . Similarly, an audio encoder transforms audio signals into U audio tokens \mathbf{X}^{aud}=[\mathbf{x}_{1}^{aud},...,\mathbf{x}_{U}^{aud}] . Text inputs, including a question, are tokenized into E textual tokens \mathbf{X}^{lang}=[\mathbf{x}_{1}^{lang},...,\mathbf{x}_{E}^{lang}] . These modality-specific tokens are concatenated to form a unified sequence \mathbf{X}=[\mathbf{x}_{i}]_{i=1}^{K} , where K=M+U+E , which serves as the input to the LLM decoder. During inference, the model generates outputs autoregressively, predicting the next token conditioned on all input modalities and previously generated outputs: \mathbf{y}_{t}\sim p(\mathbf{y}_{t}|\mathbf{X}^{vis},\mathbf{X}^{aud},\mathbf{X}^{lang},\mathbf{y}_{<t}), (1) where \mathbf{y}_{t} is the token at timestep t and \mathbf{y}_{<t} are past tokens. Figure 2: Attention rollout and weights across layers in VideoLLaMA2 [ 6 ] . Attention rollout reveals a progressive focus on earlier tokens, stabilizing around the middle layers and persisting in deeper layers, whereas raw attention weights alone do not exhibit such a clear pattern. 2.2 FastAV Analysis of audio-visual token importance via attention rollout. To explore the contribution of multimodal tokens across transformer layers, we use the attention rollout method [ 1 ] , which tracks how information propagates through the network as depth of the layer increases. Attention rollout aggregates attention weights from the initial