OmniDrop: Layer-wise Token Pruning for Omni-modal LLMs via Query-Guidance

Yeo Jeong Park, Hyemi Jang, Minseo Choi · 2026

Read OmniDrop: Layer-wise Token Pruning for Omni-modal LLMs via Query-Guidance by Yeo Jeong Park, Hyemi Jang, Minseo Choi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-14458.txt.

Introduction

Park and colleagues, in work published in 2026, propose OmniDrop, a layer-wise pruning framework for omni-modal models. Token explosion from audio and video bottlenecks live use and long reasoning. Existing methods prune at input on similarity proxies, which often fail. OmniDrop prunes audiovisual tokens in decoder layers, letting early layers fuse information before deeper layers remove tokens. Text queries guide task-adaptive pruning, and a diversity score balances survival. Across benchmarks, OmniDrop beats baselines by up to 3.58 points, cutting latency by 40 percent and memory use by 14.7 percent.

Catalog record

CREATORYeo Jeong Park, Hyemi Jang, Minseo Choi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50947
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.14458

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

OmniDrop: Layer-wise Token Pruning for Omni-modal LLMs via Query-Guidance Thanks: Corresponding Author Yeo Jeong Park, Hyemi Jang, Minseo Choi Abstract Omni-modal large language models have demonstrated remarkable potential in holistic multimodal understanding; however, the token explosion caused by high-resolution audio and video inputs remains a critical bottleneck for real-time applications and long-form reasoning. Existing omni-modal token compression methods typically prune tokens at the input embedding level, relying on audio-video similarity or temporal co-occurrence as proxies for semantic relevance. In practice, such assumptions are often unreliable. To address this limitation, we propose OmniDrop, a training-free, layer-wise token pruning framework that progressively prunes audiovisual tokens within the LLM decoder layers rather than at the input-level, allowing early layers to preserve sufficient omni-modal information fusion before aggressively removing tokens in deeper layers. We further utilize text queries as guidance for modality-agnostic and task-adaptive token pruning. We also introduce a temporal diversity score that encourages balanced token survival to preserve global temporal context. Experimental results across various audiovisual benchmarks demonstrate that OmniDrop outperforms all baselines by up to 3.58 points while reducing prefill latency by up to 40% and memory usage by up to 14.7%. 1 Introduction The recent paradigm shift in large language models (LLMs) has led to the emergence of Omni-modal LLMs (Omni-LLMs) 4 ; 9 ; 15 ; 19 ; 28 ; 30 , which seamlessly integrate and process multiple modalities such as text, audio, and video. By enabling a more holistic understanding of complex environments, Omni-LLMs have demonstrated significant potential in diverse applications, ranging from real-time multimodal assistants to sophisticated video reasoning tasks 27 ; 23 ; 20 ; 6 . Despite their impressive capabilities, a critical bottleneck remains: computational efficiency. A primary challenge in Omni-LLMs is the token explosion caused by high-resolution temporal data. Audio and video signals produce significantly more tokens than text 22 ; for instance, a mere one-minute video clip is projected into more than 10k audio and video tokens by encoders in Qwen2.5-Omni 28 . This massive sequence length leads to quadratic growth in both computational and memory overheads 25 ; 26 , hindering real-time interactions and limiting the model’s scalability for long-form content. Recent studies have introduced various token compression methods to address these inefficiencies, including audio-guided compression 31 , modality-asymmetric compression 7 , and semantic-aware alignment 18 . Despite their methodological diversity, these methods share a common design choice: they perform compression at the input embedding level, assuming that audio-video tokens are semantically aligned either through a shared representation space or through temporal co-occurrence. However, these assumptions are often unreliable for two reasons. First, audio and video embeddings may not be sufficiently aligned in the shared input embedding space. Although audio and video tokens are projected into the same LLM embedding space, they occupy distinct subspaces, suggesting limited cross-modal alignment before LLM processing. Second, temporal co-occurrence does not necessarily imply semantic correspondence. An audio event and a visual event may occur at the same time but describe different aspects of the scene or even originate from different sources. For example, background music, off-screen narration, ambient sounds, and scene transitions may carry information that is not directly grounded in the visible content. As a result, input-level pruning 31 ; 18 that relies on embedding-level distance between audio and video tokens or on temporal co-occurrence may discard important modality-specific evidence and lead to suboptimal token selection. In contrast, we empirically observe that audio and video tokens become progressively integrated through the LLM decoder layers. As the fusion of omni-modal representations progresses, the model begins to distinguish informative tokens from the middle layers onward. We propose OmniDrop, a training-free, layer-wise token pruning framework built on three key ideas. First, we prune audiovisual tokens progressively across decoder layers rather than at the input stage. This design allows shallow layers to first exchange omni-modal information before redundant tokens are removed in deeper layers. Second, we use the text query as a semantic mediator between modalities. Instead of directly comparing audio and video tokens, we evaluate how each modality responds to the text query. This enables dynamically preserving more audio tokens for audio-centric tasks and more visual tokens for video-centric tasks, ensuring minimal information loss during compression. Third, due to query-guided pruning, tokens from a few highly responsive tokens may dominate, while other tokens are wiped out. To counteract this, we introduce a temporal diversity score (TDS) built upon the chunked prefilling mechanism commonly used in Omni-LLMs, encouraging balanced token survival across temporal segments. Experimental results on diverse audiovisual benchmarks 8 ; 13 ; 29 ; 10 show that our method consistently outperforms existing omni-modal token compression approaches. Our framework achieves superior efficiency-performance trade-offs, offering a practical solution for scalable and responsive Omni-LLMs. Our contributions are summarized as follows: • We analyze omni-modal processing in Omni-LLMs and show that input-level audio-video similarity provides a weak signal for token pruning, as audio and video embeddings are not sufficiently aligned before LLM processing. • We propose OmniDrop, a training-free, layer-wise token pruning framework for Omni-LLMs. It adopts a progressive pruning schedule that leverages text queries to estimate query-conditioned audio/video token relevance, enabling task-adaptive token pruning without training or architectural modifications. • We validate OmniDrop through extensive experiments on diverse audiovisual understanding benchmarks. Our method aggressively reduces the token retention ratio to as low as 20%, achieving up to 3.58-point gains over other baselines while reducing prefill latency by up to 40% and memory usage by up to 14.7%. 2 Preliminary & Motivation 2.1 Architecture of Omni-LLMs Typically, an Omni-LLM consists of multimodal encoders—an audio encoder \Phi_{A} and a video encoder \Phi_{V} —integrated with an LLM backbone. Given audio data \mathcal{A} and video data \mathcal{V} , the encoders project the multimodal inputs into the joint embedding space of the LLM: Z_{A}=\Phi_{A}(\mathcal{A})\in\mathbb{R}^{n_{A}\times d},\hskip 10.00002ptZ_{V}=\Phi_{V}(\mathcal{V})\in\mathbb{R}^{n_{V}\times d}, (1) where n_{A} and n_{V} denote the number of audio and video embedding tokens, respectively, and d represents the hidden dimension of the LLM. These projected embeddings Z_{A} and Z_{V} are then injected into the LLM in a manner analogous to text tokens. To ensure temporal alignment between the visual and auditory streams, Omni-LLMs employ a time-interleaving method. The audiovisual sequence is segmented into several chunks based on a fixed temporal window ( e.g. , 2 seconds). Within each chunk i , the corresponding video representations V_{i}\in\mathbb{R}^{n_{v}\times d} and audio representations A_{i}\in\mathbb{R}^{n_{a}\times d} are arranged sequentially, where n_{v} and n_{a} represent the number of video and audio tokens per chunk. This structured arrangement ensures that tokens from both modalities share the same temporal context are processed simultaneously. Consequently, the final input sequence \mathbf{X} fed into the LLM is organized as a concatenated stream: \mathbf{X}=[\text{T}_{sys},V_{1},A_{1},V_{2},A_{2},\dots,V_{m},A_{m},\text{T}_{query}], (2) where \text{T}_{sys} is the system prompt, m is the total number of chunks and \text{T}_{query} represents the user’s instructions or query prompts. Figure 1: (a) PCA visualization of audio and video token distributions in the LLM embedding space. (b) Distribution of cosine similarities among audio-audio, video-video and audio-video token pairs. 2.2 Revisiting the utility of embedding-level cross-modal similarity Recent token compression methods 31 ; 18 ; 11 for Omni-LLMs rely on audio-video correspondence at the input embedding level. However, this signal might be unreliable, and we provide multiple evidence below. Pre-LLM embeddings are

Read the complete text