DASH: Dynamic Audio-Driven Semantic Chunking for Efficient Omnimodal Token Compression
Bingzhou Li, Tao Huang · 2026
Read DASH: Dynamic Audio-Driven Semantic Chunking for Efficient Omnimodal Token Compression by Bingzhou Li, Tao Huang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-15685.txt.
Introduction
Li and Huang trim token load in models that read audio and video together. Long mixed token streams make such omnimodal models costly to run. Fixed windows and attention pruning miss the segment shape of sound and image. Their DASH method uses audio as a semantic anchor. Cosine gaps mark segment borders, which then guide video tokens. A three-signal score picks which tokens survive inside each segment. Tests on three benchmarks keep accuracy while cutting more tokens.
Catalog record
| CREATOR | Bingzhou Li, Tao Huang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47341 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.15685
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DASH: Dynamic Audio-Driven Semantic Chunking for Efficient Omnimodal Token Compression Bingzhou Li, Tao Huang Abstract Omnimodal large language models (OmniLLMs) jointly process audio and visual streams, but the resulting long multimodal token sequences make inference prohibitively expensive. Existing compression methods typically rely on fixed window partitioning and attention-based pruning, which overlook the piecewise semantic structure of audio-visual signals and become fragile under aggressive token reduction. We propose Dynamic Audio-driven Semantic cHunking (DASH), a training-free framework that aligns token compression with semantic structure. DASH treats audio embeddings as a semantic anchor and detects boundary candidates via cosine-similarity discontinuities, inducing dynamic, variable-length segments that approximate the underlying piecewise-coherent organization of the sequence. These boundaries are projected onto video tokens to establish explicit cross-modal segmentation. Within each segment, token retention is determined by a tri-signal importance estimator that fuses structural boundary cues, representational distinctiveness, and attention-based salience, mitigating the sparsity bias of attention-only selection. This structure-aware allocation preserves transition-critical tokens while reducing redundant regions. Extensive experiments on AVUT, VideoMME, and WorldSense demonstrate that DASH maintains superior accuracy while achieving higher compression ratios compared to prior methods. Code is available at: https://github.com/laychou666/DASH . 1 Introduction Omnimodal large language models (OmniLLMs) [ 34 , 32 , 19 , 7 , 27 , 29 ] extend video-language systems [ 40 , 12 , 42 , 4 , 17 , 15 ] by jointly processing audio and visual streams within a unified transformer framework. By integrating speech, scene dynamics, and textual reasoning, these models enable richer multimodal understanding across tasks such as video question answering and real-world reasoning. However, this capability comes at a significant computational cost: a single video clip can produce tens of thousands of multimodal tokens, and the quadratic complexity of self-attention quickly makes inference memory- and latency-bound. Token compression [ 2 , 22 , 1 , 35 , 33 , 37 ] has become a practical solution to mitigate this bottleneck. By pruning or merging tokens before they enter the language model, prior work reduces sequence length without retraining large models. Yet despite steady progress, existing approaches share a critical assumption: multimodal tokens are treated as flat, uniformly structured sequences. Compression decisions are typically made within fixed windows [ 31 , 1 , 3 ] and guided primarily by attention scores. This positional partitioning neglects the piecewise-coherent structure of audio-visual sequences, where semantic transitions correspond to distributional shifts in embedding space. The core challenge lies in the intrinsic structure of audio-visual signals. Unlike static images, audio and video streams evolve over time with highly non-uniform semantic density. Speech contains natural boundaries ( e.g. , pauses, topic transitions, and speaker shifts) that often coincide with meaningful visual changes. These transitions delineate coherent semantic units that should ideally be preserved as atomic structures during compression. When tokens are grouped uniformly and selected via sparse attention alone, these structural transitions are fragmented or discarded, leading to disproportionate information loss under aggressive pruning. In other words, current compression strategies optimize local redundancy while overlooking global semantic organization. From this perspective, token compression can be viewed as allocating limited representational capacity across a temporally structured and semantically segmented signal. In this work, we argue that effective omnimodal compression should be guided by semantic structure rather than positional regularity. In particular, the audio stream provides a natural structural prior. Compared to visual tokens, audio embeddings exhibit sharper distributional discontinuities at semantic transitions (e.g., pauses and topic shifts), making them a more reliable boundary signal. Due to cross-modal synchronization, these audio transitions often coincide with meaningful visual changes. Audio therefore acts as a semantic anchor that can guide both segmentation and importance allocation across modalities. Figure 1 : Overview of DASH. Left : Former methods ( e.g . , OmniZip [ 31 ] ) use static fixed-size grouping (4 frames per group) that ignores semantic boundaries and selects tokens by attention alone. Right : DASH detects audio semantic boundaries, projects them onto video indices to form dynamic chunks, and scores tokens via tri-signal fusion (boundary probability + multi-scale Gaussian uniqueness + attention), with boundary-aware compression that preserves keyframes at semantic transitions. Building on this perspective, we propose DASH (Dynamic Audio-driven Semantic cHunking) , a training-free framework for structure-aware omnimodal token compression ( Fig. 1 ). DASH models the multimodal sequence as a set of dynamically inferred semantic segments instead of imposing fixed windows. Specifically, it detects boundary candidates from cosine-similarity drops between adjacent audio embeddings, treating sharp representational changes as proxies for semantic discontinuities. These boundaries induce variable-length chunks that respect content rhythm and approximate the underlying piecewise structure of the sequence. To establish explicit cross-modal alignment, detected audio boundaries are projected onto video token indices via temporal ratio mapping, yielding shared segmentation across modalities without additional learning. Within each segment, token selection becomes a multi-signal importance estimation problem. We combine (i) boundary probability as a structural prior, (ii) multi-scale Gaussian uniqueness capturing representational distinctiveness, and (iii) normalized attention reflecting model-perceived salience. Their fusion yields a smoother importance landscape ( Fig. 2 ) and alleviates the heavy-tailed sparsity of attention-only selection. DASH introduces no learnable parameters and operates as a lightweight plug-in between modality encoders and the LLM. By reallocating compression capacity according to semantic density and preserving transition-critical tokens, DASH maintains narrative continuity and cross-modal coherence under aggressive pruning. Experiments on AVUT [ 36 ] , VideoMME [ 6 ] , and WorldSense [ 9 ] with Qwen2.5-Omni (7B and 3B) show that DASH maintains competitive accuracy at 25% token retention, matching or exceeding prior methods operating at 35%, while achieving up to 3.8 \times prefill speedup and 1.7 \times end-to-end latency reduction. Our contributions are threefold. • We formulate omnimodal token compression as a structure-aware segmentation problem, showing that compression should follow semantic boundaries rather than fixed positional windows. • We propose DASH, a training-free, audio-anchored framework that detects semantic discontinuities in embedding space and propagates them across modalities to enable explicit cross-modal alignment and density-aware compression. • We demonstrate on three audio-video benchmarks that structure-aware compression enables stable performance at substantially lower token retention (25%) than prior methods (35%), achieving significant speedups with negligible overhead. 2 Related Work 2.1 Token Compression for Multimodal LLMs Token compression reduces input tokens to alleviate the computational bottleneck of LLM inference, exploiting redundancy in multimodal inputs. For image inputs, FastV [ 2 ] performs training-free inference acceleration by monitoring attention patterns at a designated LLM layer and pruning uninformative visual tokens during prefill. LLaVA-PruMerge [ 22 ] selects representative tokens via adaptive clustering and absorbs the rest through weighted averaging. ToMe [ 1 ] introduces a training-free bipartite matching algorithm that progressively merges similar token pairs across ViT layers. Other methods use attention-based selection [ 35 ] , progressive layer-wise reduction [ 33 , 37 ] , or learned projectors [ 13 ] . For video inputs, DyCoke [ 30 ] proposes a two-stage pipeline separating temporal and spatial redundancy, while others apply spatiotemporal merging [ 23 , 25 ] or attention-based pruning [ 10 , 26 ] (see [ 24 ] for a survey). However, these methods operate within single modalities and ignore cross-modal correlations. OmniZip [ 31 ] pioneers omnimodal compression using audio attention to guide video pruning with