MM Tok : Multimodal Coverage Maximization for Efficient Inference of VLMs

Sixun Dong Juhua Hu Mian Zhang Ming Yin Yanjie Fu Qi Qian 🖂 {\Letter}, Arizona State University Zoom Communications University of Washington, University of Texas at Dallas Duke University [email protected],[email protected],[email protected],[email protected] · 2025

Read MM Tok : Multimodal Coverage Maximization for Efficient Inference of VLMs by Sixun Dong Juhua Hu Mian Zhang Ming Yin Yanjie Fu Qi Qian 🖂 {\Letter}, Arizona State University Zoom Communications University of Washington, University of Texas at Dallas Duke University [email protected],[email protected],[email protected],[email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-18264.txt.

Introduction

Sixun Dong and co-authors prune vision tokens for vision language models by a coverage rule in 2025, using the author list given in the text head. These models turn images into many tokens, and the excess slows them down. Past pruning used one side of the data only. Their rule asks a small token set to cover both the text and the full image set.

The pick is framed as a coverage maximization task. Tests across models and data sets back the joint approach. On one hard set, speed rose near two fold at near full accuracy. With just four tokens kept, most accuracy survived on a mid size model. The authors say coverage is a fair cross mode yardstick. Code is public.

Catalog record

CREATORSixun Dong Juhua Hu Mian Zhang Ming Yin Yanjie Fu Qi Qian 🖂 {\Letter}, Arizona State University Zoom Communications University of Washington, University of Texas at Dallas Duke University [email protected],[email protected],[email protected],[email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-18264

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MM Tok : Multimodal Coverage Maximization for Efficient Inference of VLMs Sixun Dong Juhua Hu Mian Zhang Ming Yin Yanjie Fu Qi Qian 🖂 {\Letter}, Arizona State University Zoom Communications University of Washington, University of Texas at Dallas Duke University [email protected],[email protected],[email protected],[email protected] Abstract Vision-Language Models (VLMs) demonstrate impressive performance in understanding visual content with language instruction by converting visual input to vision tokens. However, redundancy in vision tokens results in the degenerated inference efficiency of VLMs. While many algorithms have been proposed to reduce the number of vision tokens, most of them apply only unimodal information (i.e., vision/text) for pruning and ignore the inherent multimodal property of vision-language tasks. Moreover, it lacks a generic criterion that can be applied to different modalities. To mitigate this limitation, in this work, we propose to leverage both vision and text tokens to select informative vision tokens by the criterion of coverage. We first formulate the subset selection problem as a maximum coverage problem. Afterward, a subset of vision tokens is optimized to cover the text tokens and the original set of vision tokens, simultaneously. Finally, a VLM agent can be adopted to further improve the quality of text tokens for guiding vision pruning. The proposed method MMTok is extensively evaluated on benchmark datasets with different VLMs. The comparison illustrates that vision and text information are complementary, and combining multimodal information can surpass the unimodal baseline with a clear margin. Moreover, under the maximum coverage criterion on the POPE dataset, our method achieves a 1.87× speedup while maintaining 98.7% of the original performance on LLaVA-NeXT-13B. Furthermore, with only four vision tokens, it still preserves 87.7% of the original performance on LLaVA-1.5-7B. These results highlight the effectiveness of coverage in token selection. † † footnotetext: \dagger Work done during internship at Zoom Communications. † † footnotetext: 🖂 Corresponding author. Figure 1: MMTok achieves better performance across multiple benchmarks. 1 Introduction By converting the visual input to vision tokens, Vision-Language Models (VLMs) can leverage powerful pre-trained Large Language Models (LLMs) to understand visual content as text ( Liu et al. 2024b ; Li et al. 2024b ; Team et al. 2023 ) . Unlike discrete text tokens, where the information is highly compressed, current vision encoders extract vision tokens directly from the original input patches, which are redundant according to previous studies ( Bolya et al. 2022 ; He et al. 2022 ) and their count can far exceed that of text tokens. For example, given “Describe the image” with less than 10 text tokens, 2,880 vision tokens can be obtained from a single image in LLaVA-NeXT ( Liu et al. 2024a ) . Since LLMs are built on self-attention layers ( Vaswani et al. 2017 ) that have a quadratic computational cost to the total number of tokens, the large volume of vision tokens can significantly challenge the inference efficiency of VLMs. To accelerate inference, many works have been proposed to sample a subset of vision tokens for inference with LLMs without compromising performance. While some work adopts an additional training process to enable vision token selection, in this work, we will focus on the training-free paradigm to reduce optimization efforts. Our experiments also confirm that the proposed training-free method can even outperform baselines with fine-tuning. Despite different architectures of VLMs ( Team 2024 ; Bai et al. 2025 ; Guo et al. 2025 ) , the leading performance is from the one containing a separated vision encoder to obtain vision tokens ( Bai et al. 2025 ) . In that architecture, both vision tokens and text tokens are available for token selection before applying LLMs. However, most of the existing work relies on unimodality for pruning while the multimodal information has not been explored sufficiently ( Zhang et al. 2024 ; Yang et al. 2025 ; Alvar et al. 2025 ) . For example, SparseVLM ( Zhang et al. 2024 ) mainly considers text tokens from language instruction to guide the pruning of vision tokens, while VisionZip ( Yang et al. 2025 ) heavily depends on the [CLS] vision token to select informative vision tokens. By investigating vision-language tasks, we find that given the same image, the answers can be different due to user-specific text queries, while the same text instruction can be applied for different images, i.e., caption tasks. Therefore, a unimodal method is hard to capture sufficient information about target tasks, implying a suboptimal performance for token selection. In order to leverage both vision and text information to obtain informative vision tokens, in this work, we propose a multimodal strategy for efficient inference. First, we formulate the token selection problem as a maximum coverage problem , which aims to cover the target tokens with a subset of source tokens. While the source tokens are vision-only, the target ones can come from either text or vision, respectively. Therefore, the framework can explicitly combine the information from different modalities. Then, we optimize the coverage problem by maximizing a submodular function defined on the similarity between target and source tokens. Although the original problem is NP-hard ( Khuller et al. 1999 ) , a simple greedy algorithm can observe an approximate solution that is not worse than (1-1/e) of the optimal solution ( Nemhauser et al. 1978 ) . Finally, a lightweight agent can be applied to enrich text information optionally. The main contributions of this work are summarized as follows. • We introduce the coverage maximum problem for vision token selection. The problem can be formulated as maximizing a submodular function, which has an efficient algorithm to obtain a near-optimal solution with a theoretical guarantee. • We apply the coverage criterion to cover both the text tokens and the entire set of vision tokens with a subset of selected vision tokens. The text-vision and vision-vision coverage explicitly help explore multimodal information for selection. • Experiments are conducted on benchmark datasets with diverse VLMs. The superior performance of the proposed method demonstrates the effectiveness of the proposed coverage criterion for the subset selection of vision tokens. For example, the proposed MMTok can achieve overall best performance under different settings as illustrated in Figure 1 (b) and shows the potential to compress to an extremely small number of vision tokens as in Figure 1 (a). 2 Related Work VLMs, such as LLaVA ( Liu et al. 2023 ) , InstructBLIP ( Dai et al. 2023 ) , and Qwen ( Bai et al. 2025 ) , have become a cornerstone for multimodal understanding by integrating large-scale vision encoders (e.g., CLIP-ViT ( Radford et al. 2021b ) ) with pre-trained language models. These models achieve strong performance by representing images as sequences of visual tokens, but their inference cost grows quadratically with token count, highlighting the need for more efficient processing. Many vision token selection methods have been proposed recently, but most of them only rely on unimodal information for pruning ( Yang et al. 2025 ; Chen et al. 2024 ; Zhang et al. 2024 ; Alvar et al. 2025 ) . For example, VisionZip ( Yang et al. 2025 ) and FastV ( Chen et al. 2024 ) prune tokens using pre-trained attention signals, either ranking by [CLS] token attention (VisionZip) or discarding low-attention vision tokens

Read the complete text