TOPS: First-Principles Visual Token Pruning via Constructing Token Optimal Preservation Sets for Efficient MLLM Inference
Tinghao Wang, Yichen Guo, Rui Huang, Zheng Lu, Qizhe Zhang,, Chenxi Li, Yuan Zhang, Jiajun Cao, Zhirong Shen, Yaosong Du,, Guangyan Gan, Wenya Wang, Lin William Cong, Shanghang Zhang, State Key Laboratory of Multimedia Information Processing,[-0.4em] School of Computer Science, Peking University · 2026
Read TOPS: First-Principles Visual Token Pruning via Constructing Token Optimal Preservation Sets for Efficient MLLM Inference by Tinghao Wang, Yichen Guo, Rui Huang, Zheng Lu, Qizhe Zhang,, Chenxi Li, Yuan Zhang, Jiajun Cao, Zhirong Shen, Yaosong Du,, Guangyan Gan, Wenya Wang, Lin William Cong, Shanghang Zhang, State Key Laboratory of Multimedia Information Processing,[-0.4em] School of Computer Science, Peking University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-27161.txt.
Introduction
Wang and colleagues prune the flood of visual tokens that slows multimodal language models. Such models split images into many tokens, and attention cost grows steeply with token count. Earlier pruning rules keep redundant tokens or ignore the user's question. Their TOPS module instead builds Token Optimal Preservation Sets, chosen from first principles. During 2026, an information-theoretic analysis yields three rules for keeping tokens: task relevance, information coverage, and semantic diversity.
TOPS needs no training and plugs into many model backbones. Tests span seven backbones and fourteen benchmarks under varied pruning settings. Prior methods are beaten across those settings. On one leading model family, over three quarters of visual tokens are removed while benchmark scores hold at or above full-token levels. Cutting redundant tokens can even reduce hallucination slightly. Lightweight multimodal design gets a principled, ready-to-use pruning tool.
Catalog record
| CREATOR | Tinghao Wang, Yichen Guo, Rui Huang, Zheng Lu, Qizhe Zhang,, Chenxi Li, Yuan Zhang, Jiajun Cao, Zhirong Shen, Yaosong Du,, Guangyan Gan, Wenya Wang, Lin William Cong, Shanghang Zhang, State Key Laboratory of Multimedia Information Processing,[-0.4em] School of Computer Science, Peking University |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.27161
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TOPS: First-Principles Visual Token Pruning via Constructing Token Optimal Preservation Sets for Efficient MLLM Inference Tinghao Wang, Yichen Guo, Rui Huang, Zheng Lu, Qizhe Zhang,, Chenxi Li, Yuan Zhang, Jiajun Cao, Zhirong Shen, Yaosong Du,, Guangyan Gan, Wenya Wang, Lin William Cong, Shanghang Zhang, State Key Laboratory of Multimedia Information Processing,[-0.4em] School of Computer Science, Peking University Abstract Multimodal large language models (MLLMs) have achieved strong multimodal reasoning capabilities, but their efficiency is limited by the large number of visual tokens, which introduces substantial computational overhead. Visual token pruning offers a natural solution, yet existing methods are imperfect: attention-based criteria tend to retain redundant tokens, while diversity-based criteria are often agnostic to user instructions. Even methods that combine multiple criteria still lack a principled formulation of the intrinsic objective of token pruning. In this paper, we revisit visual token pruning from a first-principles perspective and formulate it as constructing Token Optimal Preservation Sets. Through a top-down information-theoretic analysis, we identify three fundamental principles for effective token selection: Task Relevance, Information Coverage, and Semantic Diversity. Based on these principles, we propose TOPS , a training-free and model-agnostic pruning module that can be applied to various MLLMs. Extensive experiments on 7 MLLM backbones and 14 benchmarks demonstrate that TOPS outperforms prior methods under diverse pruning settings. Notably, on LLaVA-NeXT, TOPS removes 77.8% of visual tokens while preserving 100.0% and 100.6% performance on its 7B and 13B models, respectively, suggesting that pruning redundant visual tokens can sometimes mitigate hallucination and inspire future lightweight MLLM design. † † footnotetext: * Equal contribution. \dagger Project leader. \ddagger Corresponding author. 1 Introduction Large language models (LLMs) 1 ; 22 ; 39 ; 54 ; 47 ; 48 ; 49 ; 19 ; 43 ; 44 ; 15 ; 45 ; 46 have achieved remarkable success in language understanding and reasoning. Building on these capabilities, multimodal large language models (MLLMs) 32 ; 31 ; 26 ; 64 ; 7 ; 6 ; 14 ; 66 have made rapid progress in multimodal understanding. However, their efficiency is limited by numerous visual tokens, which are processed through all transformer layers 12 ; 62 ; 60 ; 58 . Since self-attention scales quadratically with sequence length, these tokens introduce substantial computational and memory overhead, especially for multi-image and high-resolution inputs. Therefore, reducing visual tokens while preserving performance is a critical challenge. Previous methods 12 ; 56 ; 50 ; 11 have attempted to reduce visual tokens to lower the inference cost of MLLMs. Existing pruning approaches can be broadly categorized into three types. Attention-based methods 60 ; 53 ; 62 identify token importance via cross-modal attention or cls token attention, but often retain highly similar tokens, resulting in redundancy. Diversity-based methods 2 ; 51 encourage semantic dispersion, yet are typically agnostic to user instructions and may discard task-critical evidence. Other methods combine multiple criteria or incorporate coverage-based objectives 41 ; 38 ; 60 ; 3 ; 59 to model the representativeness of selected subsets. However, despite these advances, existing methods mostly treat token pruning as a scoring problem and rank tokens based on heuristic criteria, without a principled justification for why such criteria are appropriate or sufficient for constructing an optimal token subset. Fundamentally, these approaches do not start from the intrinsic objective of pruning, but rely on heuristic scoring schemes. To address these challenges, we move beyond conventional heuristic scoring schemes 37 ; 28 ; 9 ; 12 ; 38 and revisit token pruning from a first-principles perspective. Instead of designing new scoring heuristics, we rethink the core objective of token selection, conduct a top-down analysis using information theory, and identify three fundamental principles for effective pruning— Task Relevance , Information Coverage , and Semantic Diversity . Based on these principles, we formulate token pruning as an optimal subset selection problem and propose TOPS, which constructs a compact yet sufficient token subset that satisfies the proposed properties and can be applied at any pruning point during MLLM inference. We further implement TOPS as a two-stage pipeline for fine-grained token reduction, where Stage I removes coarse visual redundancy and Stage II performs text-aware refinement. As shown in Figure. (a), TOPS better preserves task-critical visual evidence under aggressive pruning. As a simple yet effective solution, TOPS does not depend on any specific visual encoder or language model, which means it can be readily implemented across any token-based MLLM. Extensive experiments across various MLLMs demonstrate the effectiveness and efficiency of TOPS, surpassing existing methods (Figure. (b)). For instance, on LLaVA-v1.5-7B 32 and LLaVA-NeXT-7B 31 , TOPS removes nearly 90% of visual tokens while retaining 97.1% and 99.1% of the original performance. Overall, our main contributions are as follows: • We revisit token pruning from a first-principles perspective, conduct a top-down analysis using information theory and identify three criteria that govern effective token selection. • We propose TOPS , a training-free and model-agnostic pruning module, considering the fundamental criteria and dynamically constructing token optimal preservation sets in MLLMs. • We conduct extensive experiments across various MLLMs and benchmarks, demonstrating TOPS consistently achieves state-of-the-art performance across different reduction ratios. 2 Related Work Figure 1: Logit fidelity of pruning methods across token budgets (128/64/32) on 200 MME samples. We report \Delta\mathcal{L}=\mathcal{L}_{\text{pruned}}-\mathcal{L}_{\text{vanilla}} , where lower values indicate smaller output distortion. TOPS consistently achieves the lowest loss increase, demonstrating stronger fidelity under aggressive pruning. Multimodal large language models. Multimodal large language models (MLLMs) 30 ; 5 ; 14 ; 26 ; 43 ; 22 ; 44 extend large language models (LLMs) 10 ; 1 ; 4 ; 55 ; 48 ; 36 ; 8 to multimodal understanding by encoding visual inputs as token sequences and processing them together with text tokens. However, visual tokenization introduces substantial computational overhead, since visual tokens are often far more numerous than text tokens and are propagated through all LLM layers. For example, LLaVA-1.5 30 represents a 336\times 336 image with 576 tokens, while LLaVA-NeXT 31 can produce up to 2,880 tokens for high-resolution inputs. The problem becomes more severe in video understanding 29 ; 25 , where long frame sequences lead to long visual token sequences and expensive inference. Therefore, effective token reduction is essential for scalable MLLM inference. Visual token reduction. Visual token reduction aims to improve MLLM efficiency by removing redundant visual tokens 23 . Existing training-free methods can be broadly categorized by their selection criteria. Attention-based methods estimate token importance from attention signals 12 ; 53 ; 62 ; 63 ; 56 , such as CLS-to-patch attention in VisionZip 56 or text-guided cross-modal attention in FastV 12 . While effective for retaining salient tokens, they often preserve redundant tokens with similar semantics. Diversity-based methods 2 ; 51 reduce redundancy by encouraging semantic dispersion, but are usually instruction-agnostic and may discard task-critical evidence. More recent methods combine importance, diversity, saliency, coverage, or progressive pruning strategies 41 ; 38 ; 60 ; 61 ; 3 ; 59 ; 42 ; 50 ; 16 ; 33 ; 58 . Despite improved performance, they still rely on heuristic scoring schemes and lack a principled formulation of the intrinsic objective of token pruning. 3 Motivation Figure 2: Overview of TOPS. Left: TOPS is a plug-and-play pruning module that can be applied at multiple