D²Pruner: Debiased Importance and Structural Diversity for MLLM Token Pruning
Evelyn Zhang, Fufu Yu, Aoqi Wu, Zichen Wen, Ke Yan, Shouhong Ding, Biqing Qi, Linfeng Zhang · 2025
Read D²Pruner: Debiased Importance and Structural Diversity for MLLM Token Pruning by Evelyn Zhang, Fufu Yu, Aoqi Wu, Zichen Wen, Ke Yan, Shouhong Ding, Biqing Qi, Linfeng Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-19443.txt.
Introduction
Evelyn Zhang, Fufu Yu, Aoqi Wu, and Zichen Wen prune visual tokens for multimodal language models without wrecking fine detail. Long image token streams make these models slow. Older pruning either trusts importance scores warped by position bias, or enforces diversity while ignoring the user's prompt and the image layout.
D2Pruner first locks in a core of pivot tokens by a debiased attention score. The rest are placed on a graph linking near and similar tokens, and a maximal independent set picks the survivors. Importance and structure are both honored. The authors report that localization tasks, where earlier methods failed badly, hold up far better.
Catalog record
| CREATOR | Evelyn Zhang, Fufu Yu, Aoqi Wu, Zichen Wen, Ke Yan, Shouhong Ding, Biqing Qi, Linfeng Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-19443
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
D²Pruner: Debiased Importance and Structural Diversity for MLLM Token Pruning Evelyn Zhang, Fufu Yu, Aoqi Wu, Zichen Wen, Ke Yan, Shouhong Ding, Biqing Qi, Linfeng Zhang Abstract Processing long visual token sequences poses a significant computational burden on Multimodal Large Language Models (MLLMs). While token pruning offers a path to acceleration, we find that current methods, while adequate for general understanding, catastrophically fail on fine-grained localization tasks. We attribute this failure to the inherent flaws of the two prevailing strategies: importance-based methods suffer from a strong positional bias, an inherent model artifact that distracts from semantic content, while diversity-based methods exhibit structural blindness, disregarding the user’s prompt and spatial redundancy. To address this, we introduce D²Pruner, a framework that rectifies these issues by uniquely combining debiased importance with a structural pruning mechanism. Our method first secures a core set of the most critical tokens as pivots based on a debiased attention score. It then performs a Maximal Independent Set (MIS) selection on the remaining tokens, which are modeled on a hybrid graph where edges signify spatial proximity and semantic similarity. This process iteratively preserves the most important and available token while removing its neighbors, ensuring that the supplementary tokens are chosen to maximize importance and diversity. Extensive experiments demonstrate that D²Pruner has exceptional efficiency and fidelity. Applied to LLaVA-1.5-7B for general understanding tasks, it reduces FLOPs by 74.2% while retaining 99.2% of its original performance. Furthermore, in challenging localization benchmarks with InternVL-2.5-8B, it maintains 85.7% performance at a 90% token reduction rate, marking a significant advancement with up to 63. 53% improvement over existing methods. Code — https://github.com/EvelynZhang-epiclab/D2Pruner Introduction The remarkable capabilities of Multimodal Large Language Models (MLLMs) in visual understanding and localization are fundamentally constrained by their computational demands. The quadratic complexity of self-attention and the massive memory footprint of the Key-Value (KV) cache create a severe bottleneck, particularly when processing high-resolution images or long videos. This bottleneck critically hinders their efficient inference and real-world deployment, making inference acceleration an urgent priority. Figure 1: Illustration of positional bias in MLLMs and our debiasing solution. Left : Averaging attention maps over 1k images reveals a strong model bias prior (e.g., towards the bottom for LLaVA). Right : This causes the naive attention-based method (e.g., FastV) to fail by pruning salient objects like the umbrella. Our relative attention, which normalizes by this model bias prior, rectifies the pruning decision and answer correctly. Figure 2: Qualitative and Quantitative Analysis of D²Pruner against Prior Methods. Left: Visual comparison on localization tasks. Importance-based method, FastV , is misled by positional bias, concentrating tokens on the image bottom, while Diversity-based method, DART , suffers from structural blindness and semantic myopia. In contrast, our Debiased Importance (DI) component successfully overcomes positional bias. Our full model, D²Pruner (DI+SD) , jointly addresses both semantic and spatial redundancy, resulting in a more balanced and informative token selection that enhances localization performance. Right: Quantitative evaluation of average performance on general understanding and fine-grained localization tasks. D²Pruner achieves the best performance across all reduction ratios, with notable improvements on localization tasks. To mitigate this, token pruning has emerged as a promising strategy for reducing the input sequence length. While existing methods show moderate success on general understanding and reasoning tasks, they catastrophically fail on fine-grained tasks that demand precise spatial awareness and localization. Our investigation pinpoints two critical, previously overlooked failure modes in these approaches: Position Bias in Importance-based Methods. These methods ( 8 ) estimate the importance of visual tokens by leveraging the attention scores assigned to them by the last token. However, such scores are inherently vulnerable to position bias, which can distort the true importance distribution. Prior studies ( 32 ; 38 ) have reported that LLaVA consistently exhibits a content-agnostic emphasis on the bottom region of the image. Building on this observation, we conduct a broader analysis across several representative MLLMs and reveal that position bias is not limited to LLaVA but manifests in diverse forms. Specifically, we average the attention maps of over 1,000 randomly sampled images from the COCO dataset ( 18 ) to uncover these patterns. As highlighted in Fig. 1 , our findings not only corroborate LLaVA’s bottom-focused tendency but also expose distinct biases in other models: InternVL ( 10 ) and QwenVL ( 5 ) , for instance, exhibit a pronounced preference for tokens near the image periphery. We posit these biases stem from the models’ positional embeddings. LLaVA’s absolute embeddings foster a location-based shortcut (bottom-focus), while the relative embeddings (e.g., RoPE ( 31 ) ) in InternVL and QwenVL encourage a structural one (periphery-focus). This persistent position bias fundamentally undermines the reliability of raw attention as a proxy for semantic importance. Structural Blindness in Diversity-based Methods. Previous methods ( 33 ; 2 ) prune tokens by assessing diversity purely in the feature space, typically via cosine similarity. This strategy operates on a flawed premise: that visual information can be processed as a one-dimensional sequence, after flattening the original 2D grid. This action strips away all topological information, forcing the model to treat tokens as an unstructured set and rendering it blind to their crucial spatial relationships. This motivates our approach to adopt a representation that explicitly preserves the 2D structure of images by modeling tokens and their semantic–spatial relationships as a graph. To address these challenges, we introduce D²Pruner , a pruning framework built on the principles of D ebiased Importance (DI) and Structural D iversity (SD). It first secures a set of the most critical tokens as pivots based on the debiased attention score. It then performs a greedy Maximal Independent Set (MIS) selection on the remaining tokens, which are modeled on a hybrid graph where edges signify both spatial proximity and semantic similarity. This process iteratively preserves the most important available token while pruning its neighbors, ensuring the supplementary tokens are chosen to maximize importance and structural diversity. Leveraging its dual-stage design, D²Pruner not only excels on general understanding tasks but demonstrates unparalleled strength in fine-grained localization, as illustrated in Fig. 2 . On challenging localization benchmarks ( 15 ; 25 ) , our method maintains 85.7% performance on InternVL-2.5-8B ( 10 ) even at a 90% pruning rate—surpassing prior diversity-based and importance-based methods by a massive 63.5 and 51.2 points, respectively. This is complemented by strong general performance, retaining 95.5% of LLaVA-Next-7B’s capabilities. Critically, these results are achieved with remarkable efficiency: D²Pruner delivers a 5× prefill acceleration and a 7.6× reduction in KV cache size, while the pruning process itself adds a negligible overhead of just 0.8% to the forward pass. In summary, our contributions are as follows. • We analyze the pervasive position bias in various MLLMs and reveal distinct bias patterns across models. To address this, we propose a debiasing method that effectively mitigates bias in token importance estimation. • We present D²Pruner, a token pruning framework that jointly considers both spatial and semantic redundancy by modeling token relationships on a hybrid graph, enabling more informed and effective pruning decisions. • Extensive experiments demonstrate that D²Pruner achieves superior performance, especially on fine-grained localization tasks, while delivering significant acceleration and memory savings. Figure 3: Overall pipeline for our proposed D²Pruner. (a) Debiased Importance Score: The original attention