Why and When Visual Token Pruning Fails? A Study on Relevant Visual Information Shift in MLLMs Decoding

Jiwan Kim 1, Kibum Kim 1, Wonjoong Kim 1, Byung-Kwan Lee 2, Chanyoung Park 1 · 2026

Read Why and When Visual Token Pruning Fails? A Study on Relevant Visual Information Shift in MLLMs Decoding by Jiwan Kim 1, Kibum Kim 1, Wonjoong Kim 1, Byung-Kwan Lee 2, Chanyoung Park 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-12358.txt.

Introduction

Jiwan Kim and four colleagues study why cutting visual tokens harms multimodal models on hard reasoning. They trace the failures to a shift in what visual detail matters as answers form. Their training-free add-on, DSTP, keeps pruning matched to those changing needs during decoding. Tests show it recovers much lost accuracy on reasoning tasks and helps simpler image tasks too. The gains come at little extra computing cost.

Catalog record

CREATORJiwan Kim 1, Kibum Kim 1, Wonjoong Kim 1, Byung-Kwan Lee 2, Chanyoung Park 1
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73014
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.12358

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Why and When Visual Token Pruning Fails? A Study on Relevant Visual Information Shift in MLLMs Decoding Jiwan Kim 1, Kibum Kim 1, Wonjoong Kim 1, Byung-Kwan Lee 2, Chanyoung Park 1 Abstract Recently, visual token pruning has been studied to handle the vast number of visual tokens in Multimodal Large Language Models. However, we observe that while existing pruning methods perform reliably on simple visual understanding, they struggle to effectively generalize to complex visual reasoning tasks, a critical gap underexplored in previous studies. Through a systematic analysis, we identify Relevant Visual Information Shift (RVIS) during decoding as the primary failure driver. To address this, we propose D ecoding-stage S hift-aware T oken P runing ( DSTP ), a training-free add-on framework that enables existing pruning methods to align visual tokens with shifting reasoning requirements during the decoding stage. Extensive experiments demonstrate that DSTP significantly mitigates performance degradation of pruning methods in complex reasoning tasks, while consistently yielding performance gains even across visual understanding benchmarks. Furthermore, DSTP demonstrates effectiveness across diverse state-of-the-art architectures, highlighting its generalizability and efficiency with minimal computational overhead. 1 Introduction Multimodal Large Language Models (MLLMs) [ 29 , 3 , 11 ] have demonstrated strong capabilities in both visual understanding (e.g., visual question answering (VQA)) and visual reasoning , including visual-centric math, logical puzzles, and STEM-related tasks. 1 1 1 STEM stands for Science, Technology, Engineering and Mathematics. However, these models typically rely on a massive number of visual tokens generated by a vision encoder [ 35 , 51 ] . Such a large number of visual tokens incur prohibitive computational and memory overhead during LLM processing, posing considerable challenges for the practical and efficient deployment of MLLMs. To mitigate this, a surge of studies [ 9 , 53 , 44 , 48 ] has focused on visual token pruning, aiming to reduce computational overhead by removing redundant visual tokens. The core objective of these studies lies in detecting visual tokens that contain Relevant Visual Information , which is defined as the essential visual cues required to resolve a given task [ 53 ] . Existing pruning methods typically determine the most relevant visual tokens by leveraging internal MLLM attention [ 9 , 44 ] , vision encoder outputs [ 1 , 45 ] , or a hybrid of both [ 53 , 48 ] . Consequently, by retaining a small number of relevant tokens, these methods significantly accelerate inference while maintaining reliable performance on general visual tasks. Figure 1 : (a) Performance retention rates of various token pruning methods on Qwen3-VL [ 4 ] and InternVL3.5 [ 42 ] across VQA and VMR. (b)–(f) Attention heatmaps for a MathVerse [ 52 ] sample on Qwen3-VL relative to the text token whose attention shifts drastically, reflecting the reasoning context. 3 3 3 Regarding how the specific text token for the attention heatmap is selected, please refer to Section 3 . Full sentences are provided to offer complete context for each reasoning step. However, most existing pruning methods [ 9 , 1 , 53 ] are primarily evaluated on straightforward visual understanding such as recognizing objects and their attributes using simple VQA benchmarks [ 15 , 37 , 30 ] . Under this paradigm, existing methods have implicitly relied on the expectation that relevant visual information identified during the prefill stage—the initial phase where the model processes the input sequence—would remain sufficient to resolve the given task. This assumption holds for simple visual understanding where necessary visual cues are concentrated in narrow regions. However, as the MLLMs evolve toward reasoning-centric models [ 39 , 40 , 4 ] , there arises a growing need to address complex visual reasoning . Yet, the efficacy of pruning methods remains largely underexplored in these sophisticated step-by-step reasoning tasks, necessitating the exploration of broader visual regions. 4 4 4 A detailed comparison between visual understanding and visual reasoning is in Appendix 0.A . To explore this gap, we adopt visual-centric math as a representative task for complex visual reasoning and evaluate prominent pruning methods (i.e., FastV [ 9 ] and DivPrune [ 1 ] ) with state-of-the-arts MLLMs [ 4 , 42 ] on visual math reasoning (VMR) benchmarks. Furthermore, we also evaluate these methods on simple VQA for comparison. 5 5 5 VQA include SQA [ 30 ] , GQA [ 15 ] , and \text{VQA}^{\text{T}} [ 37 ] , and VMR include MathVerse [ 52 ] , WeMath [ 34 ] , and DynaMath [ 55 ] . Footnote 2 (a) shows the average performance normalized to vanilla model as 1.0. As shown in Footnote 2 (a), we observe that while pruning methods effectively preserve performance on VQA, they exhibit precipitous performance drops on VMR. This consistent decline on VMR suggests that existing pruning methods fundamentally struggle to generalize to complex visual reasoning. To uncover why pruning methods that succeed in VQA fail to generalize to VMR, we compare the visual attention map from the prefill stage ( Footnote 2 (b)) with those from key decoding steps ( Footnote 2 (c)–(f)), visualizing how the model’s focus transitions across the image throughout the decoding process. 6 6 6 The corresponding attention visualizations for VQA benchmarks are provided in the Appendix 0.B . Our visualization reveals a dynamic shift as reasoning progresses, demonstrating that the model’s visual focus does not remain fixed on the regions identified during the prefill stage; rather, it frequently transitions to entirely different visual areas to align with shifting reasoning requirements. We term this dynamic shift of visual focus during decoding stage the Relevant Visual Information Shift (RVIS) , indicating that the visual evidence required to resolve a task changes over time—a behavior fundamentally distinct from the static visual focus observed in VQA tasks. 6 In this regard, we hypothesize that the failure of existing pruning methods in VMR stems from the occurrence of RVIS during the decoding stage, which their prefill-stage pruning strategies inherently struggle to address. To validate this hypothesis, we conduct a detailed examination of the attention patterns of MLLMs in Section 3 to uncover the root cause of the observed performance degradation. Our diagnostic analysis yields two primary insights: 1) Visual reasoning exhibits drastic and frequent fluctuations in Relevant Visual Information throughout the decoding process, whereas such shifts are not observed in visual understanding. 2) This shift, which we term RVIS, serves as the primary driver of pruning failure, with performance declining sharply as its frequency increases during decoding. Consequently, these observations underscore the necessity for an adaptive pruning approach during the decoding stage to effectively address RVIS. To this end, we introduce D ecoding-stage S hift-aware T oken P runing ( DSTP ), a training-free and simple add-on to existing pruning methods. It adaptively restores previously discarded tokens by detecting RVIS in the model’s visual focus during decoding, thereby aligning the visual tokens with the reasoning step. For this, DSTP preserves the tokens discarded by existing pruning methods for potential retrieval during decoding. This mechanism is driven by two integrated modules: the Relevant Visual Information Shift Detect (RISD) , which monitors visual attention during decoding to identify RVIS through a threshold-based mechanism; and the Context-Preserving Visual Token Swap (CPTS) , which swaps

Read the complete text