Improving Visual Token Reduction via Rectifying Distortions for Efficient Multimodal LLM Inference
Hyeonwoo Cho, Donghyeon Baek, Yewon Kim, Bumsub Ham · 2026
Read Improving Visual Token Reduction via Rectifying Distortions for Efficient Multimodal LLM Inference by Hyeonwoo Cho, Donghyeon Baek, Yewon Kim, Bumsub Ham (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01711.txt.
Introduction
Cho and co-authors fix the distortions that visual token reduction creates. Multimodal models pay a steep cost for huge visual token counts. Memory and latency suffer from that cost. Cutting tokens eases the burden, but past methods distort the sequence. Position and attention patterns shift versus the full sequence. RESTORE rectifies those two distortions while keeping the savings. A calibration step restores lost visual attention. It boosts attention weights based on distances between tokens.
A distinctive anchor choice guides token merging in RESTORE. It cuts information loss during feature averaging. Tests on many benchmarks show steady accuracy gains. RESTORE lifts each reduction method it pairs with. Top results come with the efficiency gains intact.
Catalog record
| CREATOR | Hyeonwoo Cho, Donghyeon Baek, Yewon Kim, Bumsub Ham |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.01711
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Improving Visual Token Reduction via Rectifying Distortions for Efficient Multimodal LLM Inference Hyeonwoo Cho, Donghyeon Baek, Yewon Kim, Bumsub Ham Abstract Recent advancements in Multimodal Large Language Models (MLLMs) have achieved remarkable success in vision-language tasks, yet the quadratic computational complexity arising from the vast number of visual tokens incurs significant memory and latency bottlenecks. While visual token reduction (VTR) strategies have been explored to mitigate this burden, existing methods overlook the positional and attentional consistency between the full and reduced sequences, resulting in a distorted representation. To this end, we propose RESTORE, a novel VTR framework that rectifies the positional and attentional distortions while maintaining efficiency. Specifically, we present a simple yet effective calibration method that restores lost visual attention by augmenting attention weights based on relative distances. We also introduce a distinctive anchor selection for token merging to mitigate information loss during feature averaging. Experimental results on multiple benchmarks demonstrate that our method consistently improves the accuracy of various reduction methods, achieving state-of-the-art performance while maintaining computational efficiency. Project page is available at https://cvlab.yonsei.ac.kr/projects/RESTORE 1 Introduction Recent advancements in Multimodal Large Language Models (MLLMs) ( 15 ; 14 ; 3 ; 13 ) have achieved remarkable success in interpreting complex visual information and translating it into coherent textual narratives, bridging the modality gap between vision and language. MLLMs leverage the generalization capabilities of Large Language Models (LLMs) ( 20 ; 5 ; 26 ; 2 ) with a pretrained visual encoder such as CLIP ( 19 ) . An input image is projected into a sequence of visual tokens by the visual encoder, then processed via the LLM to generate text responses based on text inputs. Given that the computational complexity of the attention mechanism scales quadratically with sequence length, the vast number of visual tokens introduces a substantial bottleneck as they often reach into the thousands for high-resolution images or video inputs. Such an extensive visual sequence incurs significant memory and latency overheads. To mitigate the computational burden, recent research has explored diverse strategies for visual token reduction (VTR) by pruning redundant tokens ( 6 ; 33 ; 32 ; 34 ) or merging similar tokens ( 4 ; 30 ; 22 ) . Token pruning maintains the original information of retained tokens but sacrifices the context from pruned ones. Conversely, although token merging preserves global context by aggregating multiple tokens, averaging features during token merging leads to information loss of fine-grained details. A hybrid approach, VisionZip ( 30 ) , attempts to combine the strengths of these approaches by first pruning and then merging the remaining tokens. Specifically, it preserves high-attention tokens and merges the rest into representatives ( i.e. , anchor tokens) chosen via uniform sampling. However, the anchor tokens might fail to represent their groups, leading to significant information loss during the merging process. Figure 1 : Illustration of the impact of visual token reduction on the internal attention mechanism of the LLM within MLLMs (e.g., LLaVA ( 14 ) ). (a) The full token sequence. (b) Reindexing position indices assigns contiguous indices to the reduced sequence. (c) Retaining position indices preserves the original indices of the retained tokens from (a). (d) We rectify distortions by retaining original position indices and calibrating the attention weights of the retained tokens. Beyond the reduction strategies, we observe that existing VTR methods fail to preserve the total attention weights of visual tokens compared to the full token sequence. This attenuation stems from the normalization property of the softmax function. As the number of visual tokens decreases, the probabilities originally allocated to the reduced tokens are redistributed to the retained tokens. Due to the exponential nature of softmax, the redistribution amplifies the remaining tokens that possessed large attention weights. The loss of visual attention weights causes the model to neglect visual contexts and rely on textual information, potentially leading to weak visual grounding or hallucinations. We show in Fig. 1 the token sequences and their attention weights when the last text token serves as the query. Figure 1(a) illustrates the full token sequence, where attention is distributed across all tokens without reduction. For the reduced token sequence, existing methods either assign new contiguous position indices ( 6 ; 29 ; 32 ) (Fig. 1 (b)) or retain the initial position indices from the full token sequence ( 30 ; 1 ; 34 ) (Fig. 1 (c)). The reindexing strategy reduces the relative distances between tokens, which helps to partially preserve the total attention weights of visual tokens. That is, squeezing the relative distances mitigates the positional bias of rotary position embedding (RoPE) ( 24 ) in the original indices of the full sequence. However, this strategy disturbs spatial relationships between tokens. For instance, the substantial distance between the first visual token and the last text token is shortened, disrupting the spatial relationships compared to the full token sequence. This problem could be addressed by retaining original position indices, but at the cost of a significant drop in visual attention. Due to the positional bias, the attention weights originally assigned to reduced visual tokens are redistributed towards text tokens that are closer to the query. In this paper, we propose RESTORE , a novel framework that RE ctifies di S tortions in visual TO ken RE duction. To address this, we introduce a calibration method that restores attention weights while preserving original position indices (Fig. 1 (d)). This attention calibration counteracts the positional bias of RoPE, recovering the lost total attention weights of visual tokens. We also introduce a novel anchor selection strategy for token merging that considers both representativeness and discriminativeness, minimizing information loss during feature averaging. Extensive experiments on several MLLM benchmarks demonstrate that our method consistently improves the accuracy of various VTR methods while maintaining efficiency. Our main contributions are summarized as follows: • We analyze the attentional and positional distortions overlooked by existing VTR methods and propose a calibration method that rectifies the distortions. • We introduce a distinctive anchor token selection strategy, mitigating information loss during token merging by selecting representative and discriminative anchor tokens. • We provide comprehensive experimental results on multiple MLLM benchmarks, demonstrating that our method significantly improves accuracy across various VTR methods while preserving efficiency. 2 Related Work Text-aware visual token reduction. Early VTR approaches for MLLMs focus on retaining visual tokens that are most relevant to the text input. The seminal work of FastV ( 6 ) introduces a text-aware token pruning method that leverages the cross-attention technique between visual and text tokens to identify and retain the most relevant visual tokens. Specifically, exploiting the autoregressive nature of LLMs, it selectively prunes visual tokens that exhibit low attention weights with respect to the last text token. FastV demonstrates that MLLMs maintain high accuracy even with a significantly reduced number of visual tokens. Subsequent works build upon this idea with advanced mechanisms to assess token importance. SparseVLM ( 33 ) introduces a pruning metric that averages cross-attention weights towards a selected subset of text tokens that are highly correlated with the visual input. FitPrune ( 31 ) identifies the importance score of visual tokens by multiplying visual self-attention weights with cross-attention weights, considering visual saliency and text relevance jointly. While