ShortV: Efficient Multimodal Large Language Models by Freezing Visual Tokens in Ineffective Layers

Qianhao Yuan, Qingyu Zhang, Yanjiang Liu, Jiawei Chen, Yaojie Lu, Hongyu Lin, Jia Zheng, Xianpei Han, Le Sun · 2025

Read ShortV: Efficient Multimodal Large Language Models by Freezing Visual Tokens in Ineffective Layers by Qianhao Yuan, Qingyu Zhang, Yanjiang Liu, Jiawei Chen, Yaojie Lu, Hongyu Lin, Jia Zheng, Xianpei Han, Le Sun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-00502.txt.

Introduction

Nine authors present ShortV in 2025, a no-train trick that speeds large mixed models. Such models burn cost on size and on swarms of sight tokens in the stack. A new gauge, Layer Contribution, scores how much each layer moves sight and text tokens.

Tests show many layers add near nil to sight token work in the stack. ShortV locks sight token steps in those weak layers and skips the cost. On one big model, flop cost fell by half with top marks kept. Near 60 percent of layers can lock.

Catalog record

CREATORQianhao Yuan, Qingyu Zhang, Yanjiang Liu, Jiawei Chen, Yaojie Lu, Hongyu Lin, Jia Zheng, Xianpei Han, Le Sun
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-00502

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ShortV: Efficient Multimodal Large Language Models by Freezing Visual Tokens in Ineffective Layers Qianhao Yuan, Qingyu Zhang, Yanjiang Liu, Jiawei Chen, Yaojie Lu, Hongyu Lin, Jia Zheng, Xianpei Han, Le Sun Abstract Multimodal Large Language Models (MLLMs) suffer from high computational costs due to their massive size and the large number of visual tokens. In this paper, we investigate layer-wise redundancy in MLLMs by introducing a novel metric, Layer Contribution (LC), which quantifies the impact of a layer’s transformations on visual and text tokens, respectively. The calculation of LC involves measuring the divergence in model output that results from removing the layer’s transformations on the specified tokens. Our pilot experiment reveals that many layers of MLLMs exhibit minimal contribution during the processing of visual tokens. Motivated by this observation, we propose ShortV, a training-free method that leverages LC to identify ineffective layers, and freezes visual token updates in these layers. Experiments show that ShortV can freeze visual token in approximately 60% of the MLLM layers, thereby dramatically reducing computational costs related to updating visual tokens. For example, it achieves a 50% reduction in FLOPs on LLaVA-NeXT-13B while maintaining superior performance. The code is publicly available at https://github.com/icip-cas/ShortV . 1 Introduction Large language models (LLMs) have achieved remarkable performance in natural language tasks [ 49 , 12 , 1 , 37 , 47 ] . Building upon LLMs, Multimodal Large Language Models (MLLMs) [ 31 , 32 , 38 , 39 ] take a significant step towards understanding the real physical world by incorporating visual information into their processes. Typically, an MLLM consists of a visual encoder, a projector, and an LLM backbone. Most of them preprocess visual information through a visual encoder, e.g . a CLIP-ViT [ 41 , 14 ] , and project the patch-level visual features into visual tokens through a projector. Then they concatenate visual and text tokens and feed them into the LLM backbone. (a) ShortV (b) Performance vs . the Number of ShortV Layers Figure 1 : (a) Illustration of ShortV. We identify ineffective layers for visual tokens and replace these layers with sparse ShortV layers. In ShortV layers, we freeze visual tokens, and eliminate computations related to updating them. ShortV improves MLLM efficiency in a training-free manner and involves no parameter updates. Notably, ShortV is compatible with token pruning methods, e.g . FastV. (b) Performance vs . the number of ShortV layers. Average Performance means a normalized average score on multiple benchmarks. ShortV can freeze visual tokens in approximately 60% of the MLLM layers with nearly no performance degradation. However, MLLMs face a substantial increase in computational overhead. This burden primarily stems from the large scale of the LLM backbones and the significantly extended length of the concatenated visual-text token sequences. To address this issue, Chen et al . [ 10 ] discovers significant token-wise redundancy in MLLMs. Based on this redundancy, they propose FastV, which identifies and prunes unimportant visual tokens in MLLMs to improve their efficiency. Other than token-wise redundancy, in this paper, we reveal that MLLMs also exhibit significant layer-wise redundancy in processing visual tokens. Specifically, we propose Layer Contribution (LC), a metric that quantifies how much a layer’s transformations on certain tokens contribute to the model’s output. In the LC calculation of a layer, we freeze certain tokens in this layer, i.e . keep the hidden states of the tokens unchanged, and then compute the Kullback-Leibler (KL) divergence between the resulting model’s output logits and those of the original model. This metric provides a direct measure of a layer’s importance for certain tokens. By comparing LC scores on visual tokens and those on text tokens, we discover that MLLM layers are ineffective for visual tokens, and their transformations on visual tokens contribute minimal to the model’s output. This phenomenon inspires us to propose ShortV, a simple but effective method to improve the efficiency of MLLMs. In ShortV, we first utilize the LC metric to identify layers least effective at transforming visual tokens, and then replace these layers with sparse ShortV layers. Within these sparse layers, visual tokens remain frozen, and the corresponding computations for updating them are eliminated, as shown in Figure 1(a) . To validate the effectiveness of ShortV, we conduct evaluations across multiple benchmarks, including MME [ 15 ] , MMBench [ 33 ] , MMMU [ 55 ] , MMStar [ 9 ] , SEED-Bench [ 25 ] , GQA [ 19 ] , and Flickr30K [ 40 ] . Figure 1(b) illustrates the correlation between the normalized average performance on these benchmarks and the number of replaced layers. As observed, ShortV can replace approximately 60% of MLLM layers without performance degradation. Unlike FastV and other token pruning methods, ShortV reduces computations of per visual token rather than reducing the number of visual tokens. Therefore, ShortV and token pruning methods are orthogonal and compatible. Furthermore, we demonstrate that combining ShortV and FastV can further enhance MLLM efficiency. We summarize our contribution as follows. • We propose Layer Contribution (LC), a metric to quantify how much a layer’s transformations on specific tokens contribute to the model’s output. • Leveraging LC, we reveal significant redundancy in MLLM layers for visual tokens. Transformations on visual tokens in many layers contribute minimally and are thus ineffective. • Based on the observation above, we propose ShortV, which improves MLLM efficiency by freezeing visual tokens in ineffective layers. ShortV can freeze visual tokens in approximately 60% of MLLM layers without performance degradation. Extensive experiments and ablation studies demonstrate ShortV’s effectiveness. 2 Layer Redundancy in MLLMs In this section, we first introduce the background of layer redundancy in text-only LLMs, and then propose a metric to measure MLLM layer redundancy for different types of tokens. Next, we conduct a pilot experiment to investigate layer redundancy in MLLMs. 2.1 Background Typically, MLLMs are built on text-only LLMs. MLLMs employ pre-trained visual encoders, such as CLIP-ViT [ 41 , 14 ] , to convert images into visual features, and then use projectors to project them into visual tokens in the text token embedding space. The visual tokens are concatenated with text tokens and fed into the LLM backbones. For text-only LLMs, Men et al . [ 36 ] identify notable redundancy across their layer. Some layers’ transformations on the hidden states of text token contribute minimally to the overall model functionality. Consequently, these layers are considered ineffective. Removing these transformations in approximately 25% of LLM layers has minimal impact on model outputs. Such redundancy mainly occurs in middle-to-deeper layers, whereas initial layers and the last layer remain critical to the model functionality. However, this pattern may not hold for MLLMs. Huang et al . [ 18 ] demonstrate a clear modality gap in the embedding space of current MLLMs, where visual and text tokens exhibit a uniform distribution within each modality but a significant distribution gap between modalities. Such a modality gap implies that MLLMs might adopt distinct computational patterns or strategies for processing visual and text tokens, potentially affecting how redundancy is distributed across layers. This raises several key questions: Are MLLM layers as ineffective for visual tokens as LLM layers are for text tokens? To what extent does layer redundancy exist in MLLMs? How

Read the complete text