AdaFV: Rethinking of Visual-Language alignment for VLM acceleration
Jiayi Han, Liang Du, Yiwen Wu, Xiangguo Zhou, Hongwei Du, Weibo Zheng · 2025
Read AdaFV: Rethinking of Visual-Language alignment for VLM acceleration by Jiayi Han, Liang Du, Yiwen Wu, Xiangguo Zhou, Hongwei Du, Weibo Zheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-09532.txt.
Introduction
Jiayi Han and co-authors speed up vision-language models by pruning image tokens in a 2025 paper. Splitting pictures into many crops preserves detail but floods the model with redundant tokens. Earlier pruning leaned on the model's own attention, which skews toward early tokens, or on looks alone, which ignores the question asked.
Tests showed the plain text embedding aligns fairly with image tokens, free of that skew. AdaFV mixes visual salience with text-to-image similarity in early layers to keep the tokens that matter. Across tests it set the pace among training-free methods, holding up best when most tokens were cut.
Catalog record
| CREATOR | Jiayi Han, Liang Du, Yiwen Wu, Xiangguo Zhou, Hongwei Du, Weibo Zheng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-09532
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AdaFV: Rethinking of Visual-Language alignment for VLM acceleration Jiayi Han, Liang Du, Yiwen Wu, Xiangguo Zhou, Hongwei Du, Weibo Zheng Abstract The success of Vision-Language Models (VLMs) often relies on high-resolution schemes that preserve image details. However, high-resolution image input results in redundant visual tokens, significantly reducing VLM efficiency. Various research efforts have aimed to reduce visual tokens by filtering out uninformative ones or aggregating their information to enhance VLM efficiency without incurring additional training costs. Some methods propose reducing visual tokens based on VLMs’ self-attention, which can be biased and lead to inaccurate responses. Token reduction approaches that solely rely on visual cues are text-agnostic and fail to focus on areas most relevant to the query, particularly when the queried objects are non-salient in the image. In this work, we first conduct experiments to demonstrate that original text embeddings align with visual tokens without bias towards less important visual tokens. We then propose a self-adaptive cross-modality attention mixture mechanism that dynamically leverages the effectiveness of visual saliency and text-to-image similarity in the pre-LLM layers to select informative visual tokens. Extensive experiments show that the proposed approach achieves state-of-the-art training-free VLM acceleration performance, especially when the reduction rate is significantly high. 1 Introduction In recent years, vision-language models (VLMs) have demonstrated exceptional performance in various visual-grounded tasks, including image captioning, visual question answering, and layout understanding. Despite their impressive achievements, the computational cost associated with VLMs remains a significant challenge for practical deployment. A key factor contributing to this high computational cost is the large number of visual tokens. For instance, LLaVA-NEXT models Liu et al. (2024) utilize 2880 visual tokens for single-image tasks, while modern VLMs like InternVL-2.5 Chen et al. (2024b) employ over 8,000 visual tokens per task. To address this issue, numerous research efforts have focused on pruning redundant visual tokens to accelerate VLMs without additional training. For example, FastV Chen et al. (2024a) observes that the distribution of attention weights among visual tokens tends to cluster and ranks these tokens based on their attention weights, retaining only the top-ranked tokens. Similarly, SparseVLM Zhang et al. (2024b) identifies text tokens highly relevant to visual tokens, rates the significance of visual tokens within the self-attention matrix corresponding to these text tokens, and incrementally prunes irrelevant visual tokens from the hidden states. However, FasterVLM Zhang et al. (2024a) highlights a limitation in existing methods, noting that text-to-image attention weights can be biased and may not accurately reflect the importance of visual tokens. Conversely, visual saliency extracted from the visual encoder can mitigate this bias and serve as an effective metric for visual token pruning. Figure 1: The performance of different training-free VLM acceleration methods on LLaVA-NEXT-7B. AdaFV significantly increased the robustness of the model, especially for a large reduction rate. In this work, we propose leveraging the effectiveness of text-to-image (T2I) similarity and visual saliency for visual token pruning. First, we analyze the alignment of visual tokens with text embeddings in VLMs. Although FasterVLM Zhang et al. (2024a) reveals that self-attention is biased towards less important visual tokens, leading to unreliable token selection, we find that the similarity between text embeddings and visual tokens does not suffer from such misalignment. This similarity can effectively identify prompt-related visual tokens with a reduced token budget. However, certain visual cues, such as those in document images, may not be suitable for cross-modality alignment. To address these scenarios, we propose a novel self-adaptive cross-modality attention mixture (SACMAM) approach that effectively balances the weight of T2I similarity and visual saliency. To ensure co-dependence on T2I similarity and visual saliency, we measure the overall significance of the selected subset of visual tokens using an independent geometric average and employ a one-step optimization to determine the optimal collection. Extensive experiments demonstrate that the proposed method achieves state-of-the-art performance on multiple benchmark datasets and is even comparable to fine-tuning methods such as VisionZip Yang et al. (2024) . Our main contributions are as follows: 1. We analyze the patterns in multiple VLM benchmark datasets and explain why text-agnostic visual token pruning methods are effective. We also identify scenarios where these methods are sub-optimal, limiting model performance. 2. We propose a novel self-adaptive cross-modality attention mixture mechanism (SACMAM) that adaptively determines the dominance of T2I attention and visual saliency for visual token pruning, effectively combining the strengths of both metrics. 3. Extensive experiments show that our approach achieves state-of-the-art performance and is comparable to fine-tuned methods. 2 Related work 2.1 Vision language models (VLMs) Significant progress has been made in the development of vision-language models (VLMs). LLaVA Yifan et al. (2023) was the first approach to effectively combine large language models (LLMs) with foundational vision models. Following LLaVA, a series of VLMs have been proposed, enhancing multi-modal capabilities Lin et al. (2023) ; Bai et al. (2023) ; Chen et al. (2024c) . The initial models in the LLaVA family only utilized a single image as input, resulting in 576 visual tokens for a 336×336 image. However, this approach often led to significant information loss, thereby degrading model performance. To address this issue, many VLMs have introduced dynamic high-resolution strategies. Despite their success, these high-resolution strategies have significantly increased the number of visual tokens. For instance, LLaVA-NEXT models Liu et al. (2024) involve up to 2880 visual tokens for the same task. 2.2 VLM acceleration with token pruning Token pruning is a straightforward solution for accelerating transformer models. By merging information into a smaller number of tokens, redundant tokens can be pruned to safely accelerate transformer models, with minimal impact on performance while significantly reducing computational costs Kim et al. (2022) ; Fu et al. (2024b) ; Wang et al. (2024) . Some approaches have adapted this concept to accelerate VLMs. For example, Chen et al. (2024a) ; Ye et al. (2024) propose measuring the significance of visual tokens based on self-attention extracted from within LLMs. To further accelerate models, some approaches perform token pruning before the encoder layers of LLMs. FasterVLM, for instance, suggests that text-to-image attentions in LLM layers are biased, leading to inaccuracies, and instead proposes using the visual saliency of visual tokens as a metric to prune non-salient tokens. To enhance pruning effectiveness, some approaches also fine-tune VLMs. For example, VisionZip Yang et al. (2024) fine-tunes the MLP projector to improve alignment for text-to-image attention. 3 Evaluating alignment and similarity in vision-language tasks 3.1 Are VQA tasks aligned with visual salient information? Recent studies indicate that preserving visual tokens that significantly contribute to the [CLS] token enhances performance in visual question-answering (VQA) tasks Zhang et al. (2024a) ; Yang et al. (2024) . This suggests that current VQA tasks are closely related to salient visual information. To investigate this hypothesis, we conducted experiments on five benchmark datasets: MME, MM-Vet, TextVQA, POPE, and GQA. We employed the SAM-2 model Ravi et al. (2024) to segment objects based on text prompts and used the CLIP vision encoder Radford et al. (2021) to generate visual saliency for each image token. The area under the curve (AUC) was calculated to assess the consistency between visual saliency and text-grounded segmentation. Detailed experimental procedures are provided in the Appendix. As illustrated in Fig. 2 (a), the average AUC across datasets