VCM: Vision Concept Modeling with Adaptive Vision Token Compression via Instruction Fine-Tuning
Run Luo Renke Shan Longze Chen Ziqiang Liu Lu Wang, Min Yang Xiaobo Xia · 2025
Read VCM: Vision Concept Modeling with Adaptive Vision Token Compression via Instruction Fine-Tuning by Run Luo Renke Shan Longze Chen Ziqiang Liu Lu Wang, Min Yang Xiaobo Xia (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-19627.txt.
Introduction
Big vision-language models read every part of an image as tokens, which wastes effort on empty detail. Run Luo and co-authors propose a concept-level fix in a 2025 paper. People, by contrast, pull out the key concepts in a scene with little effort. Their framework, VCM, learns visual concepts without costly hand-made concept labels. It pairs a contrast-based method with fine-tuning on vision-language instructions.
In tests, the framework cut computing cost sharply for a known model while keeping scores strong. It also lifted the base vision encoders on classic concept tasks. Both number checks and look-based checks backed the claims. The authors argue concept modeling is a missing piece in current models. The work points toward cheaper, faster image understanding.
Catalog record
| CREATOR | Run Luo Renke Shan Longze Chen Ziqiang Liu Lu Wang, Min Yang Xiaobo Xia |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-19627
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VCM: Vision Concept Modeling with Adaptive Vision Token Compression via Instruction Fine-Tuning Run Luo Renke Shan Longze Chen Ziqiang Liu Lu Wang, Min Yang Xiaobo Xia Abstract Large vision-language models (LVLMs) have emerged as foundational tools for real-world AI applications. Despite their remarkable capabilities, current LVLMs process entire images at the token level, leading to significant inefficiencies compared to human cognition, which selectively focuses on high-level vision concepts. This token-level redundancy becomes increasingly problematic for high-resolution images and long video sequences, resulting in large computational costs and limited scalability in practical applications. To address this limitation, we introduce the concept of a vision concept model, a novel paradigm that enables LVLMs to dynamically extract the most relevant vision concepts from complex inputs, based on task-specific instructions. To optimize this vision concept modeling process, we propose VCM, a self-supervised framework that leverages vision-language correlations across diverse instances. VCM is designed to learn meaningful vision concepts without the need for expensive concept-level annotations. At its core, it employs a forward-backward optimization algorithm that supports LVLMs to adjust concept granularity and spatial alignment dynamically. Experiments demonstrate that VCM remarkably reduces computational costs ( e.g. , achieving up to 85% fewer FLOPs for LLaVA-1.5-7B), while maintaining strong performance across a series of vision-language tasks. The source code will be publicly available. 1 Introduction Large vision-language models (LVLMs) [ 1 , 2 , 3 , 4 , 5 , 6 , 7 ] play a critical role in addressing a wide range of vision-language tasks and have become a cornerstone for enabling general artificial intelligence to interact with the real world, such as in embodied intelligence [ 8 , 9 ] and autonomous driving [ 10 , 11 ] . However, current LVLMs process entire images at the token level, which is inefficient compared to humans, who analyze information and generate content at the conceptual level, extracting relevant vision concepts with minimal effort. This inefficiency becomes particularly pronounced when dealing with higher-resolution images or longer video inputs, which humans can easily handle, but LVLMs struggle with it due to the rapidly increasing computational cost. This limitation, arising from the lack of a vision concept model, restricts the usability of LVLMs in practical applications. In this work, we formally define the concept of a vision concept model . The vision concept model can dynamically determine the required vision concepts, including their quantity and spatial locations, based on given instructions. Note that previous methods [ 12 , 13 , 14 ] have attempted to improve the efficiency of LVLMs by compressing vision tokens using variable-length queries or pruning the tokens through attention mechanisms. However, these methods fail to provide semantically meaningful concepts grounded in the corresponding spatial locations of vision inputs, making them unsuitable for vision-language tasks beyond visual question answering 1 1 1 We review related works and detail their difference from our method in Appendix 5 . . As such, they cannot be categorized as vision concept models. Figure 1: Illustrations of VCM in enhancing efficiency and dense perception capability. VCM can select relevant vision concept tokens based on instructions, significantly reducing redundant attention computations in LVLMs. This reduction lowers both training and inference costs while maintaining strong performance, as shown in (a). Additionally, VCM enhances the dense concept prediction capability of the vision encoder, as illustrated in (b) through K-Means visualizations of dense feature maps from the last layer of CLIP ViT. These improvements enable broader applicability to dense perception vision-language tasks. To bridge this gap, we propose VCM, a self-supervised vision concept modeling framework. Specifically, VCM leverages vision-language correlations across multiple sampled instances and employs vision-language fine-tuning to build a vision concept model, without costly concept-level annotations. To enable the optimization with theoretical support for the vision tokens of varying lengths, we design a forward-backward algorithm based on the dynamic programming process [ 15 ] , which supports adaptive and dynamic length optimization. This allows the vision concept model to dynamically output required vision concepts and keep their corresponding spatial locations according to given instructions, as shown in Figure 1 . Benefiting from the vision concept model, VCM not only significantly reduces the computational cost of training and inference in LVLMs ( e.g. , 85% fewer FLOPs for LLaVA-1.5-7B) while maintaining strong visual question answering performance but also enhances the vision encoder’s capabilities in other dense perception vision-language tasks, such as zero-shot image classification, open-vocabulary object detection, and open-vocabulary semantic segmentation. Before delving into details, our main contributions can be summarized as follows. (1) We formally define the vision concept model, which dynamically determines the required vision concepts, including their quantity and spatial locations, based on given instructions. This model is applicable to a variety of vision-language tasks. (2) We propose the VCM framework, which employs the correlations between vision and language across multiple sampled instances and a theoretically supported forward-backward algorithm to enable dynamic vision concept learning without requiring expensive fine-grained annotations. (3) We conduct extensive qualitative analyses, quantitative experiments, and ablation studies to validate the effectiveness and efficiency of VCM. 2 Preliminaries Problem setting. The LVLM architecture generally consists of a vision encoder f_{\text{V}} , a modality projector f_{\text{M}} , and a large language model (LLM) f_{\text{LLM}} with its language model head (LMH) f_{\text{LMH}} . The vision encoder, typically a pre-trained image encoder like CLIP’s vision model [ 16 ] , converts input images \mathbf{X}^{\text{V}} into vision tokens f_{\text{V}}(\mathbf{X}^{\text{V}}) . The projector aligns these vision tokens with text instruction \mathbf{X}^{\text{I}} encoded by the LLM’s word embedding \mathbf{X}^{\text{T}} , which enables the LLM to process vision data effectively. For simplicity, we define that \mathbf{H}^{\text{V}}=f_{\text{M}}(f_{\text{V}}(\mathbf{X}^{\text{V}}))\in\mathbbm{R}^{M\times d} and \mathbf{H}^{\text{I}}=\mathbf{X}^{\text{I}}(\mathbf{X}^{\text{T}})^{\top}\in\mathbbm{R}^{N\times d} , where M denotes the length of vision tokens, N denotes length of text tokens , and d is the hidden dimension size. That is usually M\gg N . The LLM then integrates the aligned vision information and instructions to produce response \mathbf{X}^{\text{R}} , which is learned as p_{\theta}(\mathbf{X}^{R}|f_{\text{LMH}}(f_{\text{LLM}}([\mathbf{H}^{\text{V}};\mathbf{H}^{\text{I}}]))) , parameterized by \theta . Note that due to a large number of vision tokens, the training and inference of LVLMs are inefficient [ 17 ] . Therefore, if a vision concept model exists that can decrease the value of M , the efficiency of LVLMs during both training and inference will be improved 2 2 2 We provide more analysis of the efficiency improvement from the floating-point operation (FLOP) perspective. Interested readers can check Appendix D for details. . Goal and challenges. The primary goal of vision concept modeling (VCM) is to learn relevant vision conceptual information \mathbf{H}^{\text{V}}_{\text{C}} from \mathbf{H}^{\text{V}} 3 3 3 Here \mathbf{H}^{\text{V}}_{\text{C}} is a sparse version of \mathbf{H}^{\text{V}} , with \|\mathbf{H}^{\text{V}}_{\text{C}}\|_{0}\ll\|\mathbf{H}^{\text{V}}\|_{0} . , based on language priors. The vision conceptual information can be applied to various vision-language tasks, leading to significant improvements in performance or efficiency. There are two key challenges in VCM. (1) Fine-grained labeled data is scarce, as manual annotation is labor-intensive, time-consuming, and inefficient. This makes it difficult to leverage the abundant unlabeled visual question answering (VQA) datasets effectively. (2) The length of vision concepts varies dynamically across different examples, making it challenging to model and optimize. Below we present how to handle the two challenges and implement VCM. 3 Vision Concept Modeling (a) (b) (c) Figure 2: