Dynamic Rank Adaptation for Vision-Language Models DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2025; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Computer vision
Jiahui Wang, Qin Xu, Bo Jiang, Bin Luo · 2025
Read Dynamic Rank Adaptation for Vision-Language Models DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2025; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Computer vision by Jiahui Wang, Qin Xu, Bo Jiang, Bin Luo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-05668.txt.
Introduction
Jiahui Wang and colleagues present Dynamic Rank Adaptation for vision-language models. Their 2025 arXiv paper improves fine-tuning for new class recognition. Prompt and adapter methods made progress on tuning these models. Keeping strong generalization to unseen classes remains hard, though. Treating all tokens equally can cause overfitting to noise. Background and template words then crowd out useful signal.
The method groups tokens by importance using sequence attention. Each group gets an adaptation rank matched to its value. Important tokens earn higher rank and richer updates. Less useful tokens stay closer to the base model. General knowledge is thus preserved during tuning. Tests target recognition of classes never seen in training. The design aims at flexible models that keep learning.
Catalog record
| CREATOR | Jiahui Wang, Qin Xu, Bo Jiang, Bin Luo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-05668
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Dynamic Rank Adaptation for Vision-Language Models DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2025; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Computer vision Jiahui Wang, Qin Xu, Bo Jiang, Bin Luo Abstract . Pre-trained large vision-language models (VLMs) like CLIP demonstrate impressive generalization ability. Existing prompt-based and adapter-based works have made significant progress in fine-tuning VLMs but still face the challenges of maintaining strong generalization abilities, particularly towards unseen new classes. This limitation partly arises from these methods treating all tokens of the image and text encoder equally, which can lead to overfitting on less informative features (e.g., background noise, template words) and degrade the general representations that are crucial for novel concept recognition. To address this issue, we propose Dynamic Rank Adaptation (DRA), a novel adapter variant method, designed specifically to enhance new class generalization. DRA dynamically allocates adaptation ranks based on the importance of features during training to preserve general knowledge. DRA first employs token importance grouping, using sequence attention to evaluate and group tokens by their importance. Then, we adopt rank adaptation according to the importance of each token group dynamically by assigning higher feature ranks to the more important tokens. Also, we design a new channel response mechanism to prioritize the preservation and adaptation of feature channels identified as the most informative for each instance. In addition, a L1 regularization term is introduced to stabilize the training. Extensive experiments demonstrate the effectiveness and superiority of our proposed DRA over existing works, especially on enhancing the performance of new classes on various benchmarks, including base-new classes, cross-datasets evaluation and domain generalization. The source code will be published after the paper is received. 1. Introduction Pre-trained large vision-language models (VLMs) ( Radford et al. 2021 ; Huang et al. 2023 ; Li et al. 2022 ; Yao et al. 2022 ) , serving as foundational models, have recently shown significant progress. Typically trained under supervision on large-scale image-text pair data, the most representative work, CLIP ( Radford et al. 2021 ) , employs a contrastive learning strategy to map image information and text descriptions into a shared feature space. During inference, hand-crafted prompts, such as "a photo of a [ class ]", are often used for tasks like classification through image-text matching. Benefiting from large-scale pre-training, these VLMs generally exhibit excellent generalization capabilities across a range of downstream tasks without the need for fine-tuning ( Zanella and Ben Ayed 2024 ; Zhai et al. 2022 ; Phan et al. 2024 ; Javed et al. 2024 ) . Although large VLMs have shown impressive generalization capabilities, the large amount of training data and large model parameter size required hinder their further fine-tuning in downstream tasks. Prompt learning, as a widely used method, is often used to transfer pre-trained VLMs to downstream tasks. The core idea is to introduce task-specific learnable prompt tokens in the text encoder ( Zhou et al. 2022b ; Zhou et al. 2022c ; Yao et al. 2023 ; Yao et al. 2024 ) , image encoder ( Jia et al. 2022 ) , or both ( Khattak et al. 2023a ; Khattak et al. 2023b ) . During training, only prompt tokens are optimized, while other parameters of the model remain frozen. For example, CoOp ( Zhou et al. 2022c ) proposed replacing hand-designed prompts with learnable prompts and optimizing them in fine-tuning. MaPLe ( Khattak et al. 2023a ) proposed adding learnable prompts to both text branch and visual branch and using prompts to build interactions between modalities. ArGue ( Tian et al. 2024 ) proposed to generate several attribute prompts for each category to improve the discriminant ability of the model for each category. Prompt learning greatly reduces the computational requirements and provides a practical method to transfer pre-trained VLMs for downstream tasks. Beyond prompt learning, another pipeline of work adapts pre-trained large models to downstream tasks by fine-tuning lightweight external modules which are known as "Adapters" ( Rebuffi et al. 2017 ; Houlsby et al. 2019 ; Chen et al. 2022 ; Zhang et al. 2022 ; Zhao et al. 2024a ) . Adapter architectures like Clip-Adapter ( Gao et al. 2024 ) and AdaptFormer ( Chen et al. 2022 ) insert Adapter modules at the final layer or intermediate layers of the frozen network, enabling parameter-efficient fine-tuning. LoRA ( Hu et al. 2021 ) proposes performing a low-rank decomposition of the weight updates for linear layers and merging the result with the original feature path, utilizing re-parameterized techniques to avoid additional computational overhead during inference. MMA ( Yang et al. 2024 ) introduces Adapters into the image and text branches separately and facilitates cross-modal information interaction by sharing an MLP between the two branches. Compared to prompt learning, adapter-based methods often offer greater flexibility when applied to a broader range of backbone networks. Figure 1. The motivation of the proposed method is to dynamically assign different ranks to different tokens and reduce overfitting of irrelevant information. However, despite the significant progress of these fine-tuning methods, such as Prompt learning and Adapters, they often struggle to maintain strong generalization capabilities, particularly when encountering unseen new classes after being fine-tuned on the base classes ( Zhang et al. 2024 ) . These fine-tuning methods can significantly enhance the model’s performance on base classes due to their robust fitting capabilities, but often at the expense of generalization capabilities on unseen new classes. This phenomenon limits the potential of the model for application in open scenarios. We observe that existing fine-tuning methods generally give equal attention to all input tokens during the adaptation process, as shown in Figure 1 . Excessive adaptation to tokens with low informational value or weak task relevance (such as image background noise or blank word vectors in text sequences) not only introduces computational redundancy but may also compromise the model’s generalization capability due to the forgetting of common features that are critical for generalization. Based on the above observations, this paper aims to optimize the VLM fine-tuning process by introducing dynamic importance weighting to specifically enhance cross-class generalization capabilities while maintaining foundational model competencies. We propose a novel Adapter variant method called Dynamic Rank Adaptation (DRA). The core idea of DRA is to overcome the fixed-rank constraint in adapter design, dynamically allocating adaptation capacity according to the significance/importance of input text and image feature tokens. Specifically, we first employ a token attention mechanism to assess the importance of each token. This score indicates the likelihood that each token carries discriminative information within the current sample. Based on these scores, we sort the tokens and divide them into different importance groups. Then, DRA dynamically allocates adaptation ranks based on the importance of features to preserve the general knowledge during training, i.e., it adopts rank adaptation dynamically by assigning higher feature ranks to the more important tokens. In addition, we develop a new channel response mechanism to prioritize the preservation and adaptation of feature channels identified as the most informative for each instance. Finally, a L1 regularization term is introduced to stabilize the whole training. We verify the effectiveness of DRA on a wide range of benchmarks, including generalization from base