Integrated Structural Prompt Learning for Vision-Language Models

Jiahui Wang, Qin Xu, Bo Jiang, Bin Luo · 2025

Read Integrated Structural Prompt Learning for Vision-Language Models by Jiahui Wang, Qin Xu, Bo Jiang, Bin Luo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-05677.txt.

Introduction

Jiahui Wang, Qin Xu, Bo Jiang, and Bin Luo present a prompt learning method that respects structure inside vision-language models. Their 2025 work builds on models like CLIP, tuned for new tasks with small learned prompts. Past methods mostly skip how prompts relate to tokens within and across image and text sides.

The new method adds self and cross structure modules for those links. That design moves information well while keeping base features stable. A sample probing part shifts loss weights by sample hardness, so easy cases do not dominate. Tests span new-class, cross-data, and cross-domain settings. Results match or beat top rival methods, the authors report, with better balance on fresh classes.

Catalog record

CREATORJiahui Wang, Qin Xu, Bo Jiang, Bin Luo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-05677

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Integrated Structural Prompt Learning for Vision-Language Models Jiahui Wang, Qin Xu, Bo Jiang, Bin Luo Abstract Prompt learning methods have significantly extended the transferability of pre-trained Vision-Language Models (VLMs) like CLIP for various downstream tasks. These methods adopt handcraft templates or learnable vectors to provide text or image instructions in fine-tuning VLMs. However, most existing works ignore the structural relationships between learnable prompts and tokens within and between modalities. Moreover, balancing the performance of base and new classes remains a significant challenge. In this paper, we propose an Integrated Structural Prompt (ISP) for VLMs to enhance the interaction of information representations between the text and image branches. ISP introduces self-structural and cross-structural prompt modules to model the structural relationships between learnable prompts and frozen tokens within and across modalities. This enables efficient information transfer while preserving feature stability. Additionally, we propose a sample probing module that dynamically adjusts loss coefficients based on sample difficulty, preventing the mode from overfitting to simple samples and improving generalization ability to new classes. Extensive experiments on three widely used settings: base-to-new generalization, cross-dataset evaluation, and domain generalization demonstrate that the proposed ISP achieves competitive performance against state-of-the-art methods. \ArticleType RESEARCH PAPER \Year 2021 \Month \Vol \No \DOI \ArtNo \ReceiveDate \ReviseDate \AcceptDate \OnlineDate [email protected] \AuthorMark Author A \AuthorCitation Jiahui Wang, Author B, Author C, et al keywords Prompt Learning, Vision-Language Models, Transfer Learning, Structure Learning. 1 Introduction The deep learning models that are pre-trained on large-scale datasets [ 5 , 52 ] are widely used in various downstream tasks, such as image classification [ 13 , 6 , 38 ] , object detection [ 11 , 33 ] , and semantic segmentation [ 40 , 24 ] . With the recent application of image-text datasets, large-scale vision-language models (VLM) [ 47 , 46 , 21 , 1 , 31 ] have been widely used. For example, CLIP [ 31 ] uses 400 million image-text pairs to pre-train the vision-language models and demonstrates strong potential in multi-modal tasks. However, the huge parameters of VLMs make it very difficult to fine-tune and transfer to other tasks with limited labeled data. Prompt learning has been widely adopted to migrate VLM efficiently. Early works [ 22 , 26 ] usually design handcrafted templates for text token sequences to transfer VLM to different tasks. For example, CLIP [ 31 ] first pre-trained the VLMs with image-text pairs and then adopted handcraft templates to build text prompts to adapt to the image classification task. However, pre-designed prompt words are not flexible enough for further fine-tuning. CoOp [ 55 ] proposed to use learnable prompts instead of manually designed prompt words for the class names and freeze the parameters of the CLIP encoder to better generalize for the downstream tasks. KgCoOp [ 43 ] proposed to regularize the features extracted by learnable prompts with the frozen text features extracted by handicraft prompts to reduce the forgetting of original knowledge. HPT [ 39 ] suggested the use of hierarchical descriptions to supplement text features with linguistic knowledge. However, the above methods based on prompt engineering usually learn prompts separately within each modality, hindering information relevance modeling. Thus, Maple [ 23 ] adds learnable prompts to both text and visual token sequences and transfers information from the text branch to the visual branch through prompts. COMMA [ 20 ] proposed the fusion of the text prompts and visual prompts of previous layers into post layers to pass messages between modalities. MMA [ 42 ] proposed to use adapters to fine-tune the network instead of the design of prompts and add a shared linear projection between the two modalities to transfer information. However, we found that the existing work still faces several issues: 1) the structural relevance of the learnable prompts within and between modalities is not effectively utilized. The learnable prompts represent domain-related but class-independent knowledge, while the frozen image tokens bring more discriminative class-related knowledge. During fine-tuning, it is difficult for the frozen encoder layer to construct the relationship between the two types of tokens, which hinders the interaction between domain information and sample information. 2) There is a significant difference in the feature distribution between text and image features, and simply conducting feature interaction between multi-modal features may lead to network instability. 3) Due to the limited number of labeled samples, previous methods often overfit simpler samples from the base classes and demonstrate suboptimal generalization abilities to new classes. To solve the above problems, we propose integrated structural prompts for visual language models. Specifically, in order to make full use of the structural relevance between prompts and tokens, our proposed self-structural prompt constructs correlations between prompt and text/visual tokens by performing information transfer by cross attention. The optimized prompts fuse the features of the samples contained in frozen tokens while retaining the domain-related information in learnable prompts. To highlight important information and improve the efficiency of information transfer, we only use the tokens with higher responses to construct the correlation matrix. Then, to effectively transfer information between modalities and keep feature stability, we design the cross-structural prompt module. By constructing an affinity graph between the prompt of one modality and the tokens of another modality, the internal relation representation of the prompts is constructed. Then, graph convolution is used to transfer structural information between modalities to refine the prompts. Finally, to solve the problem of overfitting to simple samples, we propose the sample probing that defines the difficulty of each sample according to the difference between predicted probability after the prompt and the prediction results of zero-shot CLIP and sets different loss coefficients. Extensive experiments under widely used settings, including base-new generalization, cross-dataset evaluation, and domain generalization settings, demonstrate that our method achieves leading results on most datasets. These results display a strong generalization ability to downstream image classification tasks. In general, our contributions are summarized as follows: • Our proposed integrated structural prompt fully utilizes the structural relationship between learnable prompts and frozen tokens to transfer information within and between modalities. • The proposed self-structural prompt and cross-structure prompt to refine the learnable prompts while keeping the stability of features. • We design the sample probing that applies different loss coefficients to each sample, thereby reducing the weights of the model to simple samples to improve the generalization ability. 2 Related Works 2.1 Vision-Language Models The ability to learn joint multi-modal representations has been significantly advanced by Visual Language Models (VLMs) like CLIP [ 31 ] , ALIGN [ 21 ] , FLIP [ 46 ] , and BLIP [ 25 ] . These models leverage contrastive learning, that maximizes similarity for matched image-text pairs while minimizing it for unmatched ones and benefit greatly from pre-training on large-scale datasets [ 34 ] , demonstrating strong performance on downstream tasks. However, a major limitation arises from their large size and dependence on extensive training data, making deployment difficult in specialized environments, especially when labeled data is scarce (few-shot settings). This bottleneck has spurred investigation into methods for efficiently adapting pre-trained VLMs, as seen in recent studies [ 10 , 49 , 8 , 51 , 12 , 14 , 53 ] focusing on improving performance in few-shot and zero-shot contexts for various

Read the complete text