A Retrospect to Multi-prompt Learning across Vision and Language

Ziliang Chen, Xin Huang, Quanlong Guan, Liang Lin Weiqi Luo, Jinan University Sun Yat-sen University Pazhou Laboratory [email protected], [email protected], {Gql,lwq}@jnu.edu.cn, [email protected] · 2025

Read A Retrospect to Multi-prompt Learning across Vision and Language by Ziliang Chen, Xin Huang, Quanlong Guan, Liang Lin Weiqi Luo, Jinan University Sun Yat-sen University Pazhou Laboratory [email protected], [email protected], {Gql,lwq}@jnu.edu.cn, [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-00191.txt.

Introduction

Ziliang Chen, Xin Huang, and Quanlong Guan offer a principled retrospect on multi-prompt learning for vision-language models. Prompt learning adapts pretrained vision-language models to tasks with limited resources, but research has milled around single prompts while multi-prompt variants remain underexplored. They extend the constant modality gap phenomenon to learnable prompts.

That extension justifies the superiority of vision-language transfer with multi-prompt augmentation, supported empirically. Computer vision readers get theory for a practice that often runs ahead of understanding. Multiple prompts can cover varied phrasings and visual angles, and this retrospect explains when and why that helps.

Catalog record

CREATORZiliang Chen, Xin Huang, Quanlong Guan, Liang Lin Weiqi Luo, Jinan University Sun Yat-sen University Pazhou Laboratory [email protected], [email protected], {Gql,lwq}@jnu.edu.cn, [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-00191

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Retrospect to Multi-prompt Learning across Vision and Language Ziliang Chen, Xin Huang, Quanlong Guan, Liang Lin Weiqi Luo, Jinan University Sun Yat-sen University Pazhou Laboratory [email protected], [email protected], {Gql,lwq}@jnu.edu.cn, [email protected] Abstract The vision community is undergoing the unprecedented progress with the emergence of Vision-Language Pretraining Models (VLMs). Prompt learning plays as the holy grail of accessing VLMs since it enables their fast adaptation to downstream tasks with limited resources. Whereas existing researches milling around single-prompt paradigms, rarely investigate the technical potential behind their multi-prompt learning counterparts. This paper aims to provide a principled retrospect for vision-language multi-prompt learning. We extend the recent constant modality gap phenomenon to learnable prompts and then, justify the superiority of vision-language transfer with multi-prompt augmentation, empirically and theoretically. In terms of this observation, we propose an Energy-based Multi-prompt Learning (EMPL) to generate multiple prompt embeddings by drawing instances from an energy-based distribution, which is implicitly defined by VLMs. So our EMPL is not only parameter-efficient but also rigorously lead to the balance between in-domain and out-of-domain open-vocabulary generalization. Comprehensive experiments have been conducted to justify our claims and the excellence of EMPL. 1 Introduction Recent years have witnessed the rise of multimodal intelligence, in particular, Vision-Language Pre-training models (VLMs), e.g. , CLIP [ 40 ] , ALIGN [ 24 ] , achieving downstream tasks in low resources by converting the prior knowledge behind large language models (LLMs) [ 10 , 4 ] . Given a pair of image encoder ( e.g. , ResNet [ 19 ] , ViT [ 11 ] , etc ) and text encoder ( i.e. , LLMs), VLMs align visual features with their corresponding textual description embeddings via contrastive learning [ 12 , 18 , 56 ] . So provided a text known as prompt , VLMs may rapidly adapt to diverse tasks [ 12 , 32 ] by matching harmony visual patterns with the textual description. The principle sheds a new light in computer vision for in-domain and out-of-domain generalization. Figure 1: The overview of cross-modal single-prompt learning and multi-prompt learning (MPL). With more prompt templates, MPL brings new opportunities and challenges as discussed in the community, yet seldom giving a systematic investigation and solution. The impressive cross-modal transferability behind VLM typically owes to the problem-customized prompting style, yet demanding a great magnitude of trials and errors for selecting the ideal prompt template from a pool of candidates. Tedious workloads are consumed and do not guarantee the optimal prompt template either. Instead of the prompt engineering, prompt learning / tuning [ 55 ] sidesteps the obstacle using soft prompting: outside of the words related with what we are interested in, the rest of textual slots in the template are replaced by a sequence of learnable context vectors ahead of the text encoder. In this principle, the optimal prompt template could be achieved by fine-tuning the learnable context vectors along with the given textual semantic while keeping the rest parameters of VLMs frozen for the optimization. The data-driven merit increasingly arouses a flood of interests in the community [ 54 , 42 , 46 ] . Despite the significant progress, existing work of prompt learning focused on a single template whereas multi-prompt learnable context templates remain under-explored (Fig. 1 ). Recent NLP advance argue that instructing LLMs via more prompts may trigger its underlying in-context learning ability to master new skills [ 4 ] . In this regard, multi-prompting is also deemed to be a promising trend for VLMs. On the other hand, existing studies remain confused of how multiple prompts work for VLMs, particularly from two aspects. The first is vision-language transferrability [ 53 ] . Prompt augmentation eliminates the average cross-modal transfer shift while [ 54 ] showed that increasing the scale of context vector tokens resulted in the detrimental effect, implying a larger cross-modal disparity. The second is open-vocabulary (OV) generalization , i.e. , the model awareness of unseen classes. Different from multple textual descriptions, learning with more prompts suggests more parameters. It casted a doubt of overfitting to training classes and put unseen classes at the risk of model generalization [ 5 ] . In this paper, we provided several principled insights to understand multi-prompt learning empirically and theoretically . We first consider the cross-modal embedding space following the constant modality-gap phenomenon found by [ 53 ] . With regards to our empirical observations, we extend the conclusion to learnable prompts to show that more learnable prompts might reduce the constant modality gap more significantly. In terms of constant modality gaps, we further proved the existence of cross-modal unidentifiability issue : a paradox confusing the cross-modal model with a single prompt template in visual recognition. It could be restrained by multi-prompting empirically, thus, interpreting why multi-prompt learning could outperform single prompt for the sake of vision-language transferrability. In terms of our retrospect, the main challenge of multi-prompt learning refers to its generalizability. Derived from this concern, we propose a new methodology Energy-based Multi-Prompt Learning (EMPL) for striking the balance between in-domain generalization and open-vocabulary generalization abilities. EMPL implicitly defines an energy-based [ 27 ] prompt distribution that simultaneously use image and prompt as the variable. With this regard, our method could be rigorously treated as modeling the uncertainty to explore the image-prompt embedding pairs with concepts out of the training domains, whereas also well generalizes to examples belonging to in-domain classes. The prompts are iteratively generated via a stochastic Markov Chain Monte Carlo (MCMC) sampler [ 50 ] , which is parameter-efficient, sensitive of input knowledge from vision-text encoders, and more importantly, general enough to cooperate with existing prompt learning strategies to upgrade the performances. Experiments are comprehensively conducted to validate our claims and the superiority of our approach. 2 Related Work Vision-Language Pre-trained (VLM) models. VLM models, which unify the two most commonly used modalities, vision and language, have gained great popularity due to the success of pre-trained models [ 4 ] in CV and NLP. Among numerous VLM models [ 40 , 24 ] , CLIP [ 40 ] is the most widely used and representative one. It utilizes a pair of image and text encoders to receive information from both modalities and leverages a large amount of paired image and text data collected from the Internet. In contrast to other VLM models that use Masked language modeling [ 25 , 30 ] , Masked region prediction [ 47 , 45 ] , etc., CLIP utilizes feature vectors from both encoders to train with the Contrastive Learning [ 40 ] strategy, which successfully aligns the feature space of both modalities and has been widely employed for a variety of downstream tasks [ 17 , 15 , 34 ] . Prompt learning. Prompt tuning [ 38 , 41 ] , a technique derived from the field of natural language processing (NLP), has gained great popularity [ 55 , 54 , 31 ] in the field of VLP in recent years, which has the ability to unleash the potential of pre-trained multimodal models. CoOp [ 55 ] , a well-known text branching technique, eliminates the need for manual prompt design by transforming input context tokens into learnable vectors. CoCoOp [ 54 ] , its successor, overcomes its

Read the complete text