Differentiable Prompt Learning for Vision Language Models

Zhenhan Huang, Tejaswini Pedapati, Pin-Yu Chen, Jianxi Gao · 2025

Read Differentiable Prompt Learning for Vision Language Models by Zhenhan Huang, Tejaswini Pedapati, Pin-Yu Chen, Jianxi Gao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-00457.txt.

Introduction

Zhenhan Huang, Tejaswini Pedapati, Pin-Yu Chen, and Jianxi Gao study how to tune prompts for vision language models without hand design. Their method, differentiable prompt learning, treats prompt design as an optimization problem. It picks the best context length to add at each layer of the model. Tests use the pre-trained CLIP model and only limited data.

The team finds the method locates a deep continuous prompt setup with high confidence. On downstream tasks it beats baseline methods, raising average test accuracy by 2.60% across 11 datasets. Since it only sets prompt configuration, the method can work alongside existing baseline approaches.

Catalog record

CREATORZhenhan Huang, Tejaswini Pedapati, Pin-Yu Chen, Jianxi Gao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-00457

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Differentiable Prompt Learning for Vision Language Models Zhenhan Huang, Tejaswini Pedapati, Pin-Yu Chen, Jianxi Gao Abstract Prompt learning is an effective way to exploit the potential of large-scale pre-trained foundational models. Continuous prompts parameterize context tokens in prompts by turning them into differentiable vectors. Deep continuous prompts insert prompts not only in the input but also in the intermediate hidden representations. Manually designed deep continuous prompts exhibit a remarkable improvement compared to the zero-shot pre-trained model on downstream tasks. How to automate the continuous prompt design is an underexplored area, and a fundamental question arises, is manually designed deep prompt strategy optimal? To answer this question, we propose a method dubbed differentiable prompt learning (DPL). The DPL method is formulated as an optimization problem to automatically determine the optimal context length of the prompt to be added to each layer, where the objective is to maximize the performance. We test the DPL method on the pre-trained CLIP. We empirically find that by using only limited data, our DPL method can find deep continuous prompt configuration with high confidence. The performance on the downstream tasks exhibits the superiority of the automatic design: our method boosts the average test accuracy by 2.60 % on 11 datasets compared to baseline methods. Besides, our method focuses only on the prompt configuration ( i.e . context length for each layer), which means that our method is compatible with the baseline methods that have sophisticated designs to boost the performance. The DPL method can be deployed to large language models or computer vision models at no cost. 1 Introduction Parameter-Efficient fine-tuning (PEFT) offers a paradigm to adapt pretrained model to downstream tasks using a small set of trainable parameters. Among the PEFT methods, the prompt learning can perform few-shot learning or even zero-shot learning to adapt the pre-trained models to new scenarios. Using appropriate prompts, both text prompts Brown et al. 2020 ; Lester et al. 2021 ; Schick and Schütze 2020b ; Shin et al. 2020 ; Gao et al. 2020 and visual prompts Bar et al. 2022 ; Shtedritski et al. 2023 ; Chen et al. 2023 ; Tsai et al. 2024 , to query foundational models have shown great potential. At the same time, the selection of context tokens in prompts has a pronounced effect on the performance of the prompt learning methods Zhou et al. 2022b ; Schick and Schütze 2021 ; Schick and Schütze 2020a ; Haviv et al. 2021 . Continuous prompts use differentiable vectors that are in the embedding space of the foundation model. Continuous prompts use trainable parameters that are learned during the fine-tuning process. It avoids the time-consuming trial-and-error process using discrete context tokens. Besides, the continuous prompts are not constrained to be the embeddings of word tokens in the vocabulary. The selection for the embeddings of context tokens becomes continuous. Deep continuous prompts add prompts to multiple layers. This method has shown superior performance compared to only adding continuous prompts to the input in both vision Jia et al. 2022 ; Zhu et al. 2023b ; Zhu et al. 2023b ; Yoo et al. 2023 and language fields Lester et al. 2021 ; Li and Liang 2021 ; Liu et al. 2021 ; Liu et al. 2023 . There are two hyperparameters: the context length of continuous prompts c_{p} and prompt depth \ell_{p} . Let the hidden dimension be d , continuous prompts \mathbf{E}\in\mathbb{R}^{c_{p}\times d} are inserted to the input to the neural network layers up to \ell_{p} layers. Since \mathbf{E} are automatically determined during the fine-tuning process, a natural question is: can we automatically determine c_{p} and \ell_{p} ? Recently, it has been found that training merely part of deep neural network layers can achieve a performance comparable to or even better than training all layers. This line of work indicates that there is a subset of layers in the pre-trained model, depending on the distribution shift between the pre-training dataset and fine-tuning dataset, whose parameters might be close to a minima for downstream tasks Lee et al. 2022 ; Lodha et al. 2023 ; Panigrahi et al. 2023 ; Vettoruzzo et al. 2024 . In the prompt learning, we postulate that there might be some layers that do not need deep continuous prompts or only need continuous prompts with shorter context length compared to rest of layers. In the existing prompt learning works, however, a fixed number c_{p} is used for each layer. To examine our postulation, we design our method to insert continuous prompts with different context lengths. The context lengths of continuous prompts are determined using a differentiable formulation. If a layer has the best context length found to be 0, we do not add continuous prompts to this layer. We name the proposed method differentiable prompt learning (DPL). The few-shot learning experiments show that the DPL method can find continuous prompt configurations, i.e., the context length and depth of continuous prompts inserted to the input of each layer. The performance of downstream fine-tuning over 11 datasets shows the superiority of the proposed method. In summary, our DPL method has the following contributions: • The DPL method automates the continuous prompt learning process. It automatically determines the continuous prompts and associated hyperparameters including the context length of the continuous prompt inserted for each neural network layer and prompt depth. • The DPL method removes the constraint in the manually designed continuous prompt methods that the context length for each layer is fixed. The heterogeneous design of inserting continuous prompts enables a more flexible design for the prompt learning. The method is simple and focuses only on continuous prompts, which means it can be combined with sophisticated designs in existing prompt learning methods. • We find the optimal deep continuous prompt configuration is dataset-dependent. By tailoring deep continuous prompts for each dataset, the DPL method can achieve the performance better than manually designed prompt learning methods. 2 Background and Related Work Prompt Learning Prompt learning offers an efficient way to adapt pre-trained foundational models to downstream tasks. This technique is of great interest in both vision Wang et al. 2022b ; Wang et al. 2022a and natural language Jiang et al. 2020 ; Shin et al. 2020 . CoOp Zhou et al. 2022b inserts continuous prompts to the input of the vision-language model. The visual prompt tuning Jia et al. 2022 proposes the deep continuous prompts to vision transformer Dosovitskiy et al. 2020 . MaPLe Khattak et al. 2023 combines the ideas from these two works by using deep continuous prompts in both the text branch and the image branch. Similar to CoOp, CoCoOp, PLOT and ProGrad insert continuous prompts only in the input. CoCoOp Zhou et al. 2022a inserts the continuous prompts conditioning on the input images. PLOT Chen et al. 2022 uses the computationally costly iterative algorithm to compute the transport plan for aligning word tokens and image patches. ProGrad Zhu et al. 2023a uses the gradient-aligned knowledge distillation to avoid the overfitting problem. All of the above-mentioned methods use fixed context length as opposed to our proposed DPL method which adapts the context length for each layer dynamically based on the dataset. Neural Architecture Search Neural architecture search (NAS) automatically

Read the complete text