Token Coordinated Prompt Attention is Needed for Visual Prompting Zichen Liu, Xu Zou, Gang Hua, Jiahuan Zhou * Abstract Visual prompting techniques are widely used to efficiently fine-tune pretrained Vision Transformers (ViT) by learning a small set of shared prompts for all tokens. However, existing methods overlook the unique roles of different tokens in conveying discriminative information and interact with all tokens using the same prompts, thereby limiting the representational capacity of ViT. This often leads to indistinguishable and biased prompt-extracted features, hindering performance. To address this issue, we propose a plug-and-play Token Coordinated Prompt Attention (TCPA) module, which assigns specific coordinated prompts to different tokens for attention-based interactions. Firstly, recognizing the distinct functions of CLS and image tokens-global information aggregation and local feature extraction, we disentangle the prompts into CLS Prompts and Image Prompts, which interact exclusively with CLS tokens and image tokens through attention mechanisms. This enhances their respective discriminative abilities. Furthermore, as different image tokens correspond to distinct image patches and contain diverse information, we employ a matching function to automatically assign coordinated prompts to individual tokens. This enables more precise attention interactions, improving the diversity and representational capacity of the extracted features. Extensive experiments across various benchmarks demonstrate that TCPA significantly enhances the diversity and discriminative power of the extracted features. The code is available at https://github.com/zhoujiahuan1991/ICML2025-TCPA . Figure 1: Above: Visualization of the attention map. The existing visual prompting method VPT ( jia2022visual ) learns the same prompts for all tokens, resulting in extracted information that lacks distinguishability and comprehensiveness. Our TCPA selects corresponding prompts for different tokens and performs attention interaction, thereby enhancing the diversity and discriminability of the extracted information. Below: Comparison of time overhead and performance. 1 Introduction In recent years, the pretraining-finetuning strategy has become a foundational paradigm in the deep learning field, significantly advancing the progress of various multi-media technologies ( jang2019learning ; guo2019spottune ; iofinova2022well ; xu2025dask ; li2025caprompt ; yao2025selective ) . However, as the sizes of models and datasets have rapidly exploded, such a popular paradigm has faced critical challenges due to its high storage and computational costs ( jia2022visual ) . Addressing this, recent research ( He_2020_CVPR ; cai2020tinytl ; zhang2020side ; han20232vpt ) has focused on efficiently adapting pretrained models to specific downstream tasks. Among them, visual prompting has emerged as a leading player ( bahng2022exploring ; jia2022visual ; huang2023diversity ) by introducing a minimal set of learnable prompts into the latest vision transformer (ViT) without retraining the original model parameters. Existing visual prompting methods can be primarily categorized into two branches. Various works involve adding learnable prompts directly to the input sample itself, guiding the model to focus on discriminative information at the input-level ( bahng2022exploring ; chen2023understanding ; huang2023diversity ; tsao2024autovp ) . Besides, another branch introduces learnable tokens as prompts incorporated into each self-attention layer in ViT ( jia2022visual ; han20232vpt ; yoo2023improving ; wang2024revisiting ) . They aim to continuously prompt the model throughout the entire feature extraction process, facilitating the extraction of discriminative features. However, these methods usually learn and leverage the same prompt for all tokens without considering the different functionalities of CLS and image tokens, as well as the varying discriminative information conveyed by different image tokens. Consequently, this leads to different tokens focusing on similar regions and extracting biased discriminative information as shown in Figure 1 , thereby limiting the representation ability of ViT. To address the above issues, we introduce a plug-and-play Token Coordinated Prompt Attention (TCPA) module. It assigns specific coordinated prompts to different tokens for targeted attention-based interactions, allowing each prompt to contribute effectively to the extraction of comprehensive and discriminative information. Specifically, considering that CLS tokens and image tokens focus on global information aggregation and local feature extraction, respectively, we design CLS prompts and Image prompts for the CLS token and image tokens. These prompts interact exclusively with CLS tokens and image tokens within the attention blocks, thereby enhancing the discriminability of the extracted features. Furthermore, since different image tokens correspond to distinct image patches and the information they need to extract varies, we further disentangle CLS prompts and Image prompts into a CLS prompt Pool and an Image prompt Pool, each composed of multiple prompts. Token-coordinated prompts are automatically assigned to each token, improving the diversity of discriminative information in the extracted features. To sum up, the main contributions of this work are: (1) To address the issues in existing visual prompting methods, we introduce a plug-and-play Token Coordinated Prompt Attention (TCPA) module, which assigns specific prompts to different tokens for targeted attention-based interactions, allowing each prompt to contribute effectively to the extraction of comprehensive and discriminative information. (2) Considering the differences in the information extracted by CLS and image tokens, as well as among different image tokens, we first disentangle the prompts into CLS prompts and Image prompts. We then match corresponding prompts to different tokens, fostering coordinated interactions between tokens and prompts and enhancing the discriminative ability of the extracted features. (3) Extensive experiments on various benchmarks show that TCPA consistently enhances the performance of existing state-of-the-art visual prompting methods. Figure 2: The overall pipeline of our proposed TCPA. For each input sample, embeddings for each image patch are first obtained through the embedding layer. Then, CLS and image tokens adaptively select appropriate prompts from the corresponding CLS and Image Prompt Pools and generate a binary mask. This binary mask is then fed into the attention module to mask certain values in the attention map, enabling attention-based interactions between different tokens and different prompts. 2 Related Work 2.1 Parameter-Efficient Fine-Tuning Vision Transformer (ViT) has made significant strides in computer vision research ( dosovitskiy2020image ; liu2021swin ; arnab2021vivit ; chen2021crossvit ; wang2021pyramid ) . However, the continuously increasing model sizes and datasets pose challenges in fully fine-tuning pretrained ViT models for downstream tasks, leading to substantial storage and computational cost. Consequently, recent studies ( zhang2020side ; jia2022visual ; han20232vpt ) have shifted focus towards reducing the number of trainable parameters to streamline the fine-tuning process, broadly categorized as partial tuning-based , extra module-based , and prompt learning-based approaches. Partial tuning-based methods ( yosinski2014transferable ; He_2020_CVPR ; noroozi2016unsupervised ; zhang2016colorful ) aim to retain most of the pretrained backbone while fine-tuning a smaller subset of parameters. Although straightforward and easy to implement, these methods often exhibit a noticeable performance gap compared to full fine-tuning ( chenempirical ) . On the other hand, extra module-based approaches ( rebuffi2017learning ; zhang2020side ; cai2020tinytl ; pfeiffer2020adapterhub ; zaken2021bitfit ) introduce additional learnable plug-in architectures to fine-tune the pretrained model. However, these approaches are often tailored to specific architectures, limiting their applicability to other models. Moreover, the introduction of additional learnable parameters poses practical challenges, making them less feasible in real-world scenarios ( jia2022visual ; han20232vpt ) . 2.2 Prompt Learning Prompt learning techniques initially emerged in the realm of natural language processing (NLP), involving the incorporation of a small set of learnable soft prompts into input texts to customize language models for specific downstream tasks ( li2021prefix ; liu2021p ) . Recent research has extended prompt learning to visual tasks, known as visual prompt tuning or visual prompting ( bahng2022exploring ; jia2022visual ; han20232vpt ; liu2024dart ; liu2024compositional ; liu2024insvp ) . Compared to partial tuning-based and extra module-based methods, visual prompting-based approaches introduce significantly fewer additional parameters and exhibit superior compatibility with models of various architectures ( jia2022visual ; han20232vpt ) . Specifically, existing visual prompting methods can be mainly categorized into two types based on the location where prompts are applied: those added on the input image and those added within the token sequence. Prompting methods added on the input image overlay learnable visual prompts onto the original image to adjust pretrained models from the input level, enabling them to adapt to downstream tasks ( bahng2022exploring ; huang2023diversity ; chen2023understanding ; wang2023lion ; tsao2024autovp ) . These methods adjust pretrained models only in the input image level, making them adaptable to different network structures. However, since visual prompts are not used in the middle layers of the network, the representation capacity of prompts in such methods is limited, resulting in limited performance. Another category of visual prompting involves introducing learnable tokens into the intermediate layers of the model, which undergos self-attention along with CLS and image tokens, thereby extracting discriminative features ( jia2022visual ; han20232vpt ; yoo2023improving ; wang2024transhp ) . For instance, VPT ( jia2022visual ) introduces learnable tokens at every layer of the vision transformer, allowing for individual adjustments across layers to better suit downstream tasks. These methods continuously provide prompts during the feature extraction process of the model. However, they use the same prompts for all tokens without considering the distinct roles of CLS and image tokens, as well as the differences in discriminative information extracted by various image tokens. This results in the features extracted by different tokens being neither distinguishable nor comprehensive, which limits the model’s performance. 3 Token Coordinated Prompt Attention In this section, we illustrate the proposed TCPA in detail, and the overall pipeline is depicted in Figure 2 . 3.1 Notations In the architecture of a pretrained Vision Transformer (ViT) backbone, denoted as \mathcal{M} , there are L instances of MSA (Multi-Head Self-Attention) blocks, symbolized as \{\mathcal{B}_{j}\}_{j=1}^{L} . Each block, \mathcal{B}_{j} , integrates multi-head self-attention with feed-forward networks, incorporating both LayerNorm and residual pathways for enhanced processing. When processing an input image \boldsymbol{x} , with dimensions \mathbb{R}^{H\times W\times C} , this image is partitioned into N patches of uniform size \{\boldsymbol{x}_{i}\in\mathbb{R}^{h\times w\times C}\}_{i=1}^{N} . Here, (H,W) represents the size of \boldsymbol{x} , C is the channel count, and (h,w) denotes the size of each patch \boldsymbol{x}_{i} . The transformation of each patch \boldsymbol{x}_{i} into a D -dimensional feature space is given by: \boldsymbol{h}_{i}^{1}=\mathcal{E}(\boldsymbol{x}_{i}), (1) where \boldsymbol{h}_{i}^{1}\in\mathbb{R}^{D} , and \mathcal{E}(\cdot) is the embedding layer of \mathcal{M} . These embedded patches, \{\boldsymbol{h}_{i}^{1}\}_{i=1}^{N} , along with a classification (CLS) token \boldsymbol{c}_{1}\in\mathbb{R}^{D} , are sequentially processed through the L MSA blocks \{\mathcal{B}_{j}\}_{j=1}^{L} . The operation can be summarized as follows: [\boldsymbol{c}_{j+1},\boldsymbol{h}_{1}^{j+1},\cdots,\boldsymbol{h}_{N}^{j+1}]=\mathcal{B}_{j}([\boldsymbol{c}_{j},\boldsymbol{h}_{1}^{j},\cdots,\boldsymbol{h}_{N}^{j}]), (2) where the “ [~] ” denotes the concatenation of the vectors. The final classification is conducted by passing the last MSA block’s output, \boldsymbol{c}_{L+1} , through a classifier \mathcal{H} : \boldsymbol{y}=\mathcal{H}(\boldsymbol{c}_{L+1}) (3) 3.2 Coordinated Prompt Attention of CLS and Image Tokens In the vision transformer, an image \boldsymbol{x} is initially divided into numerous small patches, which are then converted into corresponding image tokens via an embedding layer. During the attention process, image tokens continuously extract discriminative information from the input sample \boldsymbol{x} . This information is subsequently aggregated through a CLS token, summarizing the insights gathered from all image tokens for final classification. It is evident that the role of image tokens is to extract discriminative information, whereas the CLS token’s purpose is to aggregate this information and facilitate classification, highlighting the distinct functions of these two types of tokens. Hence, we design Coordinated Prompt Attention of CLS and Image Tokens, which disentangles prompts for CLS and image tokens, aiding them in better fulfilling their respective functions. Specifically, we disentangle prompts for CLS and image tokens, denoted as \boldsymbol{\mathrm{P}}^{c} and \boldsymbol{\mathrm{P}}^{i} , respectively: \boldsymbol{\mathrm{P}}^{c}=\{\boldsymbol{p}^{c}_{j}\}_{j=1}^{L}, (4) \boldsymbol{\mathrm{P}}^{i}=\{\boldsymbol{p}^{i}_{j}\}_{j=1}^{L}, (5) where \boldsymbol{p}^{c}_{j}\in\mathbb{R}^{L_{p}\times D} and \boldsymbol{p}^{i}_{j}\in\mathbb{R}^{L_{p}\times D} are CLS and image prompt for j -th MSA block \mathcal{B}_{j} . Then we feed the CLS token \boldsymbol{c}_{j} , image tokens (\boldsymbol{h}_{1}^{j},\cdots,\boldsymbol{h}_{N}^{j}) , and the CLS prompt \boldsymbol{p}^{c}_{j} together into the MSA block \mathcal{B}_{j} , obtaining the corresponding output: \displaystyle[\boldsymbol{c}_{j+1},\boldsymbol{p}^{c,d}_{j+1},\boldsymbol{h}_{1}^{j+1,d},\cdots,\boldsymbol{h}_{N}^{j+1,d}] (6) \displaystyle=\mathcal{B}_{j}([\boldsymbol{c}_{j},\boldsymbol{p}^{c}_{j},\boldsymbol{h}_{1}^{j},\cdots,\boldsymbol{h}_{N}^{j}]), where the subscript d in \boldsymbol{p}^{c,d}_{j+1},\boldsymbol{h}_{1}^{j+1,d},\cdots,\boldsymbol{h}_{N}^{j+1,d} indicates that these outputs will be discarded and not utilized by subsequent MSA layers. In the equation above, only the output CLS token continues to be used, and therefore, the CLS prompt only affects the CLS token, not the image tokens. Similarly, we input the image prompt \boldsymbol{p}^{i}_{j} along with the CLS token \boldsymbol{c}_{j} and image tokens (\boldsymbol{h}_{1}^{j},\cdots,\boldsymbol{h}_{N}^{j}) into the MSA block \mathcal{B}_{j} , obtaining the output for the image tokens: [\boldsymbol{c}^{d}_{j+1},\boldsymbol{p}^{i,d}_{j+1},\boldsymbol{h}_{1}^{j+1},\cdots,\boldsymbol{h}_{N}^{j+1}]=\mathcal{B}_{j}([\boldsymbol{c}_{j},\boldsymbol{p}^{i}_{j},\boldsymbol{h}_{1}^{j},\cdots,\boldsymbol{h}_{N}^{j}]), (7) where the subscript d in \boldsymbol{c}^{d}_{j+1},\boldsymbol{p}^{i,d}_{j+1} indicates that these outputs will be discarded. Through the equation above, we obtain the output for the image tokens (\boldsymbol{h}_{1}^{j+1},\cdots,\boldsymbol{h}_{N}^{j+1}) , which, together with the previously obtained output of the CLS token \boldsymbol{c}_{j+1} , serves as the input for the next MSA block \mathcal{B}_{j+1} . 3.3 Coordinated Prompt Attention of Different Image Tokens In the previous text, we disentangle prompts into CLS and image prompts based on their distinct roles. However, in vision transformers, different image tokens correspond to different image patches with varying discriminative information. Using the same prompts for all image tokens can make the extracted features indistinguishable and biased. Thus, we further introduce coordinated prompt attention of different image tokens to enhance the pretrained model’s ability to extract rich, discriminative information from the input image. To simplify notation, in the following discussion, we will not explicitly differentiate prompts from different layers. It’s important to note that while the parameters of prompts vary across layers, the processing method for prompts remains consistent throughout. Specifically, we disentangle the image prompt \boldsymbol{p}^{i} into a image prompt pool \mathcal{P}^{i} composed of multiple image prompts: \mathcal{P}^{i}=\{(\boldsymbol{p}^{i}_{k},\boldsymbol{\kappa}^{i}_{k})\}_{k=1}^{N_{i}}, (8) where \boldsymbol{\kappa}^{i}_{k} represents the learnable indicator corresponding to the prompt \boldsymbol{p}^{i}_{k} , used for selecting the prompt based on the image token. For each image token \boldsymbol{h}_{m}^{j} , the distance between \boldsymbol{h}_{m}^{j} and the prompt indicator \boldsymbol{\kappa}^{i}_{k} can be measured via a cosine distance \mathcal{S}(\cdot,\cdot) : \begin{aligned} \mathcal{S}(\boldsymbol{h}_{m}^{j},\boldsymbol{\kappa}^{i}_{k})&=1-\mathrm{cos}(\boldsymbol{h}_{m}^{j},\boldsymbol{\kappa}^{i}_{k})\\ &=1-\frac{\boldsymbol{h}_{m}^{j}\cdot\boldsymbol{\kappa}^{i}_{k}}{\left\|\boldsymbol{h}_{m}^{j}\right\|_{2}\cdot\left\|\boldsymbol{\kappa}^{i}_{k}\right\|_{2}}\\ \end{aligned}. (9) Through the above process, we obtain the affinity matrix \mathrm{\mathbf{A}}\in\mathbb{R}^{N\times N_{i}} between image tokens and different image prompts, where \mathrm{\mathbf{A}}_{m,k}=\mathcal{S}(\boldsymbol{h}_{m}^{j},\boldsymbol{\kappa}^{i}_{k}) . Then, we perform binarization on the matrix \mathbf{A} , setting the top K_{i} largest elements in each row to 1 while assigning 0 to all other elements. Specifically, the binarized matrix \mathrm{\mathbf{\hat{A}}}\in\{0,1\}^{N\times N_{i}} is defined as follows: \mathrm{\mathbf{\hat{A}}}_{m,k}=\mathbb{I}\left(\sum\limits_{s=1}^{K_{i}}\mathbb{I}(k=\pi_{m}(s))>0\right), (10) where \mathbb{I}(\cdot) denotes the indicator function, which takes the value of 1 if the condition inside holds and 0 otherwise; \pi_{m} represents the index sequence obtained by sorting the elements of the i -th row of matrix \mathrm{\mathbf{A}} in descending order, i.e., satisfying \mathrm{\mathbf{A}}_{m,\pi_{m}(1)}\geq\mathrm{\mathbf{A}}_{m,\pi_{m}(2)}\geq\dots\geq\mathrm{\mathbf{A}}_{m,\pi_{m}(n)} ; and \pi_{m}(s) corresponds to the column index of the s -th largest element after sorting. Through this operation, we obtain a binary mask matrix \mathbf{M} , which selects the top K_{i} largest elements in each row of \mathrm{\mathbf{A}} while suppressing the influence of other irrelevant elements. Then, we align the dimensions of \mathrm{\mathbf{\hat{A}}} with the dimensions of the attention map to obtain the final image token mask: \mathrm{\mathbf{M}}^{i}_{m,k}=\begin{cases}0,&\text{if }m=0\\ \mathrm{\mathbf{\hat{A}}}_{m+1,k},&\text{otherwise}\end{cases} (11) To guide the CLS token corresponding to different samples in better aggregating global information, we also disentangle the CLS prompt \boldsymbol{p}^{c} into a CLS prompt pool \mathcal{P}^{c}=\{(\boldsymbol{p}^{c}_{k},\boldsymbol{\kappa}^{c}_{k})\}_{k=1}^{N_{c}} . In a similar manner to the image tokens, we can obtain the affinity matrix between the CLS token and the CLS prompts. Then, by further binarizing and expanding the dimensions, we obtain the mask \mathrm{\mathbf{M}}^{c} corresponding to the CLS token. In Vision Transformers, the core operation of the attention module is: \mathrm{Attn}=\text{Softmax}\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right), (12) where \mathrm{Attn} is the attention map. We concatenate the two masks, \mathrm{\mathbf{M}}^{c} and \mathrm{\mathbf{M}}^{i} , corresponding to the CLS token and the image token, then expand them to the same dimensions as the attention map Attn , resulting in the final mask \mathrm{\mathbf{M}} . Finally, we perform an element-wise multiplication between Attn and the mask \mathrm{\mathbf{M}} to obtain the updated attention map for subsequent operations: \text{Attn}^{\prime}=\text{Attn}\odot\mathrm{\mathbf{M}}, (13) where \odot denotes the element-wise multiplication. Although our TCPA selects specific prompts for each token, the additional computational overhead is limited only to the calculation of cosine distance for prompt selection and the self-attention process, with no increase in the computation of the feed-forward network. Furthermore, we can achieve the effect of multiple groups of tokens undergoing attention separately through a single attention computation by utilizing token masking. In TCPA, we compute the attention weights of all tokens and prompts, \text{Attn}=\text{Softmax}\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right) . Then, based on the token-prompt matching, we generate a binary mask matrix \mathrm{\mathbf{M}} and compute \text{Attn}\cdot\mathrm{\mathbf{M}}\cdot V . This approach calculates the attention weights only once, enabling efficient interaction between different tokens and prompts. Figure 3: 3D and 2D attention map of existing visual prompting method VPT ( jia2022visual ) and Ours. Table 1: The comparison results on HTA benchmark. Partial , Extra , and Prompting represent partial tuning-based, extra module-based, and prompt learning-based methods respectively. \rectanglecolor gray!201-21-14 \rectanglecolor gray!2013-213-14 \rectanglecolor gray!2015-215-14 \rectanglecolor gray!2017-217-14 \rectanglecolor gray!2019-219-14 \rectanglecolor gray!2021-221-14 Methods DTD CUB Bird Dog Flower Food Cifar100 Cifar10 GTSRB SVHN Avg Full - 64.3 87.3 82.7 89.4 98.8 84.9 68.9 97.4 97.1 87.4 85.8