SelfPrompt: Confidence-Aware Semi-Supervised Tuning for Robust Vision-Language Model Adaptation
Shuvendu Roy, Ali Etemad · 2025
Read SelfPrompt: Confidence-Aware Semi-Supervised Tuning for Robust Vision-Language Model Adaptation by Shuvendu Roy, Ali Etemad (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-14148.txt.
Introduction
Shuvendu Roy and Ali Etemad tune vision-language models with few labels in a 2025 paper. Such models link images with words and can sort new classes with little training. But when labels are scarce, the model must label the rest itself, a step called pseudo-labeling. Poorly set confidence makes those self labels noisy, and errors pile up.
Their method, SelfPrompt, groups unlabeled images into clusters to guide cleaner self labels. A confidence-aware module then mixes true-label learning with weak-label learning to use more of the unlabeled pool. A sampling trick picks a small, varied labeled set when the label budget is tight. Across 13 data sets, gains averaged above six percent in the main tests. Single-shot gains were larger still.
Catalog record
| CREATOR | Shuvendu Roy, Ali Etemad |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-14148
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SelfPrompt: Confidence-Aware Semi-Supervised Tuning for Robust Vision-Language Model Adaptation Shuvendu Roy, Ali Etemad Abstract We present SelfPrompt, a novel prompt-tuning approach for vision-language models (VLMs) in a semi-supervised learning setup. Existing methods for tuning VLMs in semi-supervised setups struggle with the negative impact of the miscalibrated VLMs on pseudo-labelling, and the accumulation of noisy pseudo-labels. SelfPrompt addresses these challenges by introducing a cluster-guided pseudo-labelling method that improves pseudo-label accuracy, and a confidence-aware semi-supervised learning module that maximizes the utilization of unlabelled data by combining supervised learning and weakly-supervised learning. Additionally, we investigate our method in an active semi-supervised learning setup, where the labelled set is strategically selected to ensure the best utilization of a limited labelling budget. To this end, we propose a weakly-supervised sampling technique that selects a diverse and representative labelled set, which can be seamlessly integrated into existing methods to enhance their performance. We conduct extensive evaluations across 13 datasets, significantly surpassing state-of-the-art performances with average improvements of 6.23% in standard semi-supervised learning, 6.25% in active semi-supervised learning, and 4.9% in base-to-novel generalization, using a 2-shot setup. Furthermore, SelfPrompt shows excellent generalization in single-shot settings, achieving an average improvement of 11.78%. 1 Introduction Vision-language models (VLMs) ( Radford et al., 2021 ) pre-trained on large-scale datasets of image-text pairs have shown strong generalization on a wide range of tasks. Nonetheless, prior works ( Zhou et al., 2022a ; Zhou et al., 2022b ) have demonstrated that VLMs require fine-tuning on a considerable amount of labelled data to perform well on downstream tasks. Additionally, the size of the foundation model makes fine-tuning in a limited labelled data setting difficult without losing generalization ( Roy & Etemad, 2024 ) . To reduce the reliance on labelled data, some recent works have explored solutions that utilize auxiliary unlabelled data ( Huang et al., 2022 ; Menghini et al., 2023 ; Zhang et al., 2024 ) to improve learning from a limited set of labelled data. Although prior works that leverage unlabelled data for tuning VLMs show substantial performance gains, we identify several limitations in such approaches. (a) Given the unlabelled set, prior works ( Menghini et al., 2023 ; Zhang et al., 2024 ) utilize the zero-shot capabilities of pre-trained VLMs to predict pseudo-labels for the unlabelled data to then use as labelled samples. However, pre-trained VLMs do not necessarily possess adequate knowledge of the downstream domain, which lead to incorrect pseudo-labels. Such wrong labels, especially in few-labelled settings, can negatively impact the final performance of the model. (b) To learn from unlabelled data more effectively, previous works ( Menghini et al., 2023 ; Zhang et al., 2024 ) have employed incremental pseudo-labelling, wherein the labelled set is continuously expanded by iteratively adding to the pseudo-label set from the unlabelled set. Nevertheless, as Figure 1 illustrates, this method often results in the accumulation of noisy pseudo-labels, ultimately leading to performance degradation. To solve the above-mentioned problems, we propose SelfPrompt , a new prompt tuning approach that uses weak supervision by the pre-trained VLM itself to fine-tune the model with a confidence-aware semi-supervised learning approach. SelfPrompt comprises two components. (a) Cluster-guided pseudo-labelling: To address the first problem, we propose a cluster-guided pseudo-labelling approach that avoids using the VLM to predict the pseudo-labels at the start of the training. Instead, we cluster all labelled and unlabelled samples in the embedding space, using the labelled set as the cluster centres. Samples near the centroids of these clusters are then selected and assigned the corresponding class labels to form the pseudo-labelled set. (b) Confidence-aware semi-supervised learning: To make the best use of the unlabelled data, we propose a confidence-aware semi-supervised module. This hybrid approach leverages high-confidence pseudo-labels in a fully supervised learning setting, while learning from low-confidence samples in a weakly-supervised manner. In a standard semi-supervised learning setup, we are provided with a limited labelling budget (few samples per class), where existing methods ( Zhang et al., 2024 ; Menghini et al., 2023 ) typically select a random subset from the unlabelled set for labelling. However, such random selection may fail to adequately represent the underlying data distribution, leading to an inefficient labelling strategy. In this work, we explore active semi-supervised learning , where a subset is strategically selected from the unlabelled set to form the labelled set. To achieve this, we propose weakly-supervised sampling, a novel approach that can be integrated with any existing semi-supervised learning method for selecting a diverse and representative subset of samples from the unlabelled data. Our weakly-supervised sampling follows a two-step protocol. First, predictions from a VLM (vision-language model) are used as a source of weak supervision to filter out both the most and least confident samples from the unlabelled set. This is based on the observation that highly confident samples have low information gain and contribute little to learning, while the least confident samples are often noisy and unrepresentative of the dataset. Next, a clustering-based selection technique identifies a diverse set of samples from the remaining unlabelled data for labelling. Figure 1 : (left) Pseudo-label accuracy; (right) Test accuracy over training sessions. We present extensive evaluations of our proposed solution on both standard semi-supervised learning and active semi-supervised learning setups. While previous works ( Menghini et al., 2023 ; Zhang et al., 2024 ) report these results on six datasets, we evaluate our solution (and the previous methods) on 13 datasets . Our evaluation shows that SelfPrompt outperforms prior works by 6.23% on average in the standard semi-supervised learning setup and by 6.25% on the active semi-supervised learning setup, with up to 11.23% and 13.82% improvements on individual datasets, respectively. Additionally, our weakly-supervised sampling module shows considerable improvement when added to the existing methods. We further demonstrate the generalizability of our solution by reducing the size of the labelled set to just one sample-per-class while outperforming prior methods by an average of 11.78%. For completeness, we also compare our solution to few-shot tuning methods ( Zhou et al., 2022a ; Khattak et al., 2022 ; Roy & Etemad, 2024 ; Li et al., 2024 ) on base-to-novel generalization, where we also show an average improvement of 4.9% over prior works in a 2-shot setup, with improvements in both base and novel classes. Finally, we present extensive ablation and sensitivity studies on different components of our proposed method. Overall, our contributions are: Figure 2 : Performance comparison to prior works on semi-supervised tuning of VLMs. • We introduce SelfPrompt, a novel prompt-tuning approach for VLMs that uses weak supervision from the pre-trained VLM itself to learn effective representations from limited labelled data while improving generalization. • We introduce three novel components: (a) Cluster-guided pseudo-labelling, which improves pseudo-label accuracy by assigning labels based on proximity to labelled samples; (b) Confidence-aware semi-supervised learning, which adapts learning based on the confidence of pseudo-labels to maximize the use of the unlabelled data; and (c) Weakly-supervised labelled set sampling technique that selects a representative and diverse set of labelled samples; • Our comprehensive experiments across 13 datasets demonstrate that SelfPrompt surpasses existing methods on various benchmarks, achieving average improvements of 6.23% in standard semi-supervised learning, 6.25% in active semi-supervised learning, and 4.9% in base-to-novel generalization, in the standard 2-shot evaluation setup. Additionally, SelfPrompt