Beyond Words: Augmenting Discriminative Richness via Diffusions in Unsupervised Prompt Learning

Hairui Ren, Fan Tang, He Zhao, Zixuan Wang, Dandan Guo, Yi Chang · 2025

Read Beyond Words: Augmenting Discriminative Richness via Diffusions in Unsupervised Prompt Learning by Hairui Ren, Fan Tang, He Zhao, Zixuan Wang, Dandan Guo, Yi Chang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-11930.txt.

Introduction

Hairui Ren and five co-authors grow synthetic images to teach vision-text models with no hand labels. Such teaching needs fake labels for wild sets, yet text-first labels often clash with what the image shows. Their 2025 method, AiR, makes high-grade synthetic images for each class and trains a side classifier on them. The task thus shifts from text-image pairs to image-image pairs, a firmer ground.

The spread of synthetic images also feeds prompt learning with more variety of view per class. On five bench sets and three learning kinds, bare, half-labeled, and fresh-class, the method beats top peers by firm, even steps. Rich synthetic views, they hold, are the missing wealth in label-free teaching.

Catalog record

CREATORHairui Ren, Fan Tang, He Zhao, Zixuan Wang, Dandan Guo, Yi Chang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-11930

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Words: Augmenting Discriminative Richness via Diffusions in Unsupervised Prompt Learning Hairui Ren, Fan Tang, He Zhao, Zixuan Wang, Dandan Guo, Yi Chang Abstract Fine-tuning vision-language models (VLMs) with large amounts of unlabeled data has recently garnered significant interest. However, a key challenge remains the lack of high-quality pseudo-labeled data. Current pseudo-labeling strategies often struggle with mismatches between semantic and visual information, leading to sub-optimal performance of unsupervised prompt learning (UPL) methods. In this paper, we introduce a simple yet effective approach called A ugmenting D i scriminative R ichness via Diffusions (AiR), toward learning a richer discriminating way to represent the class comprehensively and thus facilitate classification. Specifically, our approach includes a pseudo-label generation module that leverages high-fidelity synthetic samples to create an auxiliary classifier, which captures richer visual variation, bridging text-image-pair classification to a more robust image-image-pair classification. Additionally, we exploit the diversity of diffusion-based synthetic samples to enhance prompt learning, providing greater information for semantic-visual alignment. Extensive experiments on five public benchmarks, including RESISC45 and Flowers102, and across three learning paradigms-UL, SSL, and TRZSL-demonstrate that AiR achieves substantial and consistent performance improvements over state-of-the-art unsupervised prompt learning methods. Code is available. 1 Introduction Large pre-trained vision-language models (VLMs) [ 24 , 32 , 12 ] have achieved impressive performance in open-world visual concept learning without requiring task-specific training; however, they still benefit from adaptation for optimal results. Prompt tuning [ 7 , 13 , 26 , 25 ] offers an efficient strategy to enhance VLMs’ discriminative accuracy on downstream tasks by learning inputs to the model. While prompt learning can yield notable improvements, a large amount of labeled data is still required to enhance VLMs’ performance across diverse downstream tasks. This adaptation requirement leads to considerable labeling costs, as labeled data is hard to obtain [ 30 , 36 , 14 , 35 ] . To address this, unsupervised prompt learning (UPL) [ 11 , 21 , 17 , 28 ] has been introduced, leveraging the zero-shot capabilities of VLMs to generate pseudo labels, enabling effective use of unlabeled data. For pseudo-labeling, traditional methods [ 11 ] typically form a text classifier by inputting prompts with class names into the VLMs’ text encoder. Then, image embeddings of unlabeled data are matched to these class-based text classifiers, assigning pseudo-labels to images with high-confidence predictions. Some other approaches [ 21 , 33 ] generate multiple candidate pseudo-labels by adjusting confidence thresholds or optimize the model with different numbers of pseudo-labeled samples at various training stages. While these methods apply distinct strategies for filtering pseudo-labels or refining prompts, most ultimately rely on pseudo-labels produced by text classifiers, making the quality of these pseudo-labels critically important. Figure 1 : Left: Confusion matrix for ground truth labels vs. text classifier predictions on the Flowers102 dataset, highlighting persistent generation of incorrect pseudo-labels. Right: An example of misclassification in Flowers102, where “Thorn apple” is misidentified as “Giant white arum lily” due to its white petals resembling the latter’s semantic information. This misclassification can be alleviated by jointly considering text and image predictions. In this work, we aim to improve the pseudo label quality in UPL, which can thus be easily combined with other UPL methods. Here, we first explore conduct a pilot experiment to empirically demonstrate the deficiency of text classifiers. Specifically, we leverage the text encoder of CLIP [ 24 ] to generate pseudo labels on the Flowers102 dataset [ 22 ] and subsequently calculated the confusion matrix between the ground-truth labels and predicted class labels. The results, as shown in Fig. 1 , reveal that a substantial number of samples are incorrectly predicted, e.g. , most “Thorn apple” flowers are misclassified as “Giant white arum lily”, “Tree mallow” is misclassified as “Mexican petunia”. Why this happened? The discrimination of unlabeled images heavily relies on the precision of text classifiers, which are constructed using the words of prompt with category label names. However, when only using the text of “a photo of a + class name” to describe the image, using “Giant white arum lily” as the class name is even closer in semantics to the given image than “Thorn apple”. Because the Thorn apple flower appears to be a shade of white but the word “Thorn apple” does not express that, resulting in the misclassified samples. It motivates us to learn a richer discriminating way to represent the class comprehensively and thus facilitate classification. Since an image has rich information, we consider a novel solution: in addition to the text classifier built on the short text description, can we design another auxiliary classifier based on the most representative images for each category? However, this solution is problematic since we do not know the ground-truth label of each training image in the unsupervised prompt learning task, making it a dead-end loop. To address the above problem, in this paper, we draw inspiration from the image generation literature. Recent advancements in image generation have made it possible to generate fidelity and diverse images with only text word descriptions, such as Stable Diffusion [ 27 ] . It is effective in generating diverse images with richer visual appearance variation while preserving the key semantics. Surprisingly, we find that this text-to-image generation model can be naturally utilized to generate the synthetic representative images for each class in UPL tasks only with the text descriptions on the class names. Therefore, we propose Beyond Words: A ugmenting D i scriminative R ichness via Diffusions, a new framework AiR which applies diffusion models to generate synthetic images, which serve as a bridge from text-image-pair to image-image-pair classification, as a way to discriminate the unlabeled images. In our AiR framework, we adopt Stable Diffusion [ 27 ] to generate discriminative richness, where we fine-tune the Stable Diffusion model with LoRA [ 10 ] by means of pseudo-labeled samples with high confidence to ensure prediction fidelity. Then, we introduce cosine similarity-based filtration to select the most representative synthetic image of each class generated by fine-tuned Diffusion, which is used to build the auxiliary classifier from the view of the image. Combining the predictions of the original text classifier and the auxiliary classifier, we can endow the training images with more accurate pseudo labels, improving the UPL performance. Besides, the diffusion-based synthetic samples can also be utilized to learn the prompt. Notably, our proposed AiR only needs to know the dataset name and class name of downstream tasks, which can be seamlessly integrated into most UPL methods . Experimental results show that AiR achieves a notable improvement in classification accuracy by an average of 2.6% in comparison to the state-of-the-art UPL method. To sum up, our contributions can be included: • We highlight that relying solely on the text classifier to generate pseudo labels in UPL is inadequate and propose a richer discriminating way to represent the class comprehensively and thus facilitate classification. • We present a novel approach AiR: A ugmenting D i scriminative R ichness via Diffusions. This method leverages the diversity and fidelity of synthetic samples to build an auxiliary classifier that captures richer variations in visual appearance while preserving essential semantics, bridging text-image-pair classification to image-image-pair classification. • Our AiR demonstrates substantial improvement across three distinct unsupervised paradigms

Read the complete text