Towards Fine-Grained Adaptation of CLIP via a Self-Trained Alignment Score

Eman Ali, Sathira Silva, Chetan Arora, Muhammad Haris Khan · 2025

Read Towards Fine-Grained Adaptation of CLIP via a Self-Trained Alignment Score by Eman Ali, Sathira Silva, Chetan Arora, Muhammad Haris Khan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-09615.txt.

Introduction

Eman Ali and co-authors sharpen how CLIP adapts to fine class splits in a 2025 paper. CLIP sorts images by text match with no task training, but telling near twin classes apart, like bird species, needs finer cues. Past adaption tricks lean on fixed match scores or on costly fake labeling rounds.

Their method, FAIR, pairs local image traits with class description anchors built from rich text. A learned alignment score then acts as a shifting classifier that mints better fake labels for self training. A weight rule trims label noise where classes blur. Across 13 fine grained sets the method beat top rivals by a clear margin.

Catalog record

CREATOREman Ali, Sathira Silva, Chetan Arora, Muhammad Haris Khan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-09615

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Fine-Grained Adaptation of CLIP via a Self-Trained Alignment Score Eman Ali, Sathira Silva, Chetan Arora, Muhammad Haris Khan Abstract Vision-language models (VLMs) like CLIP excel in zero-shot learning by aligning image and text representations through contrastive pretraining. Existing approaches to unsupervised adaptation (UA) for fine-grained classification with VLMs either rely on fixed alignment scores that cannot capture evolving, subtle class distinctions or use computationally expensive pseudo-labeling strategies that limit scalability. In contrast, we show that modeling fine-grained cross-modal interactions during adaptation produces more accurate, class-discriminative pseudo-labels and substantially improves performance over state-of-the-art (SOTA) methods. We introduce Fine-grained Alignment and Interaction Refinement ( FAIR ), an innovative approach that dynamically aligns localized image features with descriptive language embeddings through a set of Class Description Anchors (CDA). This enables the definition of a Learned Alignment Score (LAS), which incorporates CDA as an adaptive classifier, facilitating cross-modal interactions to improve self-training in unsupervised adaptation. Furthermore, we propose a self-training weighting mechanism designed to refine pseudo-labels in the presence of inter-class ambiguities. Our approach, FAIR , delivers a substantial performance boost in fine-grained unsupervised adaptation, achieving a notable overall gain of 2.78% across 13 fine-grained datasets compared to SOTA methods. 1 1 1 Our code will be made public upon acceptance. . 1 Introduction Figure 1 : Class-wise classification accuracy (%) comparison between FAIR and state-of-the-art methods on the EuroSAT [ 19 ] dataset. FAIR achieves significant improvements, especially in visually and semantically similar classes where state-of-the-art methods struggle. FAIR distinguishes fine-grained classes, demonstrating robust adaptation to challenging class distributions. Large pretrained vision-language models (VLMs), such as CLIP [ 43 ] , have demonstrated remarkable zero-shot capabilities by learning joint visual and textual representations through contrastive training on large-scale image–text pairs. This breakthrough has led to CLIP’s widespread application across multiple computer vision tasks, including image classification [ 43 , 26 , 61 ] , object detection [ 17 , 47 , 51 ] , and semantic segmentation [ 29 , 55 , 35 ] , all without task-specific supervision. However, adapting CLIP to fine-grained classification tasks, where visually similar classes must be distinguished, remains a significant challenge. In such tasks, CLIP’s reliance on global image features and generic prompts (e.g., “ a photo of a { CLASS },”) often fails to capture the subtle visual cues necessary to separate fine-grained categories [ 58 , 40 ] . This leads to degraded performance on datasets such as EuroSAT, where different classes may share highly overlapping visual characteristics (see Fig. 1 : ZS CLIP). At the same time, acquiring labeled training data for these domains can be costly or infeasible [ 62 , 22 ] . However, state-of-the-art (SOTA) methods often assume access to a set of target-domain class names, even in the absence of labeled images [ 23 , 38 , 21 , 3 ] . Recent works have proposed unsupervised adaptation (UA) of CLIP in this setting by utilizing Large Language Models (LLMs) to generate informative prompts or captions conditioned on class names [ 31 , 42 , 56 ] . For example, LaFTer [ 38 ] demonstrates that LLM-generated descriptions enable effective pseudo-labeling for adapting CLIP to fine-grained target datasets. LaFTer first uses these descriptions to fine-tune a classifier, which is then frozen and employed to generate pseudo-labels while fine-tuning CLIP’s visual branch. However, a major limitation lies in its reliance on fine-tuned global image features, while the alignment scores are still computed using a fixed pretrained classifier. In contrast, DPA [ 3 ] relies on fixed, handcrafted prompt ensembles, averaging them to create prototypical classifiers that are fine-tuned during adaptation. Similarly, ReCLIP [ 21 ] uses fixed prompts to refine the embedding space and label assignments through label propagation and cross-modality self-training. However, the dependence of both DPA and ReCLIP on manually crafted prompts limits their effectiveness, particularly on fine-grained datasets, as illustrated in Fig. 1 . WCA [ 31 ] , a recent zero-shot method, enhances the alignment score function by integrating LLM-generated captions with localized visual prompts (image crops), improving classification by aligning crop features with textual descriptions. However, its reliance on generating a large number of local image crops for pseudo-labeling significantly increases computational cost in unsupervised adaptation, as shown in Fig. 2 . These limitations motivate us to enhance WCA’s alignment score function to improve both efficiency and performance in CLIP’s unsupervised adaptation. To this end, we introduce F ine-grained A lignment and I nteraction R efinement ( FAIR ), a novel unsupervised adaptation framework for CLIP that introduces a Learned Alignment Score (LAS) based on selected local image crops and adaptive class representations. Our contributions can be summarized as follows: • We propose a novel approach that combines selective crop relevance weighting and adaptive alignment scoring to enhance cross-modal alignment. Specifically, we (1) utilize the [CLS] token from the visual encoder to compute weights that rank image crops by relevance, selecting the top- k most relevant crops for further alignment, and (2) introduce Class Description Anchors (CDA) as an adaptive classifier. These components define Learned Alignment Score (LAS) , enabling fine-grained interactions between the top- k crops and CDA. This process, dubbed F ine-grained A lignment and I nteraction R efinement ( FAIR ), significantly enhances pseudo-labeling accuracy and self-training. • We propose a novel weighting mechanism for the self-supervision loss function in FAIR , to enhance pseudo-labeling accuracy in the presence of confusing classes. • Through extensive evaluation on 13 fine-grained datasets, FAIR achieves a substantial gain of 2.78\% , outperforming the state-of-the-art. Figure 2 : Efficiency comparison of FAIR vs. WCA [ 31 ] as a pseudo-labeling method using the EuroSAT dataset. The radius of each circle corresponds to the amount of GPU memory allocated. 2 Related Work Vision-Language Models (VLMs): Vision-Language Models (VLMs) [ 43 , 33 , 30 , 24 , 44 , 57 , 2 ] demonstrate remarkable zero-shot performance in visual recognition tasks, owing to pretraining on vast image–text pairs from the web. CLIP [ 43 ] , a pioneering VLM, excels in capturing vision–language correspondences, enabling zero-shot predictions across diverse inputs. However, CLIP’s performance is highly susceptible to prompting, making optimal prompt selection complex and time-consuming [ 64 ] . Recent research [ 64 , 63 , 34 , 26 , 37 , 9 ] has introduced methods to refine prompts using few-shot samples from target datasets, aiming to reduce reliance on manual prompt engineering. In contrast, our work proposes an entirely label-free approach to adapting CLIP to the target task. Unsupervised Adaptation for CLIP: Despite advancements in supervised fine-tuning of CLIP, obtaining even a few labeled samples per class is often impractical in many applications [ 62 , 22 ] . To address this limitation, recent research has focused on fine-tuning CLIP with unlabeled samples from target datasets [ 23 , 46 ] . However, these methods heavily rely on zero-shot CLIP’s pseudo-labeling, which has shown limitations in fine-grained recognition tasks [ 58 , 40 ] . CPL [ 60 ] uses multiple candidate pseudo-labels as possible true labels instead of single hard labels. However, its effectiveness relies on candidate quality, as the true label may not be included among them. LaFTer [ 38 ] leverages LLMs

Read the complete text