Rethinking Few Shot CLIP Benchmarks: A Critical Analysis in the Inductive Setting
Alexey Kravets, Da Chen, Vinay P. Namboodiri · 2025
Read Rethinking Few Shot CLIP Benchmarks: A Critical Analysis in the Inductive Setting by Alexey Kravets, Da Chen, Vinay P. Namboodiri (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-20834.txt.
Introduction
Kravets, Chen, and Namboodiri challenge how few-shot image sorting is scored in a 2025 paper. The base model, CLIP, sorts images with only a few examples. Most test sets were in CLIP's training diet, so scores may flatter true skill on fresh data. The team uses an unlearning step to build fairer test sets.
In the cleaned setting, thirteen known methods lost over half their score on average. Checks with all-knowing base models backed the unlearning step. The team then offers a stronger method that topped the field across thousands of runs. They frame the work as a reset for how few-shot skill should be judged.
Catalog record
| CREATOR | Alexey Kravets, Da Chen, Vinay P. Namboodiri |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 94 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-20834
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Rethinking Few Shot CLIP Benchmarks: A Critical Analysis in the Inductive Setting Alexey Kravets, Da Chen, Vinay P. Namboodiri Abstract CLIP is a foundational model with transferable classification performance in the few-shot setting. Several methods have shown improved performance of CLIP using few-shot examples. However, so far, all these techniques have been benchmarked using standard few-shot datasets. We argue that this mode of evaluation does not provide a true indication of the inductive generalization ability using few-shot examples. As most datasets have been seen by the CLIP model, the resultant setting can be termed as partially transductive . To solve this, we propose a pipeline that uses an unlearning technique to obtain true inductive baselines. In this new inductive setting, the methods show a significant drop in performance ( -55% on average among 13 baselines with multiple datasets). We validate the unlearning technique using oracle baselines. An improved few-shot classification technique is proposed that consistently obtains state-of-the-art performance over 13 other recent baseline methods on a comprehensive analysis with 5880 experiments - varying the datasets, differing number of few-shot examples, unlearning setting, and with different seeds. Thus, we identify the issue with the evaluation of CLIP-based few-shot classification, provide a solution using unlearning, propose new benchmarks, and provide an improved method. Our code and models are available here . 1 Introduction Contrastive Language Image Pretraining (CLIP) model [ 28 ] is a widely used vision-language foundational model. Radford et al. [ 28 ] demonstrated the ability for CLIP to improve its classification using a few examples (termed linear probing). Subsequently, various works have further improved the ability for a CLIP model to be used as a few-shot classification method. [ 10 , 40 , 13 , 30 , 45 , 43 , 44 , 18 , 19 , 39 ] Figure 1 : Comparisons between the previous evaluation pipeline (Top: partially transductive) and the proposed pipeline (Bottom: inductive). Top: The tested methods ( M_{1-3} ) apply the pretrained CLIP model which the target classes are likely to be included during training. Bottom: The proposed inductive pipeline aims to provide the test methods with an updated CLIP model with target class information unlearned. The target dataset, unlearning module and few-shot methods are all replaceable. The improvement obtained from the few-shot classification is crucial as it can benefit the classification tasks with limited samples. Existing methods can achieve good results with tasks such as classifying the latest dress styles for which the related knowledge has been learned during pretraining on CLIP. However, there are also lots of practical scenarios that need to classify unseen classes with few-shot techniques, e.g., when it is used to identify new types of AI-generated harmful content or emerging diseases. For these use cases, the methods need to be evaluated using inductive benchmarks where the model learns specific information about a class from the few-shot examples of the class. However, the current evaluation of few-shot CLIP classification models is obtained using standard few-shot classification datasets. These results are not indicative of true few-shot generalization performance for unseen classes. As CLIP is trained with a large corpus, most existing datasets would have been seen by the model. Evaluating using a few-shot classification method would result in a partially transductive evaluation - where the performance for a specific class is based on previously seen test samples for that specific class. As shown in Fig. 1 , we investigate this aspect of the problem and propose a pipeline for evaluating few-shot methods in an inductive manner. Specifically, we provide a means to fairly evaluate on any benchmark by incorporating unlearning. In this new setting, a significant difference in performance for all 13 CLIP-based few-shot baselines is observed as indicated in Fig. 2 . Moreover, one could question whether the unlearning method itself is successful in unlearning a class. E.g., it may be that the unlearning method itself demonstrates poor classification performance, but the knowledge may still be retained within the model. To ensure the validity of the unlearning method, we train two types of CLIP models from scratch, one using the data from all the classes in the ImageNet dataset and another set of CLIP models trained on subsets of ImageNet that exclude certain classes. In this way, we can provide an oracle baseline where the model has not seen the class data as a fair baseline. The trend in few-shot classification performance is similar in the oracle baseline and the corresponding unlearned baselines. The methodology we propose is flexible as it can be used to create an inductive benchmark out of any dataset and any model for which a reliable unlearning method exists, i.e., an unlearning method that can retain most of the model’s original knowledge after removing the information about a specific benchmark dataset. In this paper, a baseline CLIP-based few-shot learning method is also proposed. Unlike previous methods which tend to modify the weights towards the end of the CLIP model [ 40 ] or modify the weights internally based on the latent space prompt [ 39 ] , the proposed method called Self-Enhanced Prompt Tuning with Residual Textual Features (SEPRES) tends to modify weights both internally and towards the end of the model simultaneously. As shown in Fig. 2 and other experimental results in Sec. 5 , SEPRES consistently achieves the best performance against other baselines in various settings. We make the following contributions: • A problem with the few-shot CLIP classification is identified. Existing few-shot CLIP classification methods are not evaluated using inductive benchmarks. The usual high performance for most methods is due to the partially transductive nature of evaluation (as shown in Fig. 2 ). • A novel pipeline is proposed to obtain inductive benchmarks using unlearning - as shown in Tab. 1 . Accuracy of methods like CoOp [ 43 ] drops sharply from 71.4% to just 15.3% on average across datasets and shots in the proposed pipeline for inductive evaluation. A detailed analysis with all components under different setups of the proposed pipeline is conducted with 5880 experiments as shown in Fig. 4(a) . • We validate the unlearning method using oracle baselines. The results in Fig. 6 show that the proposed solution is consistent with oracle baselines. • A baseline few-shot learning method, SEPRES, is proposed, which achieves state-of-the-art results against all 13 baselines under different settings and datasets. Figure 2 : Unlearned vs Not Unlearned performance for all few-shot methods. All baseline methods perform much worse in our setting with our proposed SEPRES method being robust in both settings. 2 Related Work Few-shot learning in CLIP In the field of CLIP-based few-shot learning, prompt tuning has emerged as a key strategy to improve generalization and adaptability to novel tasks. Traditional prompt engineering involves manually crafting prompts, but recent methods [ 43 , 44 , 45 , 13 ] automate this process by learning continuous prompts. Later methods [ 18 , 19 , 19 , 14 , 39 ] expanded to multimodal prompting by jointly optimising text and image prompts and enhancing visual-linguistic synergy. Beyond prompt-based methods, adapters [ 10 ] offer another approach by introducing learnable parameters that adjust pre-trained features without disrupting the model’s original knowledge. [ 30 ] integrates