Tuning the Right Foundation Models is What you Need for Partial Label Learning
Kuang He, Wei Tang, Tong Wei, Min-Ling Zhang · 2025
Read Tuning the Right Foundation Models is What you Need for Partial Label Learning by Kuang He, Wei Tang, Tong Wei, Min-Ling Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05027.txt.
Introduction
Kuang He and co-authors test whether big pretrained models rescue learning from fuzzy labels in this 2025 paper. In partial label learning, each training example carries several candidate labels, only one correct. Older methods train small networks to slowly sort out the truth.
The team benchmarked eleven foundation models across thirteen methods, eight datasets, and three ambiguity settings. Adding foundation models lifted every method, yet the methods then scored much alike, and results hinged on which big model was picked and how it was tuned. Their fine-tuning scheme, PartialCLIP, uses text-embedding starts and zero-shot filtering of candidates to good effect. The authors say future gains lie in adaptation, not more label-cleaning tricks.
Catalog record
| CREATOR | Kuang He, Wei Tang, Tong Wei, Min-Ling Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-05027
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Tuning the Right Foundation Models is What you Need for Partial Label Learning Kuang He, Wei Tang, Tong Wei, Min-Ling Zhang Abstract Partial label learning (PLL) seeks to train generalizable classifiers from datasets with inexact supervision, a common challenge in real-world applications. Existing studies have developed numerous approaches to progressively refine and recover ground-truth labels by training convolutional neural networks. However, limited attention has been given to foundation models that offer transferrable representations. In this work, we empirically conduct comprehensive evaluations of 11 foundation models across 13 PLL approaches on 8 benchmark datasets under 3 PLL scenarios. We further propose PartialCLIP, an efficient fine-tuning framework for foundation models in PLL. Our findings reveal that current PLL approaches tend to 1) achieve significant performance gains when using foundation models, 2) exhibit remarkably similar performance to each other, 3) maintain stable performance across varying ambiguity levels, while 4) are susceptible to foundation model selection and adaptation strategies. Additionally, we demonstrate the efficacy of text-embedding classifier initialization and effective candidate label filtering using zero-shot CLIP. Our experimental results and analysis underscore the limitations of current PLL approaches and provide valuable insights for developing more generalizable PLL models. The source code can be found at https://github.com/SEU-hk/PartialCLIP . 1 Introduction Partial label learning (PLL) is an important weakly supervised learning framework and has been studied a lot in the past decade [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 ] . PLL aims to learn a classifier from datasets with inexact supervision in the label space, i.e., each training instance is associated with a set of candidate labels among which only one is correct. This framework alleviates the burden of precise data annotation, making it particularly valuable in scenarios where obtaining exact labels is costly or impractical. Therefore, PLL has been extensively studied across various real-world domains such as image annotation [ 9 ] , web mining [ 10 ] , ecoinformatics [ 11 ] , and natural language processing [ 12 ] . The core challenge in PLL lies in accurately identifying the ground-truth label from the candidate label set. Existing PLL methods can be broadly categorized into two groups: average-based and identification-based methods. The average-based methods [ 13 , 5 ] treat each candidate label equally by averaging the model outputs corresponding to all candidate labels. By contrast, identification-based methods [ 14 , 15 ] progressively identify the ground truth label from the candidate label set through iterative refinement. Recently, deep neural network techniques have further enhanced PLL performance, such as PRODEN [ 16 ] and CRDPLL [ 17 ] . Despite these advancements, standard PLL (ST-PLL) methods often underperform in real-world scenarios, particularly in long-tailed PLL (LT-PLL) [ 18 ] and instance-dependent PLL (ID-PLL) [ 19 ] settings. This suboptimal performance can be attributed to the fact that ST-PLL assumes that the number of instances across all categories is uniform, and the false-positive labels in the candidate label sets are often generated randomly. Moreover, prevailing PLL methods predominantly employ convolutional networks such as ResNet [ 20 ] . Training these models typically requires 200–1,000 epochs to reach convergence, involving extensive parameter updates of the entire model. This process imposes substantial demands on computation and memory. Even with such investments, the quality of learned representations often degrades in highly ambiguous or imbalanced regimes, resulting in subpar classification performance. In this work, we explore the efficacy of fine-tuning existing open-source foundation models on PLL benchmarks. Specifically, we empirically assess the classification performance across three PLL scenarios, i.e., ST-PLL, LT-PLL, and ID-PLL, with 11 models for 13 PLL approaches on 8 datasets. To facilitate comprehensive evaluations, we introduce PartialCLIP, a unified fine-tuning framework tailored to PLL. Experimental results demonstrate that PartialCLIP significantly enhances the performance of existing PLL approaches with markedly fewer training epochs (refer to Figure 1a ). Furthermore, our findings reveal that current PLL approaches: • exhibit remarkably similar performance to each other under the standard PLL and ID-PLL scenarios (see Figure 1b ), indicating the effectiveness of transferred representations. • maintain superior and stable performance across varying levels of label ambiguity, even under high ambiguity conditions (see Figure 1b ). • are susceptible to the choice of foundation models (see Figure 1c ) and fine-tuning methods (see Table 5 ), underscoring the importance of selecting appropriate models. Additionally, we explore the vision-language alignment capability of CLIP in two aspects: 1) classifier initialization and 2) candidate label filtering. For classifier initialization, we employ class-specific embeddings derived from CLIP’s text encoder, prompted with ‘‘a photo of a [CLASS]’’ . This approach leverages CLIP’s semantic understanding to reduce reliance on extensive supervised data during training. For candidate label filtering , we filter out semantically irrelevant candidate labels according to the cosine similarities between image embeddings and textual prompts. Remarkably, pruning over 50\% of candidate labels does not degrade performance and even yields improvements. This finding shifts the emphasis from the quantity of candidate labels to the quality - driven selection. (a) CIFAR100 ( \eta=0.05 ) (b) CIFAR100 (c) Benchmark evaluation Figure 1: (a) Performance comparison between ST-PLL approaches and their PartialCLIP-enhanced variants in terms of test accuracy and the number of training epochs. Marker sizes represent the number of learnable parameters in each model. (b) The impact of partial rate on model accuracy across different ST-PLL methods. (c) Evaluation of various foundation models under three scenarios with CRDPLL (ST-PLL), RECORDS (LT-PLL), and POP (ID-PLL) serving as representative methods. In summary, our contributions are as follows. 1) We propose an early foundation model benchmark for PLL, evaluated on eight datasets across three PLL settings; 2) We present two ways of leveraging vision-language model alignment to address the challenges of learning from inexact labels; 3) We identify three new findings of tuning foundation model in PLL to guide the future research. 2 Related Works Partial Label Learning ST-PLL methods can be roughly divided into two categories, i.e., the averaged-based strategy (ABS) and the identification-based strategy (IBS). ABS excludes non-candidate labels and treats every candidate label equally. It averages the model output of all candidate labels for prediction [ 13 , 5 ] . IBS views the ground truth label as a latent variable and gradually eliminates the label ambiguity during the training process [ 14 , 15 ] . PRODEN [ 16 ] suggested a strategy to progressively uncover the ground-truth label. CC [ 21 ] devised label disambiguation approaches that are provably risk-consistent and classifier-consistent from a mathematical perspective. LWS [ 22 ] introduced a suite of leveraged weighted loss functions. CAVL [ 23 ] utilizes the class activation value to guide the model in selecting the ground-truth label from the candidate label set during training. ABS-MAE and ABS-GCE [ 24 ] revisited the average-based strategy methods. Following MoCo [ 25 ] , PiCO [ 26 ] incorporated the widely used contrastive loss into PLL. CRDPLL [ 17 ] applied consistency regularization [ 27 ] within the candidate label sets. PAPI [ 28 ] computed similarity scores between feature prototypes and instance embeddings. CROSEL [ 29 ] employed two models to sift out reliable samples from the dataset through cross-selection for the training stage. Long-Tailed Partial Label Learning