When VLMs Meet Image Classification: Test Sets Renovation via Missing Label Identification
Zirui Pang Haosheng Tan Yuhan Pu, Zhijie Deng Zhouan Shen Keyu Hu Jiaheng Wei · 2025
Read When VLMs Meet Image Classification: Test Sets Renovation via Missing Label Identification by Zirui Pang Haosheng Tan Yuhan Pu, Zhijie Deng Zhouan Shen Keyu Hu Jiaheng Wei (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16149.txt.
Introduction
Zirui Pang and co-authors clean the answer keys of famous image classification tests using vision-language models, in a 2025 paper. Benchmark sets like ImageNet and CIFAR still carry wrong labels, and many images actually show several classes while listing one. Those missing labels quietly corrupt model rankings. The team's framework, REVEAL, combines several vision-language models with automated and crowd label-checking tools. It flags suspect labels, pools the predictions, and filters by confidence and consensus.
Human verifiers confirmed the fixes across six benchmark test sets, bringing labels much closer to human judgment. The framework outputs soft labels with likelihoods rather than blunt verdicts. An analysis of the models involved surfaces ten observations about their strengths and blind spots. The authors argue fair evaluation starts with honest test data.
Catalog record
| CREATOR | Zirui Pang Haosheng Tan Yuhan Pu, Zhijie Deng Zhouan Shen Keyu Hu Jiaheng Wei |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-16149
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
When VLMs Meet Image Classification: Test Sets Renovation via Missing Label Identification Zirui Pang Haosheng Tan Yuhan Pu, Zhijie Deng Zhouan Shen Keyu Hu Jiaheng Wei Abstract Image classification benchmark datasets such as CIFAR, MNIST, and ImageNet, serves as critical tools for model evaluation. However, despite the cleaning efforts, these datasets still suffer from pervasive noisy labels, and often contain missing labels due to the co-existing image pattern where multiple classes appear in an image sample. This results in misleading model comparisons and unfair evaluations. Existing label cleaning methods focus primarily on noisy labels, but the issue of missing labels remains largely overlooked. Motivated by these challenges, we present a comprehensive framework named REVEAL , integrating state-of-the-art pre-trained vision-language models (e.g., LLaVA, BLIP, Janus, Qwen) with advanced machine/human label curation methods (e.g., Docta, Cleanlab, MTurk), to systematically address both noisy labels and missing label detection in widely-used image classification test sets. REVEAL detects potential noisy labels and omissions, aggregates predictions from various methods, and refines label accuracy through confidence-informed predictions and consensus-based filtering. Additionally, we provide a thorough analysis of state-of-the-art vision-language models and pre-trained image classifiers, highlighting their strengths and limitations within the context of dataset renovation by revealing 10 observations . Our method effectively reveals missing labels from public datasets and provides soft-labeled results with likelihoods. Through human verifications, REVEAL significantly improves the quality of 6 benchmark test sets, highly aligning to human judgments and enabling more accurate and meaningful comparisons in image classification. 1 Introduction Noisy and Missing Labels in Image Classification Test Sets Image classification is a foundational computer vision task and numerous benchmark datasets have been established to evaluate models’ visual recognition performance. However, noisy labels are occasionally present in widely used image-classification datasets (CIFAR ( Krizhevsky09learningmultiple ) , Caltech ( griffin2007caltech ) , Imagenet ( deng2009imagenet ) , MNIST ( lecun1998gradient ) ), especially in the corresponding test sets, which can compromise the reliability of model evaluations. Similar to ( wei2021learning ; northcutt2021labelerrors ) , we categorize these errors into two types ( northcutt2021labelerrors ) : (1) noisy label errors , in which an incorrect label is assigned by referring to the ground-truth label natarajan2013learning ; liu2016classification ; wei2020optimizing ; zhu2021clusterability ; Wei2022ToSO , and (2) missing-label errors , where multiple object classes appear within an image but only a single label is provided as ground truth wei2021learning ; northcutt2021labelerrors ; wei2023aggregate . More illustrations come as follows. Caveat 1: Noisy Labels in Existing Test Benchmarks [Termed as noisy-label samples] Figure 1: Exemplary CIFAR-100 test set with noisy labels. The text below each picture denotes the CIFAR-100 original label (first row) and the cleaned label in CIFAR-100 by ( northcutt2021labelerrors ) (second row). Although benchmark test sets are intended to be precise and clean, noisy labels frequently persist, thereby undermining the integrity of evaluations. Figure 1 illustrates several instances of such errors in the CIFAR-100 test benchmark. Caveat 2: Multi-Label Issues in Existing Test Benchmarks [Termed as multi-label samples] Multi-label issues are common in image-classification benchmarks, where real-world images often contain multiple objects, but only the primary object is annotated. This leads to co-exist images, where valid labels are omitted, potentially biasing model evaluations. Figure 2 illustrates examples drawn from the CIFAR-100 test set. For instance, the first sub-figure on the left is labeled as “forest”, yet two other classes (man, boy) which also belong to valid dataset classes, are clearly visible. Consequently, an image possibly conveys additional semantic information beyond the assigned ground-truth label, potentially biasing models evaluated on these benchmarks. Figure 2: Exemplary CIFAR-100 training images with multiple labels. The text below each picture denotes the CIFAR-100 original label (first row) and the human annotated supplementary label (second row). We did not exhaust all possible labels subjectively. Figure 3: REVEAL renovation pipeline. Both the VLM-based and human-annotated methods first assign labels to each image independently. These preliminary labels are then aggregated using a weighted voting ensembling strategy. To refine the results, a score threshold is applied to filter the aggregated labels, followed by a softmax operation to compute the corresponding likelihoods. This process ultimately yields a soft-labeled output suitable for downstream tasks. Dataset Renovation via Missing Label Imputation Motivated by these observations, we propose a dataset renovation framework REVEAL for correcting noisy-label errors and addressing multi-label omissions in widely-used public image-classification test sets. Recent advances in Vision-Language Models (VLMs) offer promising solutions for this task, given their superior image recognition capabilities and robust performance in image-captioning tasks. For instance, BLIP ( li2022blip ) leverages a pretrained VLM decoder to synthesize high-quality training data, illustrating the potential of utilizing pretrained VLMs’ prior knowledge. Inspired by this insight, we introduce a comprehensive pipeline (illustrated in Figure 3 ) that employs multiple state-of-the-art VLMs to renovate diverse test benchmarks and aggregates their outputs via a weighted voting ensembling strategy. By combining annotations derived from VLMs with targeted human judgments, our approach effectively reduces label noise and generates complementary labels in a reliable and cost-effective manner. Our contributions include: \bullet Missing Labels in Benchmarks: We highlight the critical yet overlooked issue of missing labels in widely-used image classification test sets, where images often contain multiple valid classes. \bullet Systematic Analysis of VLMs: We investigate performances and limitations of VLMs in zero-shot image classification, revealing their strengths and weaknesses. \bullet Revealing New Observations: We identify potential failure modes in VLM predictions, including hallucination, repetition, and reluctance to abstain, especially in fine-grained or large label sets. Missing labels often reflect uncertainty or true multi-object presence, where semantic confusion arises in dense label spaces. \bullet Test sets Renovation with Ensembling: Building on these observations, we propose REVEAL , leveraging multiple VLMs to renovate six test benchmarks, aggregating predictions via an ensembling strategy to provide accurate soft labels and relevant annotations. REVEAL shows strong alignment with human judgment in most datasets. 2 Related Work Label Noise Detection & Missing Label Imputation Noisy and missing labels remain major obstacles in image classification. The most reliable remedy, manual re - annotation, scales poorly to today’s large datasets ( chang2017revolt ; krivosheev2020detecting ; wei2021learning ; liu2024automatic ) . Automated approaches typically involve label prediction or confusion matrix estimation ( goldberger2017training ; hendrycks2018using ; yu2019does ; wei2020combating ; tanno2019learning ; chen2019understanding ; xia2019anchor ; kumar2021constrained ) , yet they often underperform in the real-world label noise wang2019symmetric ; amid2019robust ; wang2021policy ; ma2020normalized ; liu2020peer ; cheng2020learning ; wei2021open ; liu2025human . Recent advances such as CLIP ( radford2021learning ) have enabled zero-shot prediction techniques ( zhu2023unmasking ; wei2024vision ) , while the Confident Learning framework (CL) provides a principled way to identify noisy labels and model uncertainty, particularly under class imbalance ( northcutt2021confident ; northcutt2021pervasive ; wei2023fairness ; wei2023distributionally ) . Beyond noise, the issue of missing labels is particularly pronounced in multi-label settings, where exhaustive annotation is prohibitively expensive. To mitigate this, Durand et al. ( durand2019learning ) proposed a partial binary cross-entropy loss and a curriculum learning strategy that iteratively infers missing labels using model feedback. Ben-Cohen et al. ( ben2022multi ) introduced a temporary model to estimate the distribution of unobserved labels, coupled with an asymmetric loss that emphasizes known labels. Ma et al. ( ma2022label ) preserved label semantics