Unleashing the Power of Vision-Language Models for Long-Tailed Multi-Label Visual Recognition
Wei Tang, Zuo-Zheng Wang, Kun Zhang, Tong Wei, and Min-Ling Zhang, Senior Member, IEEE · 2025
Read Unleashing the Power of Vision-Language Models for Long-Tailed Multi-Label Visual Recognition by Wei Tang, Zuo-Zheng Wang, Kun Zhang, Tong Wei, and Min-Ling Zhang, Senior Member, IEEE (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-20641.txt.
Introduction
Tang, Wang, Zhang, Wei, and Zhang rescue rare labels in multi-label image recognition. Their 2025 paper presents CapNet for long-tailed visual recognition. Real images carry several labels, but common ones dominate training data. Rare tail classes get ignored by biased models. CLIP's design favors single-label matching over multi-label tagging.
CapNet models label correlations from CLIP's own text encoder. A graph network propagates information along those correlations. Learnable soft prompts refine the label embeddings further. A distribution-balanced focal loss reweights training toward rare classes. Test-time ensembling combines predictions for better generalization. Parameter-efficient fine-tuning adapts vision to text cheaply. On VOC-LT, mean average precision reached 93.03 percent. That beat the LMPT baseline by over five points. COCO-LT results hit 76.36 percent, up to ten points ahead. Ablations on NUS-WIDE confirmed each component helps. Head classes kept their accuracy while tails improved.
Catalog record
| CREATOR | Wei Tang, Zuo-Zheng Wang, Kun Zhang, Tong Wei, and Min-Ling Zhang, Senior Member, IEEE |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-20641
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Unleashing the Power of Vision-Language Models for Long-Tailed Multi-Label Visual Recognition Wei Tang, Zuo-Zheng Wang, Kun Zhang, Tong Wei, and Min-Ling Zhang, Senior Member, IEEE Abstract Long-tailed multi-label visual recognition poses a significant challenge, as images typically contain multiple labels with highly imbalanced class distributions, leading to biased models that favor head classes while underperforming on tail classes. Recent efforts have leveraged pre-trained vision-language models, such as CLIP, alongside long-tailed learning techniques to exploit rich visual-textual priors for improved performance. However, existing methods often derive semantic inter-class relationships directly from imbalanced datasets, resulting in unreliable correlations for tail classes due to data scarcity. Moreover, CLIP’s zero-shot paradigm is optimized for single-label image-text matching, making it suboptimal for multi-label tasks. To address these issues, we propose the correlation adaptation prompt network ( CapNet ), a novel end-to-end framework that explicitly models label correlations from CLIP’s textual encoder. The framework incorporates a graph convolutional network for label-aware propagation and learnable soft prompts for refined embeddings. It utilizes a distribution-balanced Focal loss with class-aware re-weighting for optimized training under imbalance. Moreover, it improves generalization through test-time ensembling and realigns visual-textual modalities using parameter-efficient fine-tuning to avert overfitting on tail classes without compromising head class performance. Extensive experiments and ablation studies on benchmarks including VOC-LT, COCO-LT, and NUS-WIDE demonstrate that CapNet achieves substantial improvements over state-of-the-art methods, validating its effectiveness for real-world long-tailed multi-label visual recognition. I Introduction Long-tailed visual recognition is a challenging area associated with learning from highly imbalanced data, where head classes dominate with abundant samples while tail classes suffer from scarcity [ 1 , 2 , 3 , 4 ] . Recent progress in long-tailed visual recognition has mainly centered on the single-label multi-class setting. However, real-world images often encompass multiple objects and concepts, giving rise to long-tailed multi-label visual recognition (LTML), where each image is associated with multiple labels exhibiting long-tailed distributions [ 5 , 6 , 7 , 8 , 9 , 10 ] . LTML introduces additional intricacies compared to its single-label multi-class counterpart, including label co-occurrences, which create interdependencies among classes, and intra-class imbalances between positive and negative instances. \begin{overpic}[./figs/CAPN_illustration.pdf] \put(570.0,309.0){\footnotesize(a) zero-shot CLIP} \put(800.0,510.0){\tiny$p(\boldsymbol{y}|\boldsymbol{x})$} \put(930.0,480.0){\tiny 0.97} \put(802.0,450.0){\tiny 0.02} \put(800.0,420.0){\tiny 0.0004} \put(800.0,389.0){\tiny 0.0007} \put(800.0,358.0){\tiny 0.00008} \put(650.0,53.0){\footnotesize(b) {{CapNet}}} \put(800.0,250.0){\tiny$p(\boldsymbol{y}|\boldsymbol{x})$} \put(905.0,220.0){\tiny 0.78} \put(885.0,190.0){\tiny 0.56} \put(875.0,160.0){\tiny 0.54} \put(830.0,129.0){\tiny 0.21} \put(808.0,98.0){\tiny 0.02} \put(85.0,5.0){\small bottle} \put(256.0,5.0){\small chair} \put(410.0,5.0){\small dinner table} \put(700.0,5.0){\small sofa} \put(850.0,5.0){\small sheep} \end{overpic} Fig. 1: Comparison between zero-shot CLIP [ 11 ] and our proposed CapNet . Image embeddings are represented by red four-angle stars, and text embeddings by circles. The proximity indicates the higher probability that the category of text embeddings matches the image. Existing LTML methods have primarily relied on strategies like class re-sampling, loss re-weighting, and specialized architectures, typically built upon ImageNet-pre-trained convolutional neural networks (CNNs) as backbones [ 5 , 6 , 7 ] . Motivated by the emergence of vision-language models (VLMs) like contrastive language-image pre-training (CLIP) [ 11 ] , recent works such as LMPT [ 8 ] have pioneered the use of CLIP for long-tailed multi-label visual recognition, which introduces additional captions and class-specific embedding losses to learn fine-grained and class-related embeddings. Despite these advances, CLIP’s zero-shot paradigm is inherently designed for single-label image-text matching, making it suboptimal for multi-label tasks. As illustrated in Fig. 1 , zero-shot CLIP tends to prioritize the most salient class, e.g., assigning 0.97 confidence to bottle , neglecting co-occurring labels such as dinner table . Intuitively, leveraging label co-occurrences is a feasible way to mitigate long-tailed imbalances, as it enables knowledge transfer from head to tail classes. For example, the tail-class parking meters often accompany the head-class cars , while the medium-class keyboards are commonly found with the head-class laptops . Consequently, such label co-occurrences can synchronously enhance the performance of head-to-tail categories. While LMPT [ 8 ] attempts to model these dependencies using shared image-caption pairs, it suffers from noisy and incomplete semantic representations: (1) the captions are generated by models or annotated manually, introducing noise and high costs; (2) the captions inherit the long-tailed biases of the datasets, failing to capture nuanced inter-class connections, particularly for tail classes. Therefore, the model may not capture all the nuanced connections between various classes, which potentially limits the effectiveness in the long-tailed multi-label visual recognition scenarios. To address these limitations and unleash the potential of VLMs for LTML, we propose the correlation adaptation prompt network ( CapNet ), an end-to-end framework that adapts CLIP for robust LTML. Firstly, CapNet enables the computation of independent probabilities for each label, thereby achieving genuine multi-label compatibility without mutual exclusivity assumptions. Secondly, CapNet explicitly captures label correlations via a graph convolutional network (GCN) [ 12 , 13 ] , constructing a robust correlation matrix from CLIP’s textual encoder rather than noisy captions-derived priors. Thirdly, we adapt the distribution-balanced Focal loss [ 5 ] with re-balanced weights tailored for class-aware sampling [ 14 ] . Finally, we introduce a test-time ensembling strategy that aggregates diverse predictions to enhance generalization, and a parameter-efficient fine-tuning (PEFT) method, which learns a sparse set of task-specific parameters to prevent overfitting, especially on tail classes. To verify the effectiveness of our proposed CapNet , extensive evaluations on benchmarks like VOC-LT, COCO-LT, and NUS-WIDE [ 5 , 15 , 16 ] demonstrate that CapNet achieves substantial performance gains over state-of-the-art approaches, highlighting its practical utility. Our contributions can be summarized as follows: • We propose CapNet , an end-to-end framework that adapts CLIP for LTML, incorporating independent label predictions, a GCN-based module for robust label correlations, and a distribution-balanced Focal loss with class-aware re-balancing. • We introduce a test-time ensembling strategy and a parameter-efficient fine-tuning approach with sparse task-specific parameters to enhance generalization and mitigate overfitting, especially for tail classes. • Through comprehensive experiments on benchmarks such as VOC-LT, COCO-LT, and NUS-WIDE, we demonstrate that CapNet significantly outperforms state-of-the-art methods, validating its effectiveness. The remainder of this paper is organized as follows: Section II reviews related work. Section III elucidates the preliminaries and CapNet ’s technical details. Section IV reports experimental results, and Section V concludes the paper. II Related Work II-A Multi-Label Visual Recognition Multi-label visual recognition presents a core challenge in machine learning [ 17 , 18 ] . In contrast to multi-class recognition, where each instance receives a single label, multi-label classification assigns multiple labels to an instance. Early approaches relied on independent binary classifiers [ 19 , 20 ] or k-nearest neighbors [ 21 ] , often overlooking label correlations. Research has demonstrated that exploiting these correlations improves model generalization. To this end, Wang et al. [ 22 ] integrated convolutional and recurrent neural networks to model label dependencies explicitly. Subsequent methods adopted graph neural networks for embedding such correlations [ 12 , 23 , 24 , 25 ] . Nevertheless, these approaches heavily depend on the quality of label supervision. To alleviate annotation burdens, multi-label learning with partial labels emerged, providing only label subsets. Huynh et al. [ 26 ] inferred missing labels via mini-batch co-occurrences, while Chen et al. [ 27 ] incorporated semantic correlations. These techniques, however, fall short in imbalanced distributions. Recent advancements leverage vision-language models to align visual and textual features in multi-label tasks [ 28 , 29 , 13