Retrieval-enriched zero-shot image classification in low-resource domains
Nicola Dall’Asen Yiming Wang Enrico Fini Elisa Ricci · 2024
Read Retrieval-enriched zero-shot image classification in low-resource domains by Nicola Dall’Asen Yiming Wang Enrico Fini Elisa Ricci (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-00988.txt.
Description
Abstract Low-resource domains, characterized by scarce data and annotations, present significant challenges for language and visual understanding tasks, with the latter much under-explored in the literature. Recent advancements in Vision-Language Models (VLM) have shown promising results in high-resource domains but fall short in low-resource concepts that are under-represented ( e.g.
Catalog record
| CREATOR | Nicola Dall’Asen Yiming Wang Enrico Fini Elisa Ricci |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-00988
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Retrieval-enriched zero-shot image classification in low-resource domains Nicola Dall’Asen Yiming Wang Enrico Fini Elisa Ricci Abstract Low-resource domains, characterized by scarce data and annotations, present significant challenges for language and visual understanding tasks, with the latter much under-explored in the literature. Recent advancements in Vision-Language Models (VLM) have shown promising results in high-resource domains but fall short in low-resource concepts that are under-represented ( e.g. only a handful of images per category) in the pre-training set. We tackle the challenging task of zero-shot low-resource image classification from a novel perspective. By leveraging a retrieval-based strategy, we achieve this in a training-free fashion. Specifically, our method, named CoRE (Combination of Retrieval Enrichment), enriches the representation of both query images and class prototypes by retrieving relevant textual information from large web-crawled databases. This retrieval-based enrichment significantly boosts classification performance by incorporating the broader contextual information relevant to the specific class. We validate our method on a newly established benchmark covering diverse low-resource domains, including medical imaging, rare plants, and circuits. Our experiments demonstrate that CoRE outperforms existing state-of-the-art methods that rely on synthetic data generation and model fine-tuning. 1 Introduction Low-resource domains refer to those rare domains where the data or its annotation is truly scarce. Similarly, low-resource languages are those that have significantly less content available online Magueresse et al. (2020) with respect to other high-resource languages, like English. There exist abundant research on the topic in the context of natural language processing Ranathunga et al. (2023) ; Adams et al. (2017) ; Fadaee et al. (2017) ; Pan et al. (2017) . However, surprisingly, the vision counterpart, i.e. low-resource visual domains , is much under-explored despite the numerous practical applications. In this paper, we focus on classifying images in low resource domains, i.e. where we can find only a handful of images per category. The causes for such limited data can be various: for example, when only certain devices are capable of capturing the visual content, e.g. astronomy or medical imaging; the visual content itself is sensitive or private, e.g. due to privacy issues, or rarely appears in nature, e.g. deep ocean animals, or other long-tailed categories. Their associated annotations can also be limited due to the expertise requires, in particular for niche fields, e.g. electric design or phytology. Figure 1: Our retrieval-based solution enriches both images and textual descriptors with real-world captions which contain domains and classes . Even when the captions are generic (third row for each example), they can still restrict the focus to the correct domain. Recent large vision-language models (VLMs) have fostered a paradigm shift in image classification. Their flexibility and generalization, enabled by web-scale pre-training with text-image pairs, makes them versatile tools in many sub-fields of computer vision. Numerous works have appeared, with the objective of tuning VLMs, e.g. CLIP Radford et al. (2021) or SigLIP Zhai et al. (2023) , to address image zero-shot Jia et al. (2022) or few-shot Chowdhury et al. (2023) ; da Costa et al. (2023) image classification. However, the images involved in those studies are mostly in high-resource image domains, where there exist thousands of images on the Internet for VLMs to learn from during the web-scale pre-training Udandarao et al. (2024) . However, directly performing zero-shot classification in low-resource domains does not yield satisfactory performance due to the data scarcity in pre-training. Even supervised fine-tuning might fall short in learning the underlying data distribution due to the very limited amount of data and annotation. Among the techniques that have been explored in the pioneering work Zhang et al. (2024) , one prominent recipe is to fine-tune the VLMs on data augmented via synthetic generation ( e.g. Stable Diffusion ( Rombach et al., 2021 ) ). Despite the performance improvements, by analyzing the generated images, we observe that image generation models are also affected by the low-resource nature of the task. The generation quality is largely dependent on the noise injected on the real samples: by injecting limited noise, the synthetic images appear very similar to the original samples, being correct but not diverse, while by injecting more noise, the synthetic images diversify in appearance, but are mostly semantically incorrect and exhibit domain-specific rule violations. This is because the data distribution of rare domains is not well-represented in the generative models latent space ( Mokady et al., 2022 ; Trabucco et al., 2024 ) . Instead of generating synthetic images as data augmentation, we explore the possibility of retrieving relevant information from a textual corpus, crawled from the Internet, to enrich the data representation at inference time, as shown in Fig. 1 . It turns out that retrieval is also non-trivial in the low-resource regime as (i) pre-trained models generally under-represent the low-resource domains, thus greatly limiting the retrieval efficacy; (ii) large web-crawled databases can contain noisy or incorrect content, a problem that is more severe in low-resource domains. Thus, a careful design is required to leverage the retrieved data. In this work, we propose the first training-free and retrieval-based method, CoRE (Combination of Retrieval Enrichment), to tackle low-resource image classification. Following a VLM-based zero-shot classification paradigm, we propose to enrich the representation for both the query image and the class prototypes with textual content retrieved with different encoder backbones from large web-crawled databases. Specifically, for the query image, we employ the pre-trained image encoder from a VLM as our vision retrieval backbone. We perform image-to-text retrieval, obtaining the most relevant captions with respect to the query image. From our preliminary analysis, we observe that although the specific category ( e.g. “LED”) appears sparsely in the retrieval, its broader category (“circuit”) does occur frequently. Previous studies have empirically demonstrated that enriching the prompt with the broader concept, together with noise, can significantly boost the zero-shot recognition performance Roth et al. (2023) . We thus enrich the image embedding by combining it with the textual embedding from image-to-text retrieval. Similarly, we construct the enriched class prototypes. For each class, we form its corresponding text prompt and embed it with a pre-trained text encoder to retrieve captions that are most relevant. Then, the retrieved captions are encoded with the VLM text encoder and aggregated together with the textual embedding of the original class prompt. The final categorization is obtained by computing the cosine similarity between the enriched visual representation against the enriched textual class prototypes. To validate the effectiveness of our proposed method, we also collect a set of datasets that covers diverse low-resource domains, including medical imaging, rare plants, and circuits. CoRE can effectively improve the data representation in a training-free fashion, with a noticeable improvement in image classification performance on all the datasets, outperforming the state-of-the-art method that involves synthetic image generation and model fine-tuning. To summarize, our contributions are: • We propose the first training-free retrieval-based method CoRE for addressing zero-shot low-resource image classification; • we propose a data representation enrichment strategy for both query image and class prototypes, using the textual content retrieved from the database; • we establish a benchmark featuring zero-shot low-resource image classification, composed of representative datasets and VLM-based baselines and state-of-the-art methods; • our training-free method is effective in classifying low-resource images, outperforming competitors with training and other training-free baselines by a