Modeling Multi-modal Cross-interaction for Multi-label Few-shot Image Classification Based on Local Feature Selection

Kun Yan, Zied Bouraoui, Fangyun Wei, Chang Xu, Ping Wang, Shoaib Jameel, Steven Schockaert · 2024

Read Modeling Multi-modal Cross-interaction for Multi-label Few-shot Image Classification Based on Local Feature Selection by Kun Yan, Zied Bouraoui, Fangyun Wei, Chang Xu, Ping Wang, Shoaib Jameel, Steven Schockaert (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-13732.txt.

Introduction

Kun Yan and co-authors published a research paper in 2024 on tagging images with several labels when only a few training examples per label exist. In multi-label photos, each label lives in a different region, and prototypes built from scarce, noisy patches go wrong easily. Their strategy refines label prototypes in stages.

Word embeddings seed each prototype with prior meaning. A loss-change test then picks the training patches most representative of each label. Finally, a cross-interaction step fuses those patches with the seed prototypes, letting text and vision correct each other. On four benchmarks, from everyday photos to product images, the model substantially beat the state of the art. The authors stress gradual refinement as the key trick.

Catalog record

CREATORKun Yan, Zied Bouraoui, Fangyun Wei, Chang Xu, Ping Wang, Shoaib Jameel, Steven Schockaert
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-13732

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Modeling Multi-modal Cross-interaction for Multi-label Few-shot Image Classification Based on Local Feature Selection Kun Yan, Zied Bouraoui, Fangyun Wei, Chang Xu, Ping Wang, Shoaib Jameel, Steven Schockaert Abstract . The aim of multi-label few-shot image classification (ML-FSIC) is to assign semantic labels to images, in settings where only a small number of training examples are available for each label. A key feature of the multi-label setting is that an image often has several labels, which typically refer to objects appearing in different regions of the image. When estimating label prototypes, in a metric-based setting, it is thus important to determine which regions are relevant for which labels, but the limited amount of training data and the noisy nature of local features make this highly challenging. As a solution, we propose a strategy in which label prototypes are gradually refined. First, we initialize the prototypes using word embeddings, which allows us to leverage prior knowledge about the meaning of the labels. Second, taking advantage of these initial prototypes, we then use a Loss Change Measurement (LCM) strategy to select the local features from the training images (i.e. the support set) that are most likely to be representative of a given label. Third, we construct the final prototype of the label by aggregating these representative local features using a multi-modal cross-interaction mechanism, which again relies on the initial word embedding-based prototypes. Experiments on COCO, PASCAL VOC, NUS-WIDE, and iMaterialist show that our model substantially improves the current state-of-the-art. DOI: XXXXXXX.XXXXXXX Journal: JACM Volume: 37 4 111 8 CCS: Computing methodologies Computer vision 1. Introduction Multi-label image classification (ML-IC) has received considerable attention in recent years ( Wang et al., 2016 ; Chen et al., 2019 ; Wang et al., 2017 ; Yazici et al., 2020 ) . This task aims to assign descriptive labels to images, where images can have multiple labels. Each label typically describes an object that is shown in the image. As a baseline strategy, we can simply train a binary classifier for each label, to predict the presence of a given type of object. To improve on this strategy, various approaches have been proposed to exploit label dependencies, e.g. taking advantage of the fact that the presence of one label makes the presence of another label more (or less) likely ( Wang et al., 2016 ; Yazici et al., 2020 ; Chen et al., 2018 ; Chen et al., 2019 ) . In this paper, we are specifically interested in the few-shot setting, i.e. the case where only a limited number of training images are available for the labels (e.g., 1 or 5 images per label). Clearly, in this setting, we cannot rely on standard strategies for training binary image classifiers, nor on label co-occurrence statistics. The challenge of few-shot image classification (FSIC) has garnered significant attention, particularly in the single-label (SL) setting ( Finn et al., 2017 ; Nichol et al., 2018 ; Snell et al., 2017 ; Vinyals et al., 2016 ; Yan et al., 2021c ; Dai et al., 2023 ; Zhang et al., 2024 ) . Metric-based approaches hold a prominent position in SL-FSIC due to their strong performance and fast adaptability to new categories. However, it is worth noting that such methods cannot be directly extended to the multi-label setting. Metric-based approaches usually involve learning a prototype for each image category and subsequently assigning images to the category whose prototype is closest. These prototypes are typically derived by averaging a representation of the training images. In the seminal ProtoNet model ( Snell et al., 2017 ) , for instance, prototypes are simply defined as the average of the global feature maps of the available training examples. This strategy crucially relies on the assumption that most of the image is somehow relevant to its category. In the multi-label setting, such an assumption is highly questionable, given that different labels tend to refer to different parts of the image, as illustrated in Fig. 1 . For instance, suppose the figure on the right is the only training example with the label bus . If we simply use the global feature map of this image as the corresponding prototype, the model is likely to mistakenly assume that other images of cityscapes should also be assigned this label. To obtain a high-quality prototype, it is thus imperative that we can estimate which regions of the image correspond to a given label. Figure 1. In the 1-shot single-label setting, a given training image can be interpreted as a prototype for the considered label (left). In the multi-label setting, labels are related to different regions of the image, and these regions need to be identified before meaningful prototypes can be obtained (right). In this paper, we propose a novel metric-based model for multi-label few-shot image classification (ML-FSIC). Given the aforementioned concerns, we need a strategy that is based on local image features, allowing us to focus on those parts of the training images that are most likely to be relevant. However, as we may only have a single training example for some labels, we cannot implement such a strategy without some kind of prior knowledge about the meaning of the labels. Therefore, we rely on word embeddings, such as those from the GloVe model ( Pennington et al., 2014 ) , to provide us with this prior knowledge. Some previous works for the single-label setting have already relied on word embeddings for inferring prototypes directly ( Xing et al., 2019 ; Yan et al., 2021a ) , but as the resulting prototypes are inevitably noisy, they are normally used in combination with prototypes that are derived from visual features. Unfortunately, such visual prototypes are hard to obtain in the ML-FSIC setting. For this reason, we adopt a different approach to efficiently leverage word embeddings for ML-FSIC. We first construct initial prototypes of the labels based on their word embeddings. Specifically, we introduce a cross-modality loss to learn a joint embedding space, in which the representations of words are aligned with the corresponding visual features. The resulting word representations are too noisy to be used directly. To address this, we propose a multi-modal cross-interaction strategy guided by these noisy prototypes. Concretely, we develop multiple ways of interaction between word embeddings and visual features, including cross-modal attention and cross-modal dynamic convolution, to refine the generation of the final prototype. As an important advantage, this strategy can be applied to previously unseen labels without the need for any fine-tuning of the model. We will refer to the model described above as our Base model. As a further improvement of this base model, we propose a strategy for identifying the most representative local features prior to prototype generation. To achieve this, we propose a loss change measurement module (LCM), drawing inspiration from ( You et al., 2019 ) , to improve the selection of local features by assessing which features exert the greatest influence on the loss function. By filtering noisy local features based on this module, we end up with higher-quality prototypes, but the resulting model has the drawback that it needs to be fine-tuned before it can be applied to unseen labels. The above-described model will be referred to as the

Read the complete text