Few-shot Classification as Multi-instance Verification: Effective Backbone-agnostic Transfer across Domains

Xin Xu, Eibe Frank, Geoffrey Holmes · 2025

Read Few-shot Classification as Multi-instance Verification: Effective Backbone-agnostic Transfer across Domains by Xin Xu, Eibe Frank, Geoffrey Holmes (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-00401.txt.

Introduction

Xin Xu, Eibe Frank, and Geoffrey Holmes recast a hard learning task: sorting things into classes from just a few shown cases. Their paper, from two thousand twenty-five, works when the base model cannot be retrained.

In real use, the part that pulls features from data is often a fixed black box. The team turns the few-case task into a run of checks that ask if cases match. On that base they build a small add-on head that fits any base and costs little to run. Trained on a few cases from a new field, the head scored near the best known ways, at far lower cost. Tests span a broad set of image tasks across fields.

Catalog record

CREATORXin Xu, Eibe Frank, Geoffrey Holmes
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE91 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-00401

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Few-shot Classification as Multi-instance Verification: Effective Backbone-agnostic Transfer across Domains Xin Xu, Eibe Frank, Geoffrey Holmes Abstract We investigate cross-domain few-shot learning under the constraint that fine-tuning of backbones (i.e., feature extractors) is impossible or infeasible—a scenario that is increasingly common in practical use cases. Handling the low-quality and static embeddings produced by frozen, “black-box” backbones leads to a problem representation of few-shot classification as a series of multiple instance verification (MIV) tasks. Inspired by this representation, we introduce a novel approach to few-shot domain adaptation, named the “MIV-head”, akin to a classification head that is agnostic to any pretrained backbone and computationally efficient. The core components designed for the MIV-head, when trained on few-shot data from a target domain, collectively yield strong performance on test data from that domain. Importantly, it does so without fine-tuning the backbone, and within the “meta-testing” phase. Experimenting under various settings and on an extension of the Meta-dataset benchmark for cross-domain few-shot image classification, using representative off-the-shelf convolutional neural network and vision transformer backbones pretrained on ImageNet1K, we show that the MIV-head achieves highly competitive accuracy when compared to state-of-the-art “adapter” (or “partially fine-tuning”) methods applied to the same backbones, while incurring substantially lower adaptation cost. We also find well-known “classification head” approaches lag far behind in terms of accuracy. Ablation study empirically justifies the core components of our approach. We share our code at https://github.com/xxweka/MIV-head . Keywords Cross-Domain Few-Shot Learning, Multiple Instance Learning, Image Classification, Transfer Learning, Backbone-Agnostic Domain Adaptation 1 Introduction With the emerging popularity of cross-domain few-shot learning (CDFSL), typically in image classification [ 9 , 61 , 22 , 20 , 42 ] , numerous approaches have been proposed to tackle the key challenge of how to effectively transfer knowledge from the source domain(s) to yield an accurate classifier for the target domain using few-shot data. A setup that is particularly relevant in practical applications is to use a generic dataset such as ImageNet1K (ILSVRC-2012 [ 13 , 53 ] , henceforth referred to as ILSVRC) as the source domain, and a small set of labeled data available for target domain (henceforth ‘‘few-shot support set’’), to train a classifier. The task is to learn to classify new examples in the target domain, called ‘‘queries’’, based on the information in the labeled support set and the source domain. In practice, the latter is often given in the form of a publicly available feature extractor (or ‘‘backbone’’) that has been pretrained on ImageNet1K. This is the scenario we consider in this paper. 1 1 1 We focus on the “cross-domain” setting in FSL, not the “in-domain” scenario used by some literature ( [ 65 ] ), where the source and target domains comprise different sets of classes from the same dataset. Figure 1: Few-shot classification (FSC) represented as a series of multi-instance verification (MIV) tasks. The upper panel illustrates a standard MIV task, where a target bag and a query are paired for a binary classification. The lower panel shows that FSC can be represented as a series of MIV tasks: the support set forms a set of target bags, with each class as a bag. All the bags (or classes) are paired with a query, forming a series of MIV tasks, to solve a multi-class classification. The state-of-the-art (SOTA) in the CDFSL literature has been predominantly achieved by approaches fine-tuning the backbone, either fully or partially—they modify backbone models’ weights and/or architecture based on the new target domain. One such approach is called “adapter” approach. It adds adapters with learnable parameters inside the backbone and thus is able to adjust feature vectors for domain adaptation ( [ 62 , 39 , 70 , 8 , 47 , 56 ] . According to recent comparisons [ 42 , 4 ] ), two of the best-performing CDFSL methods are adapter methods, namely “Task-Specific Adapters” (TSA, [ 39 ] ) and “efficient Transformer Tuning” (eTT, [ 70 ] ). These are designed for different families of backbones: convolutional neural networks (CNNs) and vision transformers (ViTs), respectively. Impressive classification accuracy has been obtained by them, exceeding that of other domain-adaptation methods (excluding carefully configured full fine-tuning method like PMF in [ 29 ] , which is generally impractical because it is too computationally costly). However, adapter methods, and more generally methods that perform some form of “backbone fine-tuning”, have several critical drawbacks. First, they are unable to cope with situations where the backbone is non-modifiable or even unknown . This is a practically relevant constraint, important enough to merit attention in the research community for several reasons: One, there exists a plethora of off-the-shelf, pretrained “foundation models” that serve their users in a “black-box” manner—with frozen weights and architecture—producing outputs through Cloud-based API; Two, there is demand arising from the prevalence of vector databases that typically store static embeddings without details of underlying feature extractors; Three, even if trainable, the surging popularity of “large models” makes the backbone increasingly more difficult to fine-tune. The second drawback of “fine-tuning” approaches is that they are inflexible regarding the choice of backbone, and their application to commonly used backbones can sometimes be challenging: we found that adapter methods frequently lead to out-of-memory (OOM) errors ( cf. Section 4 ). This inflexibility is particularly problematic considering the fundamental importance of the choice of backbone ( [ 59 , 29 ] ) and off-the-shelf models available as candidate backbones ( [ 42 ] ). Finally, they are computationally expensive: the fine-tuning process needs to pass through the backbone, both forward and backward, at every step to update the adapters’ or backbones’ parameters—thus it tends to be very slow . Last block of backbone Second last block of backbone (a) MIV-head (ours) (b) TSA Figure 3: Embedding visualizations with t-SNE of the support set (circles), prototype (squares) and query (star) produced by the MIV-head ( 3(a) ) and TSA ( 3(b) ), based on an off-the-shelf Resnet-50 backbone and the same episode from the Aircraft dataset. The left and right panels of 3(a) are visualizations of embeddings from the last and the second last block of the backbone, respectively, in the MIV-head. All embeddings are colored according to their class labels, as specified by the legend. Our goal is to handle black-box feature extractors in a lightweight, versatile, backbone-agnostic fashion, and to achieve accuracy comparable to the SOTA established by adapter methods. This is a challenging problem. When applying traditional classification heads, typically with learnable parameters, to a backbone’s output, they fail to provide competitive performance close to the SOTA, as we demonstrate in Appendix C.1.3 , despite being API-friendly and computationally efficient. When the backbone is a black-box, the embeddings it produces are not adapted to the target domain, and hence their correspondence to the (unseen) class labels becomes ambiguous, sometimes highly unreliable. To alleviate this issue, unlike existing methods that implicitly treat embeddings of all support-set images as equally connected to their class labels, we explicitly model the support samples pertaining to a class as a bag of instances with unknown relevance and let them compete for a bag-level representation given the query. Inspired by the “multiple instance verification” (MIV) problem ( [ 71 ] ), which involves comparing

Read the complete text