COBRA: COmBinatorial Retrieval Augmentation for Few-Shot Adaptation
Arnav M. Das, Gantavya Bhatt, Lilly Kumari, Sahil Verma, Jeff Bilmes · 2024
Read COBRA: COmBinatorial Retrieval Augmentation for Few-Shot Adaptation by Arnav M. Das, Gantavya Bhatt, Lilly Kumari, Sahil Verma, Jeff Bilmes (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-17684.txt.
Introduction
When labeled data are scarce, fetching extra examples from a huge pool can lift a model's performance. Older retrieval methods pick the examples most similar to the target task, but those picks often repeat each other. The authors show that these selection strategies fit inside a wider family of functions called combinatorial mutual information measures. Their method, COBRA, uses a different measure from that family, one that rewards both similarity to the target and diversity within the chosen set.
COBRA is tested on image classification tasks with several few-shot learning methods, drawing extra data from the LAION-2B pool. It beats earlier retrieval approaches across these tests. The extra computing cost of the smarter selection is small. The authors argue that treating retrieval as a balance of similarity and diversity is the key step that nearest-neighbor methods miss.
Catalog record
| CREATOR | Arnav M. Das, Gantavya Bhatt, Lilly Kumari, Sahil Verma, Jeff Bilmes |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-17684
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
COBRA: COmBinatorial Retrieval Augmentation for Few-Shot Adaptation Arnav M. Das, Gantavya Bhatt, Lilly Kumari, Sahil Verma, Jeff Bilmes Abstract Retrieval augmentation, the practice of retrieving additional data from large auxiliary pools, has emerged as an effective technique for enhancing model performance in the low-data regime. Prior approaches have employed only nearest-neighbor based strategies for data selection, which retrieve auxiliary samples with high similarity to instances in the target task. However, these approaches are prone to selecting highly redundant samples, since they fail to incorporate any notion of diversity. In our work, we first demonstrate that data selection strategies used in prior retrieval-augmented few-shot adaptation settings can be generalized using a class of functions known as Combinatorial Mutual Information (CMI) measures. We then propose COBRA (COmBinatorial Retrieval Augmentation), which employs an alternative CMI measure that considers both diversity and similarity to a target dataset. COBRA consistently outperforms previous retrieval approaches across image classification tasks and few-shot learning techniques when used to retrieve samples from LAION-2B. COBRA introduces negligible computational overhead to the cost of retrieval while providing significant gains in downstream model performance. 1 Introduction Figure 1 : COBRA uses the target pool to select diverse and relevant samples from a large web-scale auxiliary pool. The retrieved data and the target data are then used to train a few-shot learner with a CLIP backbone. With the emergence of web-scale data sources, retrieval has become an immensely popular technique to improve model performance. Retrieval augmented generation (RAG) has achieved much success in natural language, where additional data is retrieved from auxiliary sources to supplement model knowledge and guide the output of large language models (LLMs) at inference time [ 54 , 41 , 85 , 75 ] . More recently, retrieval has been leveraged to select additional weakly labeled training samples from external image-caption data sources that are relevant to a target image classification task [ 58 , 100 , 94 ] . This strategy enables practitioners to train performant models on domains where labeled datasets are scarce and small, without incurring any extra labeling costs. Figure 2 : COBRA retrieves samples from LAION that are diverse and relevant to the target dataset. In contrast, Sim-Score retrieves redundant samples and sometimes even exact duplicates (shown in red boxes). Additional visualizations are presented in Section 20 . Web-scale datasets are highly heterogeneous and often contain massive amounts of data that are irrelevant to a specific downstream task [ 33 , 80 ] . Therefore, it is natural for retrieval strategies to assess the relevance of each sample from an auxiliary retrieval pool using some notion of similarity to the target dataset. To this end, past work typically considers nearest-neighbor based retrieval strategies, which score each auxiliary sample from the retrieval pool based on its similarity to a class of images in the target task, and retrieve the samples with the highest scores [ 94 , 58 , 100 , 107 , 37 ] . However, independently scoring the auxiliary data in a sample-wise manner neglects notions of diversity of the retrieved set and may induce redundancy. Diversity has been shown to be a key ingredient in data subset selection for many different tasks [ 7 , 9 , 2 , 81 ] , but has not been employed in the context of retrieval augmentation. In this work, we seek to answer the following question: Can we improve the effectiveness of retrieval augmentation by using a diversity-aware data selection strategy? We propose the use of a class of functions called Combinatorial Mutual Information (CMI) functions for retrieval. CMI functions capture the similarity between sets [ 9 , 38 ] and have been employed in various contexts that require targeted data subset selection [ 38 , 52 , 50 ] . We demonstrate that CMI measures are inherently well-suited for retrieval augmentation by showing that the underlying objective used in nearest-neighbor based retrieval strategies are instances of this function class. To address the deficiencies of nearest-neighbor based retrieval, we propose COmBinatorial Retrieval Augmentation (COBRA) which uses an alternative CMI function that retrieves a set of similar and diverse samples. Rather than independently scoring each auxiliary sample, COBRA evaluates sets and can model interactions between selected samples. As shown in Figure 2 , this property allows COBRA to avoid retrieving semantically redundant images and select a set of images that well represent all samples in the target task. Crucially, the COBRA objective is submodular and therefore can be efficiently optimized with a constant factor approximation guarantee [ 65 ] . In our work, we evaluate COBRA as a retrieval augmentation strategy within the few-shot setting for image classification, where a target dataset has only a few labeled images (shots) per class and the objective is to improve model performance by retrieving samples from an auxiliary pool. Our experimental pipeline (shown in Figure 1 ) involves: (1) utilizing a retrieval strategy to select weakly labeled samples from the auxiliary image-caption dataset, such as LAION-2B, to augment a small image classification dataset and (2) using a few-shot adaptation strategy such as [ 106 , 35 , 110 , 109 , 107 ] to adapt CLIP [ 73 ] with both the labeled and retrieved samples. Our findings highlight that COBRA consistently outperforms alternative previous retrieval strategies across various downstream/target datasets and few-shot adaptation methods, establishing the necessity of diversity in the data selection strategy. Overall, we make the following contributions: • We propose the use of combinatorial mutual information (CMI) functions in retrieval augmentation, and demonstrate that most existing retrieval strategies are a particular instance of CMI functions. • We introduce COmBinatorial Retrieval Augmentation (COBRA), an alternative CMI measure that considers similarity as well as the previously overlooked notion of diversity. • We demonstrate the efficacy of COBRA in the few-shot setting and show that it consistently outperforms previous retrieval techniques across target datasets and models. 2 Background Retrieval Augmentation We first describe the problem of retrieval augmentation for discriminative classification where we are given a target dataset ( \mathcal{D}^{\text{tar}} ) and an auxiliary dataset ( \mathcal{D}^{\text{aux}} ). \mathcal{D}^{\text{tar}} consists of labeled samples and is much smaller in scale compared to \mathcal{D}^{\text{aux}} , which is generally a pool of heterogeneous data that can come from any arbitrary source and in general has a distinct label space from \mathcal{D}^{\text{tar}} . For example, in vision applications where the target dataset consists of images and corresponding labels, the auxiliary dataset may consist of images with natural language descriptions which may or may not be semantically relevant to \mathcal{D}^{\text{tar}} . Formally, \mathcal{D}^{\text{tar}}\triangleq\{(x_{i},y_{i})\mid x_{i}\in\mathcal{X}\;,y_{i}\in\mathcal{Y}\}_{i=1}^{m} where we define \mathcal{X} as the domain of input (images) and \mathcal{Y}=\{1,2,\ldots,C\} as the domain of labels. Let \mathcal{D}^{\text{aux}}\triangleq\{z_{i}\mid z_{i}\in\mathcal{Z}\}_{i=1}^{n} where we define \mathcal{Z} as the domain of examples in auxiliary dataset, which are images with associated metadata. Note that z_{i}\in\mathcal{D}^{\text{aux}} are pairs but their labels need not conform to \mathcal{Y} . For notational simplicity, we jointly index \mathcal{D}^{\text{tar}} and \mathcal{D}^{\text{aux}} using V\triangleq\{1,2,\ldots,m,m+1,m+2,\ldots,m+n\}=[m+n] , and define V^{\text{tar}}\triangleq\{1,2,\ldots,m\} and V^{\text{aux}}\triangleq\{m+1,m+2,\ldots,m+n\} . For \mathcal{D}^{\text{aux}} , given any A\subseteq V^{\text{aux}} , we define \mathcal{D}^{\text{aux}}_{A}=\{z_{j}|\,j\in A\} . The goal of retrieval augmentation in our setting is to find A\subseteq V^{\text{aux}} , such that a classifier trained on \mathcal{D}^{\text{tar}}\cup\mathcal{D}^{\text{aux}}_{A} is more performant than the