DRUM: Learning D emonstration R etriever for Large M U lti-modal M odels
Ellen Yi-Ge Jiechao Gao Wei Han Wei Zhu · 2024
Read DRUM: Learning D emonstration R etriever for Large M U lti-modal M odels by Ellen Yi-Ge Jiechao Gao Wei Han Wei Zhu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-07619.txt.
Introduction
Ellen Yi-Ge and co-authors present DRUM in a 2024 paper on teaching big vision-language models by example. Such models learn new tasks from a few shown examples, a skill called in-context learning. The catch is picking the right examples: fixed sets or plain similarity search often miss what a given model actually needs. DRUM trains the retriever itself to serve the model.
The framework first joins image and text embeddings to improve search. Candidate examples are then re-ranked using the big model's own feedback, and a list-wise loss teaches the retriever from those rankings. A mining loop keeps finding harder training cases for the retriever. Across seven data sets and three task types, the tuned retriever lifted the big model's example-based performance clearly.
Catalog record
| CREATOR | Ellen Yi-Ge Jiechao Gao Wei Han Wei Zhu |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-07619
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DRUM: Learning D emonstration R etriever for Large M U lti-modal M odels Ellen Yi-Ge Jiechao Gao Wei Han Wei Zhu Abstract Recently, large language models (LLMs) have demonstrated impressive capabilities in dealing with new tasks with the help of in-context learning (ICL). In the study of Large Vision-Language Models (LVLMs), when implementing ICL, researchers usually adopts the naive strategies like fixed demonstrations across different samples, or selecting demonstrations directly via a visual-language embedding model. These methods does not guarantee the configured demonstrations fit the need of the LVLMs. To address this issue, we now propose a novel framework, d emonstration r etriever for large m u lti-modal m odel (DRUM), which fine-tunes the visual-language embedding model to better meet the LVLM’s needs. First, we discuss the retrieval strategies for a visual-language task, assuming an embedding model is given. And we propose to concate the image and text embeddings to enhance the retrieval performance. Second, we propose to re-rank the demonstrations retrieved by the embedding model via the LVLM’s feedbacks, and calculate a list-wise ranking loss for training the embedding model. Third, we propose an iterative demonstration mining strategy to improve the training of the embedding model. Through extensive experiments on 3 types of visual-language tasks, 7 benchmark datasets, our DRUM framework is proven to be effective in boosting the LVLM’s in-context learning performance via retrieving more proper demonstrations. 1 Introduction In-context learning (ICL) is a simple yet important learning paradigm that given a few input-output pairs (demonstrations), a model can learn to conduct predictions on a new task it never sees before. ICL is a type of emergent capability observed in large-scale pre-trained models Wei et al. 2022 . It is first observed by GPT-3 Brown et al. 2020 , and draws the attention of the whole community of artificial intelligence. And a large branch of literature have shown that large language models (LLMs) have impressive ICL capabilities across a wide range natural language processing (NLP) tasks. ICL is essential for applications, since it can quickly adapt the large pretrained models to a novel task, or a task with personalized needs, with only a few demonstrations. No fine-tuning is needed and the model need not to be deployed again. Recently, large vision-language models (LVLMs) are being rapidly developed, and its ICL capabilities are also being investigated Alayrac et al. 2022 . The LVLMs like Flamingo Alayrac et al. 2022 and Qwen-VL Bai et al. 2023 have demonstrated impressive ICL capabilities on the visual question answering (VQA), few-shot image classification (ImageCLS), and image captioning (ImageCAP) tasks. However, when implementing ICL for LVLMs, researchers usually adopts the naive strategies like fixed demonstrations or demonstrations ranked by a pre-trained visiion-language embedding model. These strategies are sub-optimal, since they do not incorporate the LVLMs’ feedbacks on how these demonstrations help them to improve the responses. Figure 1: The schematic representation of our DRUM framework. Circles, rectangles, and triangles respectively represent the images, prompts, and responses in the triplet. To address the above issue, we now present a novel framework, d emonstration r etriever for large m u lti-modal m odel (DRUM). DRUM is targeted at fine-tuning a pre-trained visual-language embedding model so that it learns to retrieve better demonstrations to meet the LVLM’s needs when conduct inference. First, assuming the embedding model is given, DRUM discusses the retrieval strategy for any visual-language tasks. And it proposes to retrieve demonstrations based on the joint embedding of input image, prompt and draft response. Second, DRUM asks the inference LVLM to re-rank the embedding model’s retrieved demonstrations via the LVLM feedback. In this work, the LVLM feedback on a demonstration is defined as the conditional log-likelihood of the target response when the demonstration is added to the prompt. With the LVLM’s reranking results, a list-wise ranking loss can be calculated and used as the optimization objective for the embedding model. Third, we propose an iterative demonstration mining strategy which updates the demonstration candidates iteratively, thus improving the training of the embedding model by providing high-quality ranking signals. We have conducted extensive experiments on 3 types of visual-language tasks, VQA, ImageCLS and ImageCAP, and totally 7 benchmark datasets. The experimental results demonstrate that our DRUM framework is effective in boosting the LVLM’s ICL performance. In addition, for commercial LVLMs like GPT-4o, the embedding model fine-tuned by DRUM can also be transferred to them, help them to retrieve better demonstrations. Our contributions are as follows: • We propose a novel framework, DRUM, to enhance the ICL capabilities of the LVLMs. • Extensive experiments have proven that DRUM is effective in boosting the LVLMs’ ICL performance on a wide range of vision-language tasks. 2 Related Work In-Context Learning in NLP. The artificial intelligence community has witnessed significant advancements in the realm of large language models (LLMs) in recent years. As these models and their training corpora expand in scale, LLMs have demonstrated emergent capabilities, such as reasoning, mathematical proficiency, and the ability to follow prompts Wei et al. 2022 . GPT-3 Brown et al. 2020 was the pioneer in revealing that sufficiently large models can learn to execute new tasks with minimal guidance, a phenomenon termed in-context learning (ICL). Subsequent studies have corroborated the impressive performance of LLMs across various tasks through ICL Mosbach et al. 2023 . The crux of ICL lies in the construction of high-quality in-context demonstration sequences Li et al. 2023c . However, the bulk of these explorations have concentrated on pure natural language processing tasks and text-centric foundation models, highlighting the necessity to extend this research to encompass other domains. The research works on in-context learning focus primarily on demonstration sequences. A series of techniques have been investigated, including: (a) utilizing similarity scores to retrieve more relevant in-context examples Li et al. 2023c , (b) employing machine-generated demonstrations Li et al. 2023b . The literature has seen a series of studies that reveals certain properties of LLMs when applied to in-context learning. Pan 2023 proposed a decomposition of ICL into the task recognition effect and the task learning effect, and quantified these capabilities of models with varying numbers of shots and scales. Additionally, Lyu et al. 2022 records the "copying effect" phenomenon in LLMs, which is also a type of shortcut inference. Our work complements this line of research by fine-tuning the vision-language embedding model to learn how to retrieve appropriate demonstrations. LVLM and ICL Inspired by the triumphs of LLMs in natural language processing, the vision-language domain has seen the rise of analogous large vision-language models (LVLMs) Du et al. 2022 . Among these, models such as BLIP2 Li et al. 2023a , MiniGPT-4 Zhu et al. 2023 , and LLAVA Liu et al. 2024 are pretrained by aligning image and text data through the use of adapters Houlsby et al. 2019 to reduce training overhead. While there are several VLMs available, it is worth noting that some of the models are unsuitable for in-context learning, as this capability demands that the LVLM handle inputs that interweave images and text content Alayrac et al. 2022 . Presently, there is scant research on multimodal ICL or ICL for LVLMs, with only a few studies focusing on rudimentary strategies. Yang et al. 2024 examines the