Meta-Adaptive Prompt Distillation for Few-Shot Visual Question Answering

Akash Gupta, Amos Storkey, Mirella Lapata · 2025

Read Meta-Adaptive Prompt Distillation for Few-Shot Visual Question Answering by Akash Gupta, Amos Storkey, Mirella Lapata (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-06905.txt.

Introduction

Akash Gupta, Amos Storkey, and Mirella Lapata wrote this 2025 paper on teaching large vision-language models to answer questions about images from just a few examples. Such models often fail to improve when given more examples, and the team blames stray detail in the image data that has nothing to do with the task. Their fix trains a small add-on module plus a fixed set of soft prompts.

The add-on reads task-relevant visual features and passes them into the prompts, and the whole setup can attach to any model of this kind. At test time, a few steps of tuning on a handful of examples adapt the prompts. On a standard test set, the method beat plain example-based learning by a wide margin and also topped fine-tuning methods.

Catalog record

CREATORAkash Gupta, Amos Storkey, Mirella Lapata
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-06905

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Meta-Adaptive Prompt Distillation for Few-Shot Visual Question Answering Akash Gupta, Amos Storkey, Mirella Lapata Abstract Large Multimodal Models (LMMs) often rely on in-context learning (ICL) to perform new tasks with minimal supervision. However, ICL performance, especially in smaller LMMs, is inconsistent and does not always improve monotonically with increasing examples. We hypothesize that this occurs due to the LMM being overwhelmed by additional information present in the image embeddings, which is not required for the downstream task. To address this, we propose a meta-learning approach that provides an alternative for inducing few-shot capabilities in LMMs, using a fixed set of soft prompts that are distilled from task-relevant image features and can be adapted at test time using a few examples. To facilitate this distillation, we introduce an attention-mapper module that can be easily integrated with the popular LLaVA v1.5 architecture and is jointly learned with soft prompts, enabling task adaptation in LMMs under low-data regimes with just a few gradient steps. Evaluation on the VL-ICL Bench shows that our method consistently outperforms ICL and related prompt-tuning approaches, even under image perturbations, improving task induction and reasoning across visual question answering tasks. 1 1 1 We release our training and evaluation code here - https://github.com/akashgupta97/MAPD 1 Introduction Humans have the remarkable ability to quickly learn new tasks in multimodal environments with just a few trial-and-error attempts. Extensive research in cognitive science suggests that this ability arises from learning hierarchical abstractions and maintaining shared structural priors across related tasks based on past experiences. ( Griffiths et al. 2019 ; Finn 2018 ; Kirsch and Schmidhuber 2022 ) . Drawing on this prior knowledge enables rapid learning in new situations and reduces the need for large amounts of task-specific demonstrations ( Finn et al. 2017 ) . Large Multimodal Models (LMMs) are able to perform a multitude of tasks ranging from reasoning to fine-grained image understanding and visual question answering ( Liu et al. 2024 ; Li et al. 2023a ; Laurençon et al. 2024 ) . They are typically built on top of a base Large Language Model (LLM) by supplementing it with a vision encoder and a connecting module that acts as a bridge for different modalities to interact. When (pre)trained at sufficient scale and finetuned on a wide range of multimodal tasks (with natural language instructions), LMMs can learn new tasks by virtue of in-context learning (ICL), i.e., by being prompted with a few input-output examples, without requiring any updates to the model parameters ( Zhao et al. 2024 ; Zong et al. 2025 ; Coda-Forno et al. 2023 ) . Although the training-free nature of ICL has led to its rapid adoption across tasks and domains, its underlying mechanism remains ill-understood ( Hendel et al. 2023 ; Huang et al. 2024 ) and its empirical behaviour can be inconsistent. Moreover, recent work ( Zong et al. 2025 ) demonstrates that ICL is most effective for large-scale LMMs ( \scriptstyle\mathtt{\sim} 72B parameters), while smaller models ( < 7B parameters) often struggle with increasing in-context examples and their performance either plateaus or deteriorates even when extending the context length or giving detailed instructions. Zong et al. 2025 attribute this limitation to the fact that smaller models struggle with the large number of image tokens in long sequences. As a result, they become confused and perform the task haphazardly or revert to default behaviors, such as drawing from their parametric knowledge, while effectively ignoring the in-context examples. Figure 1 provides an example of such a failure case with LLaVA-OneVision-7B LMM ( Li et al. 2025 ) for a task taken from the Fast Open-Ended MiniImageNet dataset ( Tsimpoukelli et al. 2021 ) . The model originally outputs a generic description about the image based on parametric knowledge and ultimately fails to give a correct answer, despite being prompted with a few examples. Figure 1 : Failure case of LLaVA-OneVision-7B ( Li et al. 2025 ) on an example from the Fast Open-Ended MiniImageNet classification task ( Tsimpoukelli et al. 2021 ) . When no in-context examples are provided (0-shot), the model initially generates a generic description of the image. As more examples (shots) are added, it begins to learn the answer format (single word), but still fails to grasp the task, producing incorrect or irrelevant predictions. Building on this observation, we hypothesize that effective few-shot adaptation at test time may be compromised by the added information introduced by the image embeddings. While a more precise set of embeddings would be preferable, the continuous nature of image embeddings makes it challenging to distill task-specific information from them. As an alternative, we propose to learn a fixed set of new embeddings that can be easily finetuned at test time. This idea of task adaptation has gained significant traction in the literature through prompt tuning ( Lester et al. 2021 ) which finetunes a set of continuous soft prompts while keeping the underlying language model frozen; the prompts are prepended in the context at test-time, effectively steering the model to perform the desired task. We introduce an approach for learning new tasks using learnable soft prompts that receive task information from the LLM in the form of loss gradients during finetuning. These gradients update the soft prompts which when fused with the image embeddings are able to distill relevant features from them. To facilitate this fusion, we propose an attention-mapper that uses a multi-head attention ( Vaswani et al. 2017 ) architecture responsible for extracting relevant task-specific image information. We adopt LLaVA v1.5 ( Liu et al. 2024 ) as our base LMM and substitute its naive MLP projection layer with our attention-mapper and set of learnable soft prompts. Needless to say, the approach outlined above relies on being able to adapt quickly to new tasks at test time after seeing only a few examples, since designing finetuning procedures for individual tasks is impractical. Prior work ( Finn et al. 2017 ; Ravi and Larochelle 2017 ; Vinyals et al. 2016 ) addressed this challenge by training a meta-learner that can infer an optimal learning strategy for a new task after being exposed to distribution of tasks. We propose to apply this meta-learning procedure in our multimodal prompt distillation setting. Specifically, we employ the widely known MAML algorithm ( Finn et al. 2017 ) and use its lightweight first-order approximation for training the attention-mapper and soft prompts. Our approach builds on previous work ( Qin et al. 2023 ; Najdenkoska et al. 2023 ; Li et al. 2023b ) highlighting the benefits of MAML for small vision-language models in tasks like fast-concept binding and classification. We focus on visual question answering (VQA; Antol et al. 2015 ; see example in Figure 1 ), a general-purpose task often used to evaluate the image understanding capabilities of LMMs, and demonstrate the benefits of MAML at scale for LLaVA v1.5 ( Liu et al. 2024 ) . Our contributions can be summarized as follows: • We propose an alternative to in-context learning (ICL) by meta-learning a fixed set of soft prompts within LMMs through distillation. Our approach, which we call MAPD (as a shorthand for M eta- A daptive P rompt D istillation), can quickly adapt to new

Read the complete text