Learning to Select Visual In-Context Demonstrations
Eugene Lee, Yu-Chi Lin, Jiajie Diao · 2026
Read Learning to Select Visual In-Context Demonstrations by Eugene Lee, Yu-Chi Lin, Jiajie Diao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26775.txt.
Introduction
Lee, Lin, and Diao train a reinforcement learning agent to pick demonstration examples for visual in-context learning. The standard nearest-neighbour pick favours look-alike examples, which can repeat the same narrow slice of answers. Their agent builds the example set step by step to lift the main model's score. A dueling DQN with a query-focused decoder drives the choices across five visual regression benchmarks. Learned picking wins big on factual regression tasks, while nearest neighbours stay best for subjective preference tasks.
Catalog record
| CREATOR | Eugene Lee, Yu-Chi Lin, Jiajie Diao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64341 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.26775
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning to Select Visual In-Context Demonstrations Eugene Lee, Yu-Chi Lin, Jiajie Diao Abstract Multimodal Large Language Models (MLLMs) adapt to visual tasks via in-context learning (ICL), which relies heavily on demonstration quality. The dominant demonstration selection strategy is unsupervised k-Nearest Neighbor (kNN) search. While simple, this similarity-first approach is sub-optimal for complex factual regression tasks; it selects redundant examples that fail to capture the task’s full output range. We reframe selection as a sequential decision-making problem and introduce Learning to Select Demonstrations (LSD), training a Reinforcement Learning agent to construct optimal demonstration sets. Using a Dueling DQN with a query-centric Transformer Decoder, our agent learns a policy that maximizes MLLM downstream performance. Evaluating across five visual regression benchmarks, we uncover a crucial dichotomy: while kNN remains optimal for subjective preference tasks, LSD significantly outperforms baselines on objective, factual regression tasks. By balancing visual relevance with diversity, LSD better defines regression boundaries, illuminating when learned selection is strictly necessary for visual ICL. 1 Introduction † † Project page and code: https://eugenelet.github.io/LSD-Project/ Multimodal Large Language Models (MLLMs) and Large Language Models (LLMs) have demonstrated remarkable abilities in complex tasks through in-context learning (ICL) 6 , including mathematical reasoning 41 . This paradigm has driven a significant shift in few-shot learning (FSL). With the advent of powerful Vision Foundation Models (VFMs) and Vision-Language Models (VLMs), ICL is now the dominant approach for few-shot adaptation. Consequently, as 58 notes, the core research question has pivoted from training few-shot learners to effectively prompting massively pre-trained models. Figure 1 : An overview of our LSD (Learning to Select Demonstrations) framework. The process is a training loop where the MLLM acts as the Environment. (1) The Agent (a Dueling DQN) receives the current state s_{t} , which contains the query embedding \mathbf{e}_{q} and the embeddings of all previously selected demonstrations \{\mathbf{e}_{1},\dots,\mathbf{e}_{t-1}\} . (2) The agent’s query-centric decoder outputs an advantage query \mathbf{a}_{s} , which is used to retrieve candidates A_{\text{cand}} from the Task’s Data via FAISS. (3) The agent selects the next best demonstration, d_{t} . (4) The full prompt (including the selected demos d_{1}\dots d_{K} and the query) is sent to the MLLM (Environment), which makes a prediction. (5) A Reward r_{t} is calculated based on the prediction’s accuracy (e.g., MAE). (6) This reward is used to update the agent’s policy. However, ICL efficacy is highly sensitive to prompt configuration, especially the selection and ordering of demonstration examples 38 ; 22 . The impact of effective ICL spans diverse applications, including data engineering 37 ; 14 ; 5 , model augmentation 30 , knowledge updating 4 , model safety 26 ; 24 , and sentiment analysis 52 ; 44 ; 36 ; 47 . The most common selection strategy relies on unsupervised nearest neighbor (kNN) retrieval based on feature similarity 21 ; 32 ; 29 . While simple, this approach is often sub-optimal due to a lack of task-specific supervision 31 ; 50 ; 38 ; 53 . Its core “similarity-priority” assumption exhibits limited predictive power 42 and frequently yields redundant demonstration sets that provide misleading contextual information 16 . To move beyond simple similarity, research has explored demonstration ordering —arranging examples by proximity 21 or complexity 22 —and demonstration construction , emphasizing diversity 2 or using LLMs to generate new demonstrations 15 ; 46 ; 12 ; 48 . For visual ICL, complex retrieval-reranking paradigms have been proposed 57 , alongside metrics designed to select for “representativeness” 11 or to explicitly model structural complexity 16 . A more fundamental critique, inspired by hard negative mining 17 , argues these approaches over-index on positive, high-similarity examples. This has prompted a paradigm shift reframing shot selection as a sequential decision-making problem aimed at finding the most ”informative” examples 23 . This view treats demonstration selection as a task for a Reinforcement Learning (RL) agent, learning a policy to maximize cumulative rewards tied to final ICL accuracy 53 , shifting retrieval from simple visual similarity to a more abstract, task-oriented “reasoning process similarity” 29 . We embrace this sequential paradigm to address a critical gap in visual ICL: understanding when learned selection is actually necessary. Building on efforts utilizing LLM feedback 53 ; 55 or RL frameworks 39 , we propose LSD (Learning to Select Demonstrations) , a novel RL framework that trains a Dueling DQN agent to sequentially construct demonstration sets for visual regression tasks. Our key hypothesis is that the optimal selection strategy depends fundamentally on whether the task is objective or subjective . For objective, factual tasks, the optimal set must contain diverse “boundary” examples that help the MLLM model the entire regression space. Conversely, for subjective preference tasks, a simple visual anchor often suffices. As shown in Fig. 1 , our agent uses a query-centric Transformer Decoder to learn a policy that actively balances visual relevance with necessary diversity, avoiding the redundancy trap of kNN to maximize accuracy on complex objective domains. Our main contributions are: • We introduce LSD, a novel framework that successfully reframes K -shot demonstration selection as a sequential decision-making problem, scaling to dataset-level action spaces using a Dueling DQN agent and a query-centric Transformer Decoder. • We conduct a comprehensive study on the efficacy of learned selection policies across five diverse visual regression benchmarks (UTKFace, AVA, SCUT-FBP5500, KonIQ-10k, and KADID-10k). • We reveal a critical, task-dependent dichotomy in visual ICL: while unsupervised similarity search (kNN) remains highly effective for subjective preference tasks, our learned, diversity-aware policy is strictly necessary to achieve state-of-the-art performance on objective visual regression tasks. 2 Related Work Our research builds upon a large body of work in visual in-context learning (ICL), particularly on the critical problem of demonstration selection. Demonstration Selection for In-Context Learning. The performance of ICL is known to be highly sensitive to the choice of demonstration examples 54 ; 22 . This has been shown in various domains, with recent work demonstrating that MLLMs like GPT-4V can classify specialized medical images (e.g., histopathology) with high accuracy using just a few well-chosen examples 10 . This sensitivity has spurred significant research into methods that move beyond simple kNN retrieval. A primary challenge is selecting an optimal set of demonstrations, not just individual relevant examples. One line of work treats this as a subset selection problem. Yang et al . 48 proposed selecting a single, representative set of demonstrations applicable to all test instances, using a Determinantal Point Process (DPP) to ensure both quality and diversity. This task-level approach contrasts with our instance-level policy. Purohit et al . 28 (CASE) framed set selection as a multi-armed bandit problem, treating each subset as an “arm” and using a novel sampling strategy to efficiently find the best set while minimizing expensive LLM calls. Another line of research explores the trade-off between the two main criteria for selection: similarity and diversity. While similarity-based retrieval is effective for simple tasks, Xiao et al . 43 systematically demonstrated that incorporating diversity is crucial for improving performance and robustness on complex tasks, such as math and code generation. This finding directly supports our hypothesis that a learned agent is necessary to intelligently balance these two competing objectives. Rather than retrieving a static set, other methods treat selection as a sequential