Embracing Collaboration Over Competition: Condensing Multiple Prompts for Visual In-Context Learning

Jinpeng Wang, Tianci Luo, Yaohua Zha, Yan Feng, Ruisheng Luo, Bin Chen, Tao Dai, Long Chen, Yaowei Wang, Shu-Tao Xia · 2025

Read Embracing Collaboration Over Competition: Condensing Multiple Prompts for Visual In-Context Learning by Jinpeng Wang, Tianci Luo, Yaohua Zha, Yan Feng, Ruisheng Luo, Bin Chen, Tao Dai, Long Chen, Yaowei Wang, Shu-Tao Xia (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-21263.txt.

Introduction

Jinpeng Wang and co-authors rethink how vision models learn from example images, in a 2025 paper. In visual in-context learning, a model solves a new task by analogy with a few picture examples, called prompts. Standard methods hunt for one ideal prompt in a pool. In practice, several prompts may each hold part of the answer, and picking one wastes the rest.

Their tool, Condenser, squeezes the useful detail out of many prompts at once. It is a light plug-in trained together with the main model, so the merged context stays sharp without losing resolution. Tests across benchmark tasks beat prior state-of-the-art methods. The tool also scaled better as more prompts were added and ran cheaper than ensemble approaches. The authors frame prompts as collaborators rather than rivals.

Catalog record

CREATORJinpeng Wang, Tianci Luo, Yaohua Zha, Yan Feng, Ruisheng Luo, Bin Chen, Tao Dai, Long Chen, Yaowei Wang, Shu-Tao Xia
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-21263

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Embracing Collaboration Over Competition: Condensing Multiple Prompts for Visual In-Context Learning Jinpeng Wang, Tianci Luo, Yaohua Zha, Yan Feng, Ruisheng Luo, Bin Chen, Tao Dai, Long Chen, Yaowei Wang, Shu-Tao Xia Abstract Visual In-Context Learning (VICL) enables adaptively solving vision tasks by leveraging pixel demonstrations, mimicking human-like task completion through analogy. Prompt selection is critical in VICL, but current methods assume the existence of a single “ideal” prompt in a pool of candidates, which in practice may not hold true. Multiple suitable prompts may exist, but individually they often fall short, leading to difficulties in selection and the exclusion of useful context. To address this, we propose a new perspective: prompt condensation . Rather than relying on a single prompt, candidate prompts collaborate to efficiently integrate informative contexts without sacrificing resolution. We devise Condenser , a lightweight external plugin that compresses relevant fine-grained context across multiple prompts. Optimized end-to-end with the backbone, Condenser ensures accurate integration of contextual cues. Experiments demonstrate Condenser outperforms state-of-the-arts across benchmark tasks, showing superior context compression, scalability with more prompts, and enhanced computational efficiency compared to ensemble methods, positioning it as a highly competitive solution for VICL. Code is open-sourced at https://github.com/gimpong/CVPR25-Condenser . 1 Introduction Humans can complete tasks through imitation and analogy, even when these tasks are difficult or unfamiliar. Could a similar paradigm be extended to machines? This fundamental question has evoked research on In-Context Learning (ICL) [ 6 ] , especially with the advent of foundational models [ 3 , 26 , 27 ] . Recent work with Large Language Models (LLMs) [ 11 , 33 ] and Multimodal LLMs (MLLMs) [ 13 , 38 , 40 ] has demonstrated that prompting models using task-specific examples can efficiently adapt downstream tasks, achieving comparable efficacy to in-weight learning [ 4 ] . More inspiringly, the vision community has revealed that vision backbones [ 2 , 30 ] can also exhibit this property, unlocking a novel paradigm as Visual ICL (VICL) [ 30 , 9 , 14 ] . As shown in Figure 1 (a), VICL enables leveraging contextual pixel cues from prompts ( i.e . , demonstrations), which helps vision backbones solve tasks adaptively and effectively. Prompt selection plays a crucial role in VICL. While the early work [ 2 ] selected prompts randomly, Zhang et al. [37] suggested choosing prompts with higher relevance can notably enhance effectiveness. Based on this insight, a range of efforts [ 37 , 24 , 34 ] has been devoted to selecting the best prompt for the query image, typically using backbone’s task performance conditioned by single prompts as signals to learn prompt retrieval or ranking. Albeit remarkable progress, these methods are soon reaching a plateau. Essentially, they operate on a competitive assumption—that there exists a perfect prompt within the database to be retrieved. However, as illustrated in Figure 1 (b), this assumption may be challenged in practice. There are often multiple suitable prompts, but each one alone may fall short of being ideal . In such cases, prompt selection may struggle to make a single choice, inevitably dropping helpful prompts as context. This limitation could restrict the generalizability of VICL. Figure 1 : ( a ) The impainting framework [ 2 ] is a typical prototype for Visual In-Context Learning (VICL), where a fixed-size input canvas is divided into regions. The prompt image and its label occupy the top half, while the query image is placed in the bottom left. The vision backbone is tasked with recovering the bottom right region, which represents the query label, by reasoning within the pixel context. Prompt selection plays a crucial role in constructing the input and significantly impacts model predictions. ( b ) Most existing VICL methods view prompt selection as a competitive process, where a single prompt is chosen, often discarding useful alternatives. Meanwhile, the top-1 prompt may not be ideal for the query image. ( c ) We introduce the concept of collaborative condensation , where multiple raw prompts are combined to leverage their individual strengths into a single condensed prompt. The proposed Condenser is a lightweight, flexible, and efficient plugin to the vision backbone, effective across a range of VICL tasks. We believe prompt candidates necessity collaboration rather than exclusive competition in VICL. Combining multiple prompts is expected to enhance the compositional generalization ability, yet preliminary solutions, including downsampling and ensemble, still have clear limitations. The downsampling methods [ 2 , 29 , 37 ] scale down sub-images proportionally to arrange more prompts on the fixed-size input canvas, while they may lose visual details in prompts. Ensemble methods [ 24 , 34 ] , on the other hand, separately combine each candidate prompt with the query image to create multiple inputs and finally aggregate their contextual outputs. Output aggregation is quite tricky for specific tasks and often requires careful adjustments. What’s worse, it raises efficiency concerns as the candidate prompts increase. Beyond these practices, we introduce a new research perspective of condensation , aiming to efficiently integrate informative contexts in different raw prompts without sacrificing resolution for better VICL. We propose an end-to-end optimized approach to condense multiple prompts, as demonstrated in Figure 1 (c). Given a set of candidate prompts retrieved along with the query image, we devise a lightweight Condenser to adaptively fuse informative cues. The Condenser identifies relevant image patterns and their associated annotations within the candidate prompts to form the reference for each query image patch, helping to extract the most relevant fine-grained context in the final prompt. We train Condenser jointly with the vision backbone, utilizing end-to-end feedback to optimize prompt condensation. Similar to retrievers, Condenser can act as a flexible external plugin to the vision backbone, making it highly adaptable and effective across VICL tasks. We conduct extensive experiments on several benchmark tasks, including foreground segmentation, single-object detection, and image coloring. Under standard protocols, even when using only a single candidate prompt, Condenser outperforms state-of-the-art methods remarkably, highlighting its strong information extraction ability. More encouragingly, Condenser performs better as the number of candidate prompts increases, further demonstrating its information integration capability. At the same time, it maintains a clear computational efficiency advantage over ensemble-based multi-prompt methods [ 24 ] . To summarize, we make the following contributions. • We successfully shift the paradigm of prompt selection in VICL from exclusive competition to collaborative condensation, which enables multiple prompts to work together effectively and efficiently. • We design a lightweight Condenser to adaptively condense multiple prompts into a single, fine-grained prompt, enhancing VICL without sacrificing resolution. • Extensive experiments demonstrate that Condenser outperforms state-of-the-art methods on multiple tasks, showing effective context compression capacity, scalability with increasing candidate prompts, and favorable computational efficiency beyond output ensemble. Figure 2 : Prompt condensation with Condenser for VICL. Given a query image I_{q} from the training set, we first retrieve a set of candidate prompts \mathcal{S}_{q} from the prompt database \mathcal{P} . Then, we assemble each prompt from \mathcal{S}_{q} with the labeled query pair P_{q}=(I_{q},L_{q}) and construct K contextualized labeled samples, namely \tilde{X}_{c_{1}},\cdots,\tilde{X}_{c_{K}} . We encode each of them with a VQGAN encoder [ 7 ] h(\cdot) and obtain a serial of query label tokens, \bm{T}_{q_{c_{1}}}^{L},\cdots,\bm{T}_{q_{c_{K}}}^{L} as signals. Next, we transform and condense raw candidate

Read the complete text