PromptHub: Enhancing Multi-Prompt Visual In-Context Learning with Locality-Aware Fusion , Concentration and Alignment

Tianci Luo, Jinpeng Wang, Shiyu Qin, Niu Lian, Yan Feng, Bin Chen, Chun Yuan, Shu-Tao Xia · 2026

Read PromptHub: Enhancing Multi-Prompt Visual In-Context Learning with Locality-Aware Fusion , Concentration and Alignment by Tianci Luo, Jinpeng Wang, Shiyu Qin, Niu Lian, Yan Feng, Bin Chen, Chun Yuan, Shu-Tao Xia (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-18891.txt.

Introduction

Luo and co-authors present PromptHub for visual in-context learning with many prompts. Vision models learn tasks by copying pixel demonstrations. Fusing several prompts helps, but patch-wise fusion wastes context cues. PromptHub fuses with locality awareness, using spatial priors to gather richer context. Concentration, alignment, and prediction goals guide training together. Data augmentation strengthens the supervision signal further. Tests on three core vision tasks show PromptHub ahead of prior fusion methods. Gains hold across out-of-distribution sets and varied retrieval setups, the authors report.

Catalog record

CREATORTianci Luo, Jinpeng Wang, Shiyu Qin, Niu Lian, Yan Feng, Bin Chen, Chun Yuan, Shu-Tao Xia
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43467
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.18891

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PromptHub: Enhancing Multi-Prompt Visual In-Context Learning with Locality-Aware Fusion , Concentration and Alignment Tianci Luo, Jinpeng Wang, Shiyu Qin, Niu Lian, Yan Feng, Bin Chen, Chun Yuan, Shu-Tao Xia Abstract Visual In-Context Learning (VICL) aims to complete vision tasks by imitating pixel demonstrations. Recent work ( 34 ) pioneered prompt fusion that combines the advantages of various demonstrations, which shows a promising way to extend VICL. Unfortunately, the patch-wise fusion framework and model-agnostic supervision hinder the exploitation of informative cues, thereby limiting performance gains. To overcome this deficiency, we introduce PromptHub, a framework that holistically strengthens multi-prompting through locality-aware fusion, concentration and alignment. PromptHub exploits spatial priors to capture richer contextual information, employs complementary concentration, alignment, and prediction objectives to mutually guide training, and incorporates data augmentation to further reinforce supervision. Extensive experiments on three fundamental vision tasks demonstrate the superiority of PromptHub. Moreover, we validate its universality, transferability, and robustness across out-of-distribution settings, and various retrieval scenarios. This work establishes a reliable locality-aware paradigm for prompt fusion, moving beyond prior patch-wise approaches. Code is available at https://github.com/luotc-why/ICLR26-PromptHub . 1 Introduction Foundation models like GPT ( 5 ) , Llama ( 32 ) , Gemini ( 31 ) and Flamingo ( 1 ) have demonstrated the emerging ability of demonstration-based prompt learning, aka In-Context Learning (ICL) ( 8 ; 51 ; 42 ) , which further facilitates their versatility in various tasks. The basic idea of ICL ( 13 ; 39 ; 38 ; 16 ) is to prompt models with some demonstrative input-output pairs in addition to the query input, which can enhance the answer robustness. The reliablity of ICL has been thoroughly validated ( 29 ; 43 ; 7 ; 33 ) . Recently, Visual ICL (VICL) ( 4 ; 36 ) have also become a popular topic, where pixel-space in-painting is the native paradigm. Figure 1: (a) Condenser performs patch-wise fusion to fuse composite prompt, while leveraging model-agnostic supervision signals at the input level. (b) PromptHub transcends Condenser by enforcing a locality-aware chain that unifies fusion-utilization-prediction. It aligns spatial priors into coherent prompt representations, reinforces the backbone’s concentration on fused cues, and integrates label prediction to maintain the integrity of VICL pipeline. (c) Comparison of Condenser and PromptHub across three tasks under both single-prompting and multi-prompting configurations. Choosing appropriate prompts is critical in VICL. Many recent works ( 50 ; 30 ; 41 ; 52 ) focused on optimizing retrievers to select better suited prompts, and 48 incorporated visual prompt tuning to enhance the robustness of VICL. Nature Language Processing (NLP) literature ( 28 ; 11 ) suggests multiple prompts can enhance ICL with mitigated bias and richer context, offering insights essential for advancing VICL. Yet visual backbones like MAE-VQGAN typically restrict inputs to a single prompt, rendering multi-prompting non-trivial. In practice, there are two heuristic strategies for extending single-prompting to multi-prompting, namely downscaling ( 35 ) and ensemble ( 30 ) . Building upon this, Condenser ( 34 ) was the first to adopt prompt fusion ( 21 ) , integrating useful information from multiple prompts into a fused prompt, as illustrated in Figure 1 (a). However, its patch-wise fusion strategy results in substantial underuse of valuable cues, while model-agnostic supervision remains insufficient. Moreover, discrepancies between fused prompt and query pair may compels backbone to distrust fused representation, falling back on its own capacity for inference. This is precisely the situation we aim to avoid. We break these limits by proposing PromptHub, which aims at (i) integrate precise knowledge from diverse prompts, (ii) mitigate fused prompt’s discrepancies to encourage the backbone’s effective trust and reliance on it, and (iii) ultimately yield superior VICL predictions. To achieve this, we propose a locality-aware fusion framework together with three cooperative learning objectives, as illustrated in Figure 1 (b). Specifically, we introduce a locality prior that applies spatially decaying weights radiating from the current patch, thereby enhancing accurate feature extraction. This design allows the fusion process to retain a global receptive field while alleviating the adverse effects of border noise. During the optimization of PromptHub, we design three complementary objectives: (i) an end-to-end semantic integrity loss to promote high-quality prompt fusion by aligning fused exemplars with query semantics, as semantically closer prompt generally benefit VICL; (ii) a utilization loss that mitigates discrepancies between the fused prompt and the query pair, thereby promoting the backbone’s trust and reliance on the fused representation for imitation learning; and (iii) a label prediction loss, retained from Condenser , which serves as the base supervision to preserve VICL’s contextual prediction behavior. Additionally, we preliminarily explored VICL-oriented data augmentation strategies to enhance the robustness of PromptHub. These designs realize chain-wide enhancements for VICL. We evaluate PromptHub on segmentation, detection, and colorization. As shown in Figure 1 (c), extensive experiments demonstrated its superiority to state-of-the-art baselines. We also demonstrate PromptHub’s promising resource efficiency, transferability and robustness to various prompt retrieval strategies including random selection. Comprehensive ablations on diverse learning objectives, data augmentation techniques, and locality fusion, confirming the success of our paradigm design. We further visualize the fused prompts, which validates the reliablity of PromptHub. These findings strongly support the efficacy of PromptHub, highlighting the significance of our approach in VICL. To sum up, we make the following contributions. • We introduce a locality-enhanced fusion strategy that balances spatial locality and receptive field, enabling more comprehensive extraction of effective information. • We propose three complementary learning objectives that collaboratively enhance prompt fusion quality, strengthen prompt concentration, and improve contextual prediction, further reinforced with VICL-specific data augmentation. • Extensive experiments show PromptHub’s efficacy beyond state-of-the-art techniques. Promising results also suggest that it is transferable across domains and robust to prompt retrieval, establishing a reliable competitive new solution in VICL. 2 Related Works 2.1 Large Vision Models The field of computer vision has witnessed substantial advancements, driven by abundant foundational models ( 6 ; 37 ; 22 ; 25 ; 44 ) . LVM ( 3 ) , an auto-regressive generative model, effectively converted visual information into language-like visual sentences and improved understanding capabilities. MAE and Point-MAE ( 12 ; 23 ) , utilizing a reconstruction strategy, established unified visual architectures across various downstream tasks in 2D and 3D domains. The ability for ICL has also been demonstrated within foundation models, as researchers employed specialized training methodologies ( 4 ; 10 ; 36 ; 40 ; 47 ; 45 ; 46 ) to endow these models with superior in-context learning capabilities, thus providing a robust foundation for the domain of Visual ICL (VICL). 2.2 Visual In-Context Learning via In-Painting MAE-VQGAN ( 4 ) and Painter ( 35 ) serves as crucial in-painting backbones for VICL, with copious works building upon and enhancing this framework. Existing work ( 50 ; 30 ; 41 ) primarily focuses on retrieval to obtain better prompt. 30 studied prompt spatial arrangement, testing eight configurations and reporting improved results through voting. 48 pioneered visual prompt tuning ( 24 ; 15 ; 20 ; 2 ) , adding a noise border to prompts. PANICL ( 49 ) employs a training-free k-nearest-neighbor fusion integrates multiple prompts to alleviate the bias inherent in single prompt. PICO ( 17 ) reformulates personalized vision problem under the VICL paradigm and

Read the complete text