Love Me, Love My Label : Rethinking the Role of Labels in Prompt Retrieval for Visual In-Context Learning

Tianci Luo, Haohao Pan, Jinpeng Wang, Niu Lian, Xinrui Chen, Bin Chen, Shu-Tao Xia, Chun Yuan · 2026

Read Love Me, Love My Label : Rethinking the Role of Labels in Prompt Retrieval for Visual In-Context Learning by Tianci Luo, Haohao Pan, Jinpeng Wang, Niu Lian, Xinrui Chen, Bin Chen, Shu-Tao Xia, Chun Yuan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-03657.txt.

Introduction

Luo and colleagues show that image in-context learning prompts chosen for image likeness alone can be label-mismatched and degrade scores. They develop LaPR, a label-aware prompt picking framework that builds an image-label joint code for prompts. It uses a mixture-of-experts setup with query-tuned routing for missing query labels. LaPR improves in-context cutting, spotting, and tinting, and carries over across feature pullers and cross-fold setups.

Catalog record

CREATORTianci Luo, Haohao Pan, Jinpeng Wang, Niu Lian, Xinrui Chen, Bin Chen, Shu-Tao Xia, Chun Yuan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35433
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.03657

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Love Me, Love My Label : Rethinking the Role of Labels in Prompt Retrieval for Visual In-Context Learning Tianci Luo, Haohao Pan, Jinpeng Wang, Niu Lian, Xinrui Chen, Bin Chen, Shu-Tao Xia, Chun Yuan Abstract Visual in-context learning (VICL) enables visual foundation models to handle multiple tasks by steering them with demonstrative prompts. The choice of such prompts largely influences VICL performance, standing out as a key challenge. Prior work has made substantial progress on prompt retrieval and reranking strategies, but mainly focuses on prompt images while overlooking labels. We reveal these approaches sometimes get visually similar but label-inconsistent prompts, which potentially degrade VICL performance. On the other hand, higher label consistency between query and prompts preferably indicates stronger VICL results. Motivated by these findings, we develop a framework named LaPR ( L abel- a ware P rompt R etrieval), which highlights the role of labels in prompt selection. Our framework first designs an image–label joint representation for prompts to incorporate label cues explicitly. Besides, to handle unavailable query labels at test time, we introduce a mixture-of-expert mechanism to the dual encoders with query-adaptive routing. Each expert is expected to capture a specific label mode, while the router infers query-adaptive mixture weights and helps to learn label-aware representation. We carefully design alternative optimization for experts and router, with a VICL performance-guided contrastive loss and a label-guided contrastive loss, respectively. Extensive experiments show promising and consistent improvement of LaPR on in-context segmentation, detection, and colorization tasks. Moreover, LaPR generalizes well across feature extractors and cross-fold scenarios, suggesting the importance of label utilization in prompt retrieval for VICL. Code is available at https://github.com/luotc-why/CVPR26-LaPR. 1 Introduction Figure 1 : We randomly sample 100 query-prompt training pairs constructed by SupPR 46 . We compute the label matching consistency and VICL performance to investigate their correlation. Figure 2 : Prompt retrieval paradigms. (a) Label-agnostic pipelines rely on image similarity and may yield label-inconsistent prompts and disturb inference. (b) LaPR considers both image similarity and label consistency for prompt retrieval, retrieving more relevant prompts. Large foundation models 4 ; 30 ; 1 are favoured in many user-centric applications for their strong in-context learning (ICL) capacity, where different demonstrative prompts enable a model to handle various tasks. Following the emergence of a series of vision backbones 7 ; 3 ; 36 ; 38 ; 2 , Visual ICL (VICL) has also gained increasing attention from the computer vision community. A milestone is MAE-VQGAN 3 , which typically formulates VICL as pixel inpainting. The input image is organized in a 2\times 2 grid: the top half contains the prompt image and its pixel-format label, the bottom-left cell holds the query image, and the lower-right cell is masked out. The model’s task is to reconstruct this masked region and yield the label prediction to the query. Existing studies 27 ; 46 ; 11 ; 12 dive into effective factors to improve ICL, where prompt selection plays a critical role. More recent works have focused on retrieving better prompts to enhance VICL performance. For instance, 46 defined positive and negative prompt pairs based on their contribution to VICL inference performance and introduced contrastive learning into retriever optimization, while 42 ; 40 reformulated prompt selection as a reranking process and achieved notable improvements through score-based prediction. However, at inference, only the query image is observable, and for input symmetry, the prompt side typically omits labels, which forfeits useful information. As shown in Figure 2 (a), for a query whose subject is a cat, the system might retrieve a prompt image containing both a cat and a flower but annotated with the label “flower”, leading to erroneous in-context predictions. Interestingly, as illustrated in Figure 1 , we observe that among prompts with relevant images, the label consistency with the query is positively correlated to the VICL performance. It motivates us to make better use of label information in prompt retrieval for VICL. As shown in Figure 2 (b), our goal is to enhance query-prompt label consistency in prompt retrieval, encouraging more accurate and reliable in-context predictions. To this end, we propose a new paradigm for prompt retrieval called LaPR ( L abel- a ware P rompt R etrieval). We design label-aware strategies for the prompt and query, respectively. For the prompt, we explicitly inject label information into the representation by fusing image and label features into joint representation. For the query, more challengingly, to tackle the unavailable label at test time, we adopt mixture-of-expert designs for the dual encoders, aiming to perceive and adapt to the implicit ( i.e . , unknown) query label. Specifically, each expert in the prompt and the query encoder is designated to capture a distinct mode ( e.g . long tail, horn, sharp beak), and the query-dependent router infers soft mixture weights to entail the implicit query label, obtaining adaptive query and prompt embeddings, as well as similarity scores adaptive to the implicit query label. In particular, to decouple the roles of the experts and the router, we adopt alternating optimization with two successive updates per mini-batch. The expert step learns via a performance-guided contrastive objective to strengthen mode-specific encodings. The router step optimizes a label-guided objective to align query-adaptive modes and adds the load-balancing regularizer to avoid expert under-utilization. Extensive experiments substantiate the effectiveness of LaPR framework. LaPR consistently achieves state-of-the-art performance across foreground segmentation, single-object detection, and colorization tasks. Furthermore, it exhibits robustness cross feature extractors generalization and remarkable transferability under cross-folder settings. These results underscore the pivotal role of label information in VICL prompt selection, and establish a new foundation for label-aware prompt retrieval in VICL. To Sum up, we make main contributions as follows: • We present the first label-aware prompt retrieval in VICL, explicitly addressing label inconsistency and offering new conceptual insights into prompt selection. • We inject prompt labels to create label-aware embeddings, and depict different modes with multiple experts on both sides. The query-specific router assigns mixture weights to experts, helping to yield an estimated query label and guide the extraction of query-relevant prompt information. • We decouple the roles of the experts and the router. The experts are trained to strength mode-specific representations, while the router is trained to infer mixture proportions over modes that align with the ground truth query label. • LaPR attains state-of-the-art results on foreground segmentation, single-object detection, and colorization tasks, and shows robust cross-backbone generalization and transferability under cross-folder scenario. 2 Related Works 2.1 In-Context Learning In-context learning (ICL) empowers large language models to learn task patterns and acquire task completion abilities by providing a few examples within the input prompt. This approach has seen extensive development 4 ; 1 and application 16 ; 47 in natural language processing (NLP) and multi-modal fields. Furthermore, the theoretical foundations of ICL have been rigorously validated 31 . Building upon its broad applicability and theoretical grounding, ICL has undergone significant advancements, particularly in improving example retrieval methods 11 ; 23 ; 41 ; 8 ; 20 ; 43 . The emergence of a series of vision backbones 38 ; 2 ; 19 ; 7 has further advanced the application of ICL in the visual domain, termed Visual In-Context Learning (VICL). MAE-VQGAN 3 and Painter 37 have demonstrated

Read the complete text