DETR-ViP: Detection Transformer with Robust Discriminative Visual Prompts

Bo Qian Dahu Shi Xing Wei, [0.3em] School of Software Engineering, Xi’an Jiaotong University, CCAI, Zhejiang University, Hikrobot Co., Ltd. · 2026

Read DETR-ViP: Detection Transformer with Robust Discriminative Visual Prompts by Bo Qian Dahu Shi Xing Wei, [0.3em] School of Software Engineering, Xi’an Jiaotong University, CCAI, Zhejiang University, Hikrobot Co., Ltd. (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14684.txt.

Introduction

Qian, Shi, and Wei argue that visual prompts for object detection underperform because they lack a global sense of what sets classes apart. Their DETR-ViP framework blends prompt features across the image, distills links between visual and text prompts, and fuses the results selectively for stability. Across COCO, LVIS, ODinW, and Roboflow100, the method beat leading rivals at detection guided by example images rather than words, with ablations confirming each added piece helped.

Catalog record

CREATORBo Qian Dahu Shi Xing Wei, [0.3em] School of Software Engineering, Xi’an Jiaotong University, CCAI, Zhejiang University, Hikrobot Co., Ltd.
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65075
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.14684

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DETR-ViP: Detection Transformer with Robust Discriminative Visual Prompts Bo Qian Dahu Shi Xing Wei, [0.3em] School of Software Engineering, Xi’an Jiaotong University, CCAI, Zhejiang University, Hikrobot Co., Ltd. Abstract Visual prompted object detection enables interactive and flexible definition of target categories, thereby facilitating open-vocabulary detection. Since visual prompts are derived directly from image features, they often outperform text prompts in recognizing rare categories. Nevertheless, research on visual prompted detection has been largely overlooked, and it is typically treated as a byproduct of training text prompted detectors, which hinders its development. To fully unlock the potential of visual-prompted detection, we investigate the reasons why its performance is suboptimal and reveal that the underlying issue lies in the absence of global discriminability in visual prompts. Motivated by these observations, we propose DETR-ViP, a robust object detection framework that yields class-distinguishable visual prompts. On top of basic image-text contrastive learning, DETR-ViP incorporates global prompt integration and visual-textual prompt relation distillation to learn more discriminative prompt representations. In addition, DETR-ViP employs a selective fusion strategy that ensures stable and robust detection. Extensive experiments on COCO, LVIS, ODinW, and Roboflow100 demonstrate that DETR-ViP achieves substantially higher performance in visual prompt detection compared to other state-of-the-art counterparts. A series of ablation studies and analyses further validate the effectiveness of the proposed improvements and shed light on the underlying reasons for the enhanced detection capability of visual prompts. 3 3 footnotetext: Project page: https://github.com/MIV-XJTU/DETR-ViP 1 Introduction Compared with traditional closed-set object detection( 7 ; 37 ; 47 ), open-set object detection breaks through the limitation of predefined categories, thereby demonstrating greater potential in real-world applications. Prompt-based object detection has further inspired the development of open-set detection by enabling flexible identification of target objects. Several prominent approaches( 6 ; 8 ; 17 ; 24 ; 28 ; 32 ; 52 ) support text-prompted open-set detection by distilling knowledge from vision-language models such as CLIP( 35 ) or from language models like BERT( 5 ) through aligning visual representations with textual descriptions. The training paradigm of text-prompted model centers on aligning visual and textual feature spaces, with zero-shot generalization largely attributed to large-scale, generic image-text pairs. However, in specialized domains, relying solely on textual descriptions of target categories or attributes often proves insufficient for reliable detection. Some methods( 15 ; 19 ; 54 ; 36 ; 22 ) employ visual prompts to define target objects of interest, further enhancing the interactivity and adaptability of object detection models. Recently, several methods( 16 ; 2 ) have begun to explore object detection frameworks that support both textual and visual prompts, achieving promising progress. 16 empirically find that, compared with text prompts, visual prompts are more effective in recognizing rare categories. This is expected because visual prompts, being sampled from the visual domain, are naturally compatible with image features, thus possessing stronger generalization ability than text prompts, which require cross-modal (text-to-vision) alignment. Nevertheless, visual prompts still underperform text prompts overall, which limits their practical applicability. (a) t-SNE Visualization (b) Similarity Distribution VGDINO +Align +Global. +Distill 0.8 1.0 1.2 1.4 1.6 IISR 14 22 30 38 46 mAP (c) IISR vs. mAP Figure 1: Analysis of visual prompts. (a) t-SNE visualization of VIS-GDINO prompts sampled from 10 COCO categories. (b) Similarity distribution between VIS-GDINO prompts of the same category and across different categories. (c) Trends of Intra-Inter Similarity Ratio (IISR) and mAP. To systematically investigate the reasons behind this suboptimal performance and enhance detection capability, we build a baseline model incorporating visual prompts on top of Grounding DINO, termed VIS-GDINO, and conduct a detailed analysis. VIS-GDINO achieves mAP scores of 21.1 and 17.2 on COCO and LVIS, respectively, which are lower than those of Grounding DINO with text prompts. For each category, we sample 128 images to extract visual prompts, and perform a t-SNE analysis on all visual prompts, as shown in Figure 1 (a). It can be observed that the visual prompts of VIS-GDINO lack clear semantic boundaries, which may contribute to frequent misclassifications when using visual prompts for categorization. To illustrate this more clearly, we compute the pairwise cosine similarities among all visual prompts and plot the distributions of similarities for same-category and different-category prompts as histograms, shown in Figure 1 (b). Same-category and different-category prompts share a relatively wide similarity range, which underscores the disorganization of the prompt embedding space. Taken together with these observations, we attribute the suboptimal performance of visual prompts to two factors: (1) visual prompts from different instances of the same category exhibit large variance; and (2) visual prompts from different categories are heavily entangled in the global embedding space, making them difficult to distinguish. In summary, visual prompts suffer from a lack of sufficient semantic discriminability. Building upon the above insight, we improve VIS-GDINO into a detection model with more robust and discriminative visual prompts, termed DETR-ViP , by simultaneously reducing the intra-class variance and enlarging the inter-class distance of visual prompts. DETR-ViP introduces a global prompt integration strategy that simulates cross-image prompt detection during training while augmenting negative samples, thereby encouraging image features to encode more distinctive global semantics. In addition, we design a visual-text prompt relation distillation loss. This loss transfers the semantic priors from the similarity matrix of textual prompts to the visual prompts, encouraging them to capture semantic proximity. Finally, we propose a selective fusion strategy. The fusion of each visual prompt is conditioned on the model’s judgment of whether the corresponding category instance exists in the target image, resulting in more stable and robust detection. To quantitatively evaluate the quality of visual prompts by measuring intra-class variance and inter-class separability, we further introduce the Intra-Inter Similarity Ratio (IISR). A larger IISR indicates stronger semantic consistency of visual prompts. As shown in Figure 1 (c), both IISR and mAP on COCO consistently improve with the proposed modifications, validating that the performance gains stem from the optimization of visual prompt distributions. We conduct comprehensive evaluations on COCO( 27 ), LVIS( 9 ), ODinW( 21 ), and Roboflow 100( 3 ). Under comparable backbone settings, DETR-ViP consistently outperforms T-Rex2. Specifically, DETR-ViP-T outperforms T-Rex2-T by +4.4 mAP on COCO, and achieves +3.7 mAP over T-Rex2-T and +6.9 mAP over YOLOE-v8-L on LVIS. In terms of AP c and AP r , our model surpasses T-Rex2 by +9.4 and +5.2, and exceeds YOLOE-v8-L by +8.7 and +1.9, respectively. On ODinW and Roboflow 100, which contain category distributions substantially different from the training datasets, DETR-ViP-T surpasses T-Rex2-L by 3.4 and 5.1 mAP, respectively. DETR-ViP-L further improves upon DETR-ViP-T, surpassing T-Rex2-L by 3.7 and 1.5 mAP on COCO and LVIS, respectively. Beyond fair comparisons with existing methods, we also conducted extensive ablation studies to validate the effectiveness of each modification from VIS-GDINO to DETR-ViP. Moreover, both the IISR metric and t-SNE visualizations of visual prompts provide intuitive evidence that these improvements progressively optimize the structure of the prompt embedding space. 2 Related Work Prompt-based object detection provides a promising pathway toward open-vocabulary detection. Text-prompted approaches( 8 ; 17 ; 24 ; 28 ; 32 ; 46 ; 48 ; 51 ) have achieved remarkable progress, exhibiting strong zero-shot and few-shot recognition capabilities. These methods mainly align visual features with text embeddings, often leveraging pretrained language models such

Read the complete text