DetPO: In-Context Learning with Multi-Modal LLMs for Few-Shot Object Detection

Gautam Rajendrakumar Gare 1,, Neehar Peri 1,⋆, Matvei Popov 2, Shruti Jain 1, John Galeotti 1, Deva Ramanan 1 · 2026

Read DetPO: In-Context Learning with Multi-Modal LLMs for Few-Shot Object Detection by Gautam Rajendrakumar Gare 1,, Neehar Peri 1,⋆, Matvei Popov 2, Shruti Jain 1, John Galeotti 1, Deva Ramanan 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-23455.txt.

Introduction

Gare and five co-authors improve few-shot object detection for multimodal language models without retraining them. Such models ground objects well on common benchmarks yet fail on odd classes and imaging modes. Few-shot examples in the prompt often hurt more than class names alone. The team instead tunes the text prompt at test time. Their DetPO method scores prompts on a few labeled examples and calibrates confidence. It needs no gradients and no access beyond the API. Gains reach 9.7 mAP over prior black-box methods.

Catalog record

CREATORGautam Rajendrakumar Gare 1,, Neehar Peri 1,⋆, Matvei Popov 2, Shruti Jain 1, John Galeotti 1, Deva Ramanan 1
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64858
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.23455

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DetPO: In-Context Learning with Multi-Modal LLMs for Few-Shot Object Detection Gautam Rajendrakumar Gare 1,, Neehar Peri 1,⋆, Matvei Popov 2, Shruti Jain 1, John Galeotti 1, Deva Ramanan 1 Abstract Multi-Modal LLMs (MLLMs) demonstrate strong visual grounding capabilities on popular object detection benchmarks like OdinW-13 and RefCOCO. However, state-of-the-art models still struggle to generalize to out-of-distribution classes, tasks and imaging modalities not typically found in their pre-training. While in-context prompting is a common strategy to improve performance across diverse tasks, we find that it often yields lower detection accuracy than prompting with class names alone. This suggests that current MLLMs cannot yet effectively leverage few-shot visual examples and rich textual descriptions for object detection. Since frontier MLLMs are typically only accessible via APIs, and state-of-the-art open-weights models are prohibitively expensive to fine-tune on consumer-grade hardware, we instead explore black-box prompt optimization for few-shot object detection. To this end, we propose Detection Prompt Optimization (DetPO), a gradient-free test-time optimization approach that refines text-only prompts by maximizing detection accuracy on few-shot visual training examples while calibrating prediction confidence. Our proposed approach yields consistent improvements across generalist MLLMs on Roboflow20-VL and LVIS, outperforming prior black-box approaches by up to 9.7%. Our code is available on GitHub . Figure 1 : Detection Prompt Optimization. We cast the problem of gradient-free few-shot object detection as multimodal in-context learning (ICL). Here, a frozen multi-modal LLM (MLLM) is presented with a class name, a textual description, and a few visual examples ( left ), similar to the instructions given to a human annotator tasked with annotating that class [ 45 ] . Rather than presenting the visual examples directly to the MLLM, we find that it is far more effective to use them to optimize a better prompt ( right ) via prompt optimization; we use another black-box MLLM to discover prompt instructions that perform better on the few-shot training dataset. These improved instructions are then fed into the target MLLM. Table 1 : Multi-Modal ICL Hurts Detection Accuracy . Current MLLMs struggle to consistently benefit from multi-modal in-context examples for object detection. We posit that rigid post-training prompt structures make it difficult to effectively leverage additional context. To address this limitation, we propose Detection Prompt Optimization (DetPO) to encode few-shot multi-modal examples into a single text-only prompt, significantly improving detection accuracy. Method Class Names Instructions Images mAP Qwen2.5-VL [ 5 ] (7B) ✓ ✗ ✗ 4.6 Qwen2.5-VL [ 5 ] (7B) ✓ ✓ ✗ 6.2 Qwen2.5-VL [ 5 ] (7B) ✓ ✓ ✓ 1.8 Qwen2.5-VL [ 5 ] (72B) ✓ ✗ ✗ 7.1 Qwen2.5-VL [ 5 ] (72B) ✓ ✓ ✗ 10.4 Qwen2.5-VL [ 5 ] (72B) ✓ ✓ ✓ 10.1 Qwen3-VL [ 4 ] (8B) ✓ ✗ ✗ 10.4 Qwen3-VL [ 4 ] (8B) ✓ ✓ ✗ 11.4 Qwen3-VL [ 4 ] (8B) ✓ ✓ ✓ 7.0 Qwen3-VL [ 4 ] (30B-A3B) ✓ ✗ ✗ 10.7 Qwen3-VL [ 4 ] (30B-A3B) ✓ ✓ ✗ 11.9 Qwen3-VL [ 4 ] (30B-A3B) ✓ ✓ ✓ 9.8 Gemini 3 Pro [ 13 ] ✓ ✗ ✗ 21.9 Gemini 3 Pro [ 13 ] ✓ ✓ ✗ 23.0 Gemini 3 Pro [ 13 ] ✓ ✓ ✓ 23.9 1 Introduction Multi-Modal LLMs (MLLMs) achieve remarkable zero-shot performance across diverse vision-language tasks like image captioning, OCR, and VQA [ 5 , 11 , 1 ] . On flagship vision tasks like object detection, generalist MLLMs like Qwen3-VL [ 4 ] perform on-par with specialist object detectors like GroundingDINO [ 34 ] on popular benchmarks like RefCOCO [ 67 ] and OdinW-13 [ 30 ] . However, such models struggle to generalize to out-of-distribution concepts (e.g. material property estimation, defect detection, and contextual action recognition) and imaging modalities (e.g. X-rays, thermal spectrum data, and aerial imagery) not typically found in internet-scale pre-training [ 45 ] . We argue that some data will always remain out-of-distribution, whether due to being sequestered from the internet or being created after a model’s training cutoff. This motivates the need to learn from few-shot examples, similar to how human annotators are taught new concepts via few-shot multi-modal instructions [ 38 ] . Few-Shot Detection via In-Context Learning (ICL). Since frontier MLLMs are typically only accessible via APIs, and state-of-the-art open-weights models are prohibitively expensive to fine-tune on consumer-grade hardware, we argue that gradient-free few-shot learning is best framed through the lens of multi-modal in-context learning (ICL). Although ICL has been extensively studied for LLMs [ 8 , 14 ] , multi-modal ICL for object detection poses unique challenges. We primarily explore this problem using the recently released Roboflow20-VL (RF20-VL) [ 45 ] benchmark, which aggregates 20 distinct datasets from domains typically not found in internet-scale pre-training. Importantly, object classes are specified via few-shot visual examples and rich textual instructions. In practice, we find that prompting with few-shot visual examples yields inconsistent benefits compared to using class names and instructions (Table 1 ). We posit that this arises from rigid prompt structures used during post-training, making it difficult to exploit additional multi-modal contextual information during zero-shot inference. Detection Prompt Optimization Improves Concept Alignment. To address these limitations, we introduce Detection Prompt Optimization (DetPO), a black-box optimization approach that iteratively refines text-only prompts by maximizing detection accuracy on few-shot visual training examples. Unlike classical detectors that primarily learn from true positive examples, we find that MLLMs benefit from seeing corner cases and negative examples , akin to how human annotators refine their understanding of a target class by learning what not to annotate. At each iteration, DetPO refines a text-only prompt based on the model’s true positive, false positive, and false negative predictions on the few-shot training set. DetPO Down-Weights False Positive Detections. We further observe that current MLLMs often overpredict bounding boxes and lack per-box confidence scores by default. We find that simply prompting models for per-box scores improves detection accuracy without additional compute cost. However, these self-reported confidence scores can sometimes be poorly calibrated. In such cases, we can optionally post-process detections with VQA Score [ 33 ] . We prompt the model with bounding box predictions overlaid on the test image and ask “Is this bounding box an instance of class {CLS} ?” We use the normalized probability of the yes token as the final bounding box confidence score. This simple post-processing step down-weights false positives and further improves mAP. Contributions. We present three major contributions. First, we benchmark state-of-the-art MLLMs on RF20-VL and LVIS and show that naively prompting with multi-modal in-context examples yields poor performance. Next, we propose Detection Prompt Optimization (DetPO) to iteratively use model predictions on the training set to refine text-only prompts and estimate better per-box confidence scores to improve concept alignment. Lastly, we extensively ablate our design choices and demonstrate that DetPO consistently improves performance across popular MLLMs, establishing a new state-of-art for black-box few-shot object detection. 2 Related Works Vision-Language Models are often pre-trained on large-scale, weakly supervised image-text pairs [ 53 ] before fine-tuning on task-specific data. While early VLMs mainly focused on image classification [ 47 ] and VQA, recent methods have extended these models for visual grounding through open-vocabulary detection. Early efforts adapted VLMs for detection by classifying region proposals [ 20 , 19 ]

Read the complete text