FOCUS: Forcing In-Context Object Localization through Visual Support Constraints and Policy Optimization

Mohammed Asad Karim, Vinay Kumar Verma · 2026

Read FOCUS: Forcing In-Context Object Localization through Visual Support Constraints and Policy Optimization by Mohammed Asad Karim, Vinay Kumar Verma (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-31145.txt.

Introduction

Karim and Verma tackle finding an object in a new image from just a few example boxes, with no category label and no retraining. Their two-stage method trains attention between support boxes and the query image, then sharpens boxes with reinforcement learning that rewards smaller location error. A 7B-parameter model trained this way beats models up to 72B parameters, the authors report, showing that well-aimed objectives can outrun sheer scale.

Catalog record

CREATORMohammed Asad Karim, Vinay Kumar Verma
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44932
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.31145

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FOCUS: Forcing In-Context Object Localization through Visual Support Constraints and Policy Optimization Mohammed Asad Karim, Vinay Kumar Verma Abstract In-context localization (ICL) seeks to localize a target object specified by a small set of support examples in a query image, operating on the fly without training or parameter updates. Despite rapid advances in vision–language models (VLMs), achieving category-agnostic and visually grounded ICL remains an open problem, even though it is essential for applications such as image editing, personalized visual search, and retrieval. Existing methods are fragile and rely on explicit category supervision, which not only limits applicability in realistic settings with unnamed or instance-specific objects but also introduces category bias that steers predictions toward semantic priors rather than visual evidence. We introduce a two-stage training framework that explicitly optimizes in-context attention between support bounding boxes and query images without category supervision. We further refine localization via reinforcement learning using Group Relative Policy Optimization (GRPO) to directly minimize localization error. This formulation enforces visual correspondence over semantic priors, yielding robust instance-level localization. Empirically, a 7B-parameter model trained with our objectives outperforms models up to 72B parameters, demonstrating that context-aware localization objectives can surpass scaling alone. Comprehensive ablations validate the contribution of each component. 1 Introduction Vision–language models (VLMs) ( 41 ; 21 ; 8 ; 22 ) have achieved remarkable success across a broad spectrum of vision and language tasks, yet they continue to struggle with in-context object localization . In-Context Object Localization (ICOL) focuses on localizing a user-specified object in a query image by relying solely on a small set of visual support examples available at inference time. In contrast to traditional object detection or grounding approaches that depend on fixed category vocabularies and extensive supervised training, ICOL enables models to infer the target concept on the fly, without parameter updates, by reasoning over visual correspondences between support and query images. This capability is critical for practical applications such as customized image editing, personalized visual search, and interactive object tracking, where the object of interest is user-defined, instance-specific, and often difficult or impossible to describe textually in advance. Achieving reliable in-context localization is therefore a key step toward flexible, user-driven visual understanding systems. Despite the success of in-context learning in large language models (LLMs) ( 2 ; 4 ; 30 ; 32 ) , transferring this paradigm to vision–language models (VLMs) for object localization remains challenging. Recent work on in-context object localization shows that VLMs ( 39 ) can use support examples with bounding box annotations to localize novel objects at inference time; however, their predictions are still strongly shaped by category-level priors rather than instance-specific visual reasoning ( 12 ) . To address this, Doveh et al. introduce pseudo-labels during training to reduce reliance on true category names and encourage visual grounding. In practice, inference continues to rely on original category labels, creating a train–test mismatch that reintroduces semantic priors at deployment. As a result, category bias is only partially mitigated, and localization decisions can still be driven by semantic cues instead of the visual evidence provided by support examples. Moreover, empirical findings suggest that even with pseudo-label training, models frequently underutilize fine-grained spatial cues such as bounding box geometry and relative position defaulting to coarse visual similarity or residual category correlations. These observations reveal a central gap: existing approaches fail to eliminate category-mediated reasoning and do not enforce strict reliance on visual support constraints for instance-level localization. To address these limitations, we propose a category-agnostic, attention-based formulation of in-context object localization that removes category names entirely from both support and query prompts. Instead of relying on true labels or pseudo-labels, our method optimizes in-context attention using only visual support examples, ensuring that localization decisions are driven by visual correspondence and spatial reasoning. By removing textual identifiers, the model is no longer influenced by semantic category priors and must infer the target object from visual appearance and geometry. This attention-based design promotes direct use of bounding box information, encouraging consistent focus on object shape, relative position, and surrounding visual cues across support and query images. To further improve query alignment, we refine bounding box prediction using GRPO-based reward optimization, which directly minimizes alignment error. As a result, the model generalizes to arbitrary object instances, including unseen categories and visually defined concepts without stable names. Overall, our approach reframes in-context localization as a visual reasoning task, enabling robust and generalizable instance-level grounding. We summarize our key contributions as follows: • We propose a category-independent, pure visual context–based in-context localization framework that overcomes category-induced bias in VLMs. • We introduce an attention map optimization that encourages the model to focus on the most relevant regions in both support and query images for robust localization. In addition, a GRPO-based reward objective is used to reduce object bounding-box alignment error. • Through extensive experiments, we demonstrate that the proposed model ( FOCUS ) enables effective in-context localization without relying on category labels or prior semantic information. 2 Related Work In-context learning in language models. In-context learning (ICL) ( 12 ) enables large language models (LLMs) to perform new tasks by conditioning on a small set of demonstrations at inference time, without parameter updates. Early work showed that few-shot performance scales with model and context size in autoregressive models ( 5 ) . Subsequent studies interpret ICL as implicit Bayesian inference ( 44 ) or gradient-descent-like computation emerging from transformer attention dynamics ( 40 ) . Empirically, ICL is sensitive to prompt composition ( 29 ; 27 ) , motivating calibration ( 51 ) and retrieval-augmented methods ( 33 ) . Instruction-tuned models further improve zero-shot generalization ( 42 ) . In-context learning in vision–language models. Compared to language models, in-context learning for vision–language models (VLMs) remains relatively underexplored ( 2 ; 49 ; 30 ) . Recent multimodal models demonstrate in-context capabilities for tasks such as visual question answering, reasoning, and visual grounding when examples are provided in the prompt ( 2 ; 4 ; 9 ) . However, these demonstrations primarily focus on semantic understanding and rely on natural language to specify the task and the target object ( 25 ; 24 ) . As a result, in-context behavior in VLMs has largely been studied through language-conditioned interactions rather than through purely visual conditioning ( 31 ; 17 ) . A recent work IPLoc ( 12 ) explores the ICL with the help of pseudo-label and visual grounding, however pseudo label reduce the generalization ability since for the novel category model have not learned the pseudo label. Semantic grounding and localization in VLMs. Most vision–language models (VLMs) for grounding and localization rely on explicit semantic supervision, such as object names, attributes, or referring expressions. Early work formulates grounding as localizing regions described by natural language queries ( 19 ; 47 ) , later extended to end-to-end multimodal transformers that condition detection on text ( 18 ) . Recent large-scale pretraining unifies open-vocabulary detection and phrase grounding by aligning visual regions with linguistic descriptions ( 23 ; 48 ) , while CLIP-based methods adapt contrastively trained models for grounding tasks ( 43 ) . Despite strong performance, these approaches assume targets can be specified unambiguously through language. This assumption limits applicability in settings

Read the complete text