Referring Multiple Regions with Large Multimodal Models via Contextual Latent Steering
Yun Xing, Hanyuan Liu, Jiahao Nie, Shijian Lu · 2026
Read Referring Multiple Regions with Large Multimodal Models via Contextual Latent Steering by Yun Xing, Hanyuan Liu, Jiahao Nie, Shijian Lu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-01827.txt.
Introduction
Yun Xing, Hanyuan Liu, Jiahao Nie, and Shijian Lu teach general multimodal models to point at several regions at once, guided by visual prompts, with no retraining. Big models read whole images well but falter at region-level referring, above all when many regions are named together or the global scene must arbitrate which patch is meant.
Contextual Latent Steering pre-computes vectors that capture referring behavior itself: telling regions apart, weighing global context. At inference it edits the model's internal representations along those vectors, steering attention without touching weights or architecture. Across multiple data sets, steered general models beat purpose-built referring models in most comparisons, setting new marks for the task. The recipe's appeal is its cheapness: behavior harvested once from the model's own geometry, then applied as a nudge wherever referring is needed.
Catalog record
| CREATOR | Yun Xing, Hanyuan Liu, Jiahao Nie, Shijian Lu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.01827
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Referring Multiple Regions with Large Multimodal Models via Contextual Latent Steering Yun Xing, Hanyuan Liu, Jiahao Nie, Shijian Lu Abstract Large Multimodal Models (LMMs) have recently demonstrated their proficiency in holistic visual comprehension. However, most of them struggle to tackle region-level perception guided by visual prompts, especially for cases where multiple regions are referred simultaneously, or scenarios where global contexts are necessary for precise visual referring. We introduce Contextual Latent Steering (CSteer) , a training-free approach for guiding general LMMs to refer multiple regions contextually, without expensive fine-tuning or architectural modifications. CSteer starts with pre-computing contextual vectors that implicitly represent visual referring behaviors, such as differentiation among regions and attention to global contexts, followed by representation editing during inference time. Experimental results on multiple datasets indicate that general LMMs with CSteer outperform tailored referring LMMs in most cases, suggesting a promising solution in training-free, and setting new state-of-the-art for this field. Code is available at https://github.com/xing0047/csteer.git . 1 Introduction Large Multimodal Models (LMMs) ( 31 ; 14 ; 5 ; 61 ) have demonstrated their proficiency in general image and video understanding, such as visual question answering ( 25 ; 26 ) or describing scenes in detail ( 7 ; 35 ) . Although existing LMMs excel in holistic visual understanding, most of them struggle to perceive and reason over localized regions, known as visual referring ( 58 ; 29 ) . This typically requires precise interpretation at the instance-level ( 44 ) , thus powering a wide range of applications with more nuanced granularity, such as robotics ( 73 ) or remote sensing ( 82 ) . Figure 1 : An example from ( 44 ) demonstrating multi-region visual referring , with a wide spectrum of region-focused scenarios and instance-level cognition. There have been ongoing explorations in fine-tuning LMMs that excel at visual referring ( 70 ; 80 ; 74 ; 6 ; 46 ; 76 ) , which are recently capable of describing nuanced details for small targets ( 29 ) , or capturing temporal dynamics for objects in videos ( 75 ) . A common practice resorts to customized designs, such as employing another encoder for explicit tokenization of input regions ( 46 ; 81 ; 72 ; 29 ; 75 ) , together with tailored fine-tuning data ( 29 ) or post-training recipe ( 58 ) . Figure 2 : LMMs know where to look when referring one region ( 78 ) but struggle when multiple regions are visually referred. We use Qwen3-VL-8B ( 5 ) for inference of the examples, where orange means regions not referred. Although these studies bring advances to visual referring , most of them focus on perceiving one isolated region, while multi-region visual referring is less explored ( 58 ; 44 ) . This examines how LMMs misperceive when multiple regions are referred simultaneously. Multi-region visual referring naturally involves two key aspects, as given in Fig. 1 . One is more region-centric , usually involving comparisons among regions, such as assessing orientations and depths ( 16 ; 56 ) . Another is more contextual , which requires precise global understanding before referring accurately, such as counting or localized captioning ( 44 ) , as in Fig. 1 . Instead of introducing customized designs for multi-region visual referring and expensive task-oriented instruction tuning, we explore how to trigger such capability from general LMMs in a training-free manner ( 68 ) . General LMMs like Qwen3-VL ( 5 ) are able to perceive visual marks naturally, such as points, boxes, and masks with numerical identifiers. Moreover, these LMMs know where to look ( 78 ) with simple one-region referring cases, as in Fig. 2 (top). Despite its simplicity, Set-of-Mark ( 68 ) over general LMMs is not necessarily effective for multi-region cases, especially for contextual ones that require precise global understanding ( e.g. , Fig. 2 bottom, where attentions ( 78 ) are not comparably faithful like one-region cases). Motivated by ( 48 ) , we propose Contextual Latent Steering (CSteer) , a training-free solution that triggers multi-region visual referring directly from general LMMs. CSteer pre-computes contextual vectors that implicitly encode behaviors of multi-region visual referring , such as differentiation among regions and more attention to global contexts, followed by applying them to the latent representations during inference. Contextual vectors capture underlying changes when a LLM judge rewrites erroneous localized captions from general LMMs, such as rectifying mismatched numerical identifiers or complementing missing regions ( e.g. , Fig. 2 ). Empirically, CSteer on the most recent general LMMs ( e.g. , Qwen3-VL ( 5 ) ) outperforms state-of-the-art tailored referring LMMs ( 44 ; 29 ) on multiple representative benchmarks ( 58 ; 44 ; 16 ) , including the recent ones that require precise contextual referring ( 58 ) , indicating its strong potential in addressing multi-region scenarios. We summarize our contributions as follows, • We propose Contextual Latent Steering (CSteer), a simple yet effective training-free approach that enables general LMMs to visually refer with multiple interactive prompts simultaneously, without tailored architecture, expensive supervised fine-tuning or policy optimization. • We perform thorough ablations, suggesting multiple design choices for both generating contextual vectors and steering. We emphasize our approach by vectoring with referential rewrites and steering with decomposition. • On top of the recent general LMMs, CSteer achieves the state-of-the-art over multiple visual referring benchmarks, such as GAR-Bench and INST-IT characterized by multi-region and contextual referring. 2 Related Works Large Multimodal Models . Recently, Large Multimodal Models (LMMs) ( 31 ; 5 ; 14 ; 61 ; 9 ; 24 ; 18 ) have achieved remarkable milestones, enabling applications such as agents ( 66 ; 84 ) or embodied intelligence ( 22 ; 85 ) . Early LMM works ( 31 ; 14 ) address perceptual tasks ( 10 ; 33 ; 26 ) , while recently more studies ( 59 ; 51 ) focus on tackling complex reasoning scenarios ( 65 ; 12 ) . Although existing LMMs excel in the global understanding of images and videos ( 77 ; 39 ) , they struggle to describe and reason at a finer-grained region-level ( 29 ; 76 ) . Figure 3 : Contextual Vector . ( left ) CSteer first obtains incorrect localized captions and corrected referential rewrites with the aid of a text-only judge, and ground-truth captions as reference. ( right ) The incorrect captions and their paired rewrites then derive positive and negative last tokens via teacher forcing (pos/neg tokens), which are contrasted and averaged across data for building contextual vectors. Visual Referring . Visual referring has long been developed to empower LMMs with region-level cognition capabilities ( 9 ; 70 ; 74 ; 79 ; 46 ; 81 ; 30 ; 29 ; 72 ; 54 ) . Specifically, they often involve an additional region encoding module to obtain local region-centric features ( 79 ; 46 ; 76 ; 29 ) , with a large collection of region-referred data used for instruction tuning ( 74 ) or reinforcement post-training ( 58 ) . Some studies ( 6 ; 44 ) train LMMs to recognize visual prompts ( 68 ) with customized referential dialogs instead. Most existing referring LMMs are