Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMs

Haochen Wang 1,2*, Yuhao Wang 3*, Tao Zhang 4, Yikang Zhou 4, Yanwei Li 5, Jiacong Wang 2, Jiani Zheng 5, Ye Tian 3, Jiahao Meng 3, Zilong Huang 5, Guangcan Mai 5, Anran Wang 5, Yunhai Tong 3, Zhuochen Wang 5, Xiangtai Li 5†, Zhaoxiang Zhang 1,2† · 2025

Read Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMs by Haochen Wang 1,2*, Yuhao Wang 3*, Tao Zhang 4, Yikang Zhou 4, Yanwei Li 5, Jiacong Wang 2, Jiani Zheng 5, Ye Tian 3, Jiahao Meng 3, Zilong Huang 5, Guangcan Mai 5, Anran Wang 5, Yunhai Tong 3, Zhuochen Wang 5, Xiangtai Li 5†, Zhaoxiang Zhang 1,2† (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-18876.txt.

Introduction

Haochen Wang and his fifteen co-authors teach multimodal models to read image regions in context. Models grasp whole scenes well. Dense scenes are weaker ground. A question may target one region among many. Earlier region models read each given region alone. The global context that gives it meaning was ignored.

Their system is called Grasp Any Region. A feature replay technique is aligned to each region of interest. The model can consult the full scene while answering about a part. It models interactions between several prompted regions at once. It moves from passive description to free-form questions about regions. A new benchmark tests single-region understanding. It also tests multi-region interaction and reasoning. The smallest model kept leading captioning performance. It beat far larger rivals on multi-region question tasks.

Catalog record

CREATORHaochen Wang 1,2*, Yuhao Wang 3*, Tao Zhang 4, Yikang Zhou 4, Yanwei Li 5, Jiacong Wang 2, Jiani Zheng 5, Ye Tian 3, Jiahao Meng 3, Zilong Huang 5, Guangcan Mai 5, Anran Wang 5, Yunhai Tong 3, Zhuochen Wang 5, Xiangtai Li 5†, Zhaoxiang Zhang 1,2†
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-18876

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMs Haochen Wang 1,2*, Yuhao Wang 3*, Tao Zhang 4, Yikang Zhou 4, Yanwei Li 5, Jiacong Wang 2, Jiani Zheng 5, Ye Tian 3, Jiahao Meng 3, Zilong Huang 5, Guangcan Mai 5, Anran Wang 5, Yunhai Tong 3, Zhuochen Wang 5, Xiangtai Li 5†, Zhaoxiang Zhang 1,2† Abstract While Multimodal Large Language Models (MLLMs) excel at holistic understanding, they struggle in capturing the dense world with complex scenes, requiring fine-grained analysis of intricate details and object inter-relationships. Region-level MLLMs have been a promising step. However, previous attempts are generally optimized to understand given regions in isolation , neglecting crucial global contexts. To address this, we introduce G rasp A ny R egion ( GAR ) for comprehensive region-level visual understanding. Empowered by an effective RoI-aligned feature replay technique, GAR supports (1) precise perception by leveraging necessary global contexts, and (2) modeling interactions between multiple prompts. Together, it then naturally achieves (3) advanced compositional reasoning to answer specific free-form questions about any region, shifting the paradigm from passive description to active dialogue. Moreover, we construct GAR-Bench , which not only provides a more accurate evaluation of single-region comprehension, but also, more importantly, measures interactions and complex reasoning across multiple regions . Extensive experiments have demonstrated that GAR-1B not only maintains the state-of-the-art captioning capabilities, e.g. , outperforming DAM-3B +4.5 on DLC-Bench, but also excels at modeling relationships between multiple prompts with advanced comprehension capabilities, even surpassing InternVL3-78B on GAR-Bench-VQA. More importantly, our zero-shot GAR-8B even outperforms in-domain VideoRefer-7B on VideoRefer-Bench Q {}^{\text{Q}} , indicating its strong capabilities can be easily transferred to videos. ] 1 NLPR, MAIS, CASIA 2 UCAS 3 PKU 4 WHU 5 ByteDance ∗ Equal Contribution † Corresponding Authors https://github.com/Haochen-Wang409/Grasp-Any-Region https://huggingface.co/HaochenWang/GAR-1B \correspondence , 1 Introduction The ambition of Multimodal Large Language Models (MLLMs) is to endow machines with human-like abilities to perceive, interpret, and reason about the dense visual world [ 60 , 22 , 21 ] . To date, renowned state-of-the-art models [ 1 , 52 , 50 , 10 , 31 , 32 , 33 ] have made remarkable strides, excelling in answering general questions about an entire image. However, this global-level perception struggles with the dense understanding of cluttered environments, intricate object details, and the complex interplay between multiple entities. Figure 1: Performance comparison. GAR achieves strong performances not only on region-level understanding, but also on general multimodal benchmarks. To address the limitation of global perception, several previous works [ 3 , 60 , 64 , 22 , 25 ] argue for a paradigm shift to region-level MLLMs. Specifically, they equip MLLMs with promptable and fine-grained interactions to achieve targeted region-level understanding, using boxes [ 64 , 3 ] or masks [ 60 , 22 ] . This mechanism transforms the model from a passive observer of the entire scene into an active participant capable of deep, localized analysis. Conventional region MLLMs [ 22 , 25 ] mainly focus on the ability to generate a descriptive caption for a single region, and thus model architectures are generally optimized to understand a given region in isolation . This design often neglects crucial global context, e.g. , misidentifying a frog-shaped slipper as a real frog in Figure 2 a. Alternatives [ 60 , 58 ] that employ pooled local features suffer from insufficient details. Therefore, a unified framework that can simultaneously resolve these issues to facilitate more sophisticated and interactive capabilities remains a significant area for investigation. To this end, we propose G rasp A ny R egion ( GAR ) for comprehensive region understanding. As shown in Figure 2 , key features include: Figure 2 : Illustration of our GAR , which is superior at leveraging necessary global context to (a) generate precise captions, where green is correct and red means wrong, (b) model complex interactions among multiple prompts , and perform reasoning such as (c) recognizing non-entities. Colors of <Prompt0> , <Prompt1> , and <Prompt2> correspond to masks with respective colors. Images are sampled from [ 39 ] , [ 23 ] , and [ 30 ] for (a), (b), and (c), respectively. (1) Precise Perception. Thanks to the leverage of necessary global contexts, GAR achieves a more precise perception of given regions, which is the fundamental capability for region MLLMs. As shown in Figure 2 a by aggregating information from the broader, unmasked scene, our GAR manages to generate much more accurate descriptions than previous crop-based approaches [ 22 ] . (2) Interactions between Multiple Prompts. Our framework moves beyond the prevailing single-prompt paradigm, which treats every region of interest as an isolated entity . As illustrated in Figures 2 b and 2 c, GAR manages to model relationships between an arbitrary number of prompts. (3) Advanced Compositional Reasoning Capabilities. Empowered with the aforementioned features, GAR is naturally equipped with advanced compositional reasoning capabilities, allowing it to answer any specific free-form questions, e.g. , recognizing non-entities shown in Figure 2 c. To achieve these capabilities, effectively encoding global contexts becomes equally crucial as local detailed features. To this end, we propose an RoI-aligned feature replay technique. Specifically, GAR first encodes the full, uncropped image (together with the mask prompt) with AnyRes [ 27 ] . Subsequently, RoI-Align [ 16 ] is employed to gather relevant features directly from the global feature map. Those gathered features are inherently context-aware , providing sufficient local details while maintaining global information simultaneously. Please refer to Figure 3 for the detailed pipeline. Furthermore, we introduce GAR-Bench , which not only provides a more accurate evaluation of single-region comprehension by constructing multiple-choice questions, but also, more importantly, measures interaction and complex reasoning across multiple regions . It includes test cases that require a model to aggregate information from multiple visual regions to arrive at a correct conclusion, thereby quantifying the ability to interpret the whole scene rather than independent parts. Empirically, shown in Figure 1 , our GAR-1B not only outperforms DAM-3B [ 22 ] and PAM-3B [ 25 ] on detailed captioning benchmarks [ 22 , 58 , 24 ] , but also excels in general multimodal benchmarks [ 51 , 43 , 53 , 4 ] . Interestingly, it even outperforms large-scale models like InternVL3-78B [ 66 ] on GAR-Bench , demonstrating its advanced comprehension capability in modeling interactions between multiple prompts. More importantly, our zero-shot GAR-8B even outperforms in-domain VideoRefer-7B on VideoRefer-Bench {}^{\text{Q}} , indicating its strong comprehension capabilities can be easily transferred to videos. We hope our work inspires the community to develop MLLMs that can perceive and understand the dense visual world more effectively. 2 Related Works Multimodal Large Language Models (MLLMs). Typical MLLMs [ 26 , 20 , 27 , 1 , 66 , 52 , 19 , 46 , 47 , 42 , 56 , 57 , 19 ] project visual features extracted from pre-trained visual encoders [ 35 , 62 ] to LLM for understanding multimodal contents. However, these models usually lack precise localization capabilities [ 22 , 25 ] and struggle to understand specific regions. One potential solution is to “think with images” [ 33 , 48 , 45 ] . But these agentic models require complex multi-turn conversations,

Read the complete text