VLM-R 3 : Region Recognition, Reasoning, and Refinement for Enhanced Multimodal Chain-of-Thought
Chaoya Jiang, Yongrui Heng, Wei Ye, Han Yang, Haiyang Xu, Ming Yan, Ji Zhang, Fei Huang, Shikun Zhang · 2025
Read VLM-R 3 : Region Recognition, Reasoning, and Refinement for Enhanced Multimodal Chain-of-Thought by Chaoya Jiang, Yongrui Heng, Wei Ye, Han Yang, Haiyang Xu, Ming Yan, Ji Zhang, Fei Huang, Shikun Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16192.txt.
Introduction
Chaoya Jiang and co-authors published this study in 2025. It upgrades multimodal models so their reasoning keeps returning to the picture. Text reasoning chains from such models often drift away from visual proof. Their framework, VLM-R3, can decide when it needs another look. It then picks where to look, crops or zooms that region, and weaves the fresh view back into its chain of thought.
Training uses a policy method that rewards picking useful regions and handling them well. A hand-built corpus of interleaved image-text reasoning gives the policy its first lessons. On math-with-figures and science question benches, the system set new best scores in zero- and few-shot play. Gains ran largest on questions needing fine spatial reading. The authors frame region revisiting as the missing habit in visual reasoning.
Catalog record
| CREATOR | Chaoya Jiang, Yongrui Heng, Wei Ye, Han Yang, Haiyang Xu, Ming Yan, Ji Zhang, Fei Huang, Shikun Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-16192
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VLM-R 3 : Region Recognition, Reasoning, and Refinement for Enhanced Multimodal Chain-of-Thought Chaoya Jiang, Yongrui Heng, Wei Ye, Han Yang, Haiyang Xu, Ming Yan, Ji Zhang, Fei Huang, Shikun Zhang Abstract Recently, reasoning-based MLLMs have achieved a degree of success in generating long-form textual reasoning chains. However, they still struggle with complex tasks that necessitate dynamic and iterative focusing on and revisiting of visual regions to achieve precise grounding of textual reasoning in visual evidence. We introduce VLM-R 3 ( V isual L anguage M odel with R egion R ecognition and R easoning), a framework that equips an MLLM with the ability to (i) decide when additional visual evidence is needed, (ii) determine where to ground within the image, and (iii) seamlessly weave the relevant sub-image content back into an interleaved chain-of-thought. The core of our method is Region-Conditioned Reinforcement Policy Optimization (R-GRPO) , a training paradigm that rewards the model for selecting informative regions, formulating appropriate transformations (e.g. crop, zoom), and integrating the resulting visual context into subsequent reasoning steps. To bootstrap this policy, we compile a modest but carefully curated Visuo-Lingual Interleaved Rationale (VLIR) corpus that provides step-level supervision on region selection and textual justification. Extensive experiments on MathVista, ScienceQA, and other benchmarks show that VLM-R 3 sets a new state of the art in zero-shot and few-shot settings, with the largest gains appearing on questions demanding subtle spatial reasoning or fine-grained visual cue extraction. 1 Introduction Multimodal Large Language Models (MLLMs) have recently emerged as a powerful paradigm, demonstrating remarkable capabilities in understanding and generating content across different modalities, primarily vision and language [ 37 , 26 , 22 , 61 , 5 , 7 ] . Models like O1 [ 36 ] , QvQ [ 3 ] , and Gemini 2.5 [ 1 ] have showcased impressive performance on a wide array of tasks such as MMMU [ 62 ] , MathVista [ 29 ] , and ScienceQA [ 30 ] . A key factor contributing to their advanced reasoning abilities is the integration of Chain-of-Thought (CoT) prompting [ 55 ] , which elicits step-by-step reasoning pathways, often leading to more accurate and interpretable outputs. Despite these advancements, a critical limitation persists in the way current MLLMs interact with visual information during complex reasoning processes. Most existing approaches [ 3 , 36 , 57 , 58 ] employing CoT predominantly confine the reasoning steps to the textual domain, with only an initial static grounding in the visual input. This paradigm falls short in scenarios demanding dynamic, iterative, and fine-grained interaction with specific visual regions throughout the reasoning chain. Figure 1: This figure visualizes our proposed VLM-R 3 approach, which integrates region grounding and refinement in an interleaved visual-textual reasoning chain. While conventional text-based reasoning fails when analyzing scenes that require dynamic, iterative, and fine-grained interaction with specific visual regions, our approach succeeds by precisely identifying and focusing on critical visual elements, such as the ’Hongdong Hotpot’ sign in this example, to derive accurate conclusions through targeted visual reasoning. As shown in Figure 1 , examples include sequentially verifying hypotheses against image details, tracking object states across visual cues, or comprehending intricate spatial relationships—all of which require a more active and adaptive visual grounding mechanism. Encouragingly, recent models such as O3 [ 2 ] which capable of interleaving image analysis with text generation—inspire a new frontier where reasoning is not merely conditioned on an image, but is continuously intertwined with ongoing visual perception and localization. Developing an MLLM that can “look again” during reasoning faces two notable hurdles: Region-grounding learning. The model must learn where to focus and how to transform the grounded region (crop, zoom) based on partial textual deliberation. Credit assignment. Simply supervising final answers does not teach the model whether a chosen region actually contributed to correct reasoning, making it hard to refine the visual-query policy. To bridge this crucial gap, we make two primary contributions. First, we introduce Visuo-Lingual Interleaved Rationale (VLIR), a pioneering dataset meticulously curated to support the development of MLLMs for interleaved text-image CoT reasoning. VLIR provides explicit annotations for visual region localization, image cropping instructions, and semantic enhancement cues, all embedded within multi-step reasoning narratives. Second, building upon this, we propose VLM-R 3 ( V isual L anguage M odel with R egion R ecognition and R easoning), a novel framework designed to master this intricate reasoning style. VLM-R 3 is trained using a distinctive strategy that combines cold-start finetuning on our VLIR dataset with a novel Region-Conditioned Reinforcement Policy Optimization (R-GRPO). This empowers VLM-R 3 to learn when and where to look within an image, how to process the localized visual evidence (e.g., by cropping or requesting enhancement), and how to integrate this dynamically acquired information into its evolving reasoning chain. Our extensive experiments on diverse multimodal reasoning benchmarks, including MME [ 14 ] , ScienceQA [ 30 ] and MathVista [ 29 ] , demonstrate that VLM-R 3 significantly outperforms existing state-of-the-art models. In summary, our contributions are: • The introduction of VLIR, the first benchmark dataset tailored for training and evaluating MLLMs on interleaved visual-textual CoT reasoning with explicit region-level interactions. • The proposal of VLM-R 3 , a novel MLLM framework, and its associated R-GRPO training strategy, which enables dynamic visual region localization and evidence integration within the reasoning process. • Comprehensive empirical validation showing that VLM-R 3 achieves superior performance on challenging multimodal reasoning tasks, setting a new benchmark for fine-grained, visually-grounded inference. 2 Related Work 2.1 Large Language Model Reasoning Reasoning in Large Language Models [ 49 , 65 , 63 , 18 ] evolved substantially with Chain of Thought (CoT) prompting [ 48 , 55 , 23 , 54 , 34 , 40 ] , which enables models to break down complex problems into intermediate steps, mimicking human reasoning. This foundational approach has expanded to include diverse structures like program-of-thoughts [ 10 ] , table-of-thoughts [ 19 ] , and tree-of-thoughts [ 60 ] , each offering unique advantages for different reasoning scenarios. Recent advances include OpenAI’s O1 [ 36 ] , which combines reinforcement learning [ 38 , 43 , 16 ] with CoT to optimize decision-making without external guidance, and DeepSeek R1 [ 12 ] , which employs pure reinforcement learning through Group Relative Policy Optimization (GRPO) [ 45 ] to enable autonomous evolution of reasoning capabilities while incorporating rule-based rewards that significantly improve performance across complex reasoning tasks. 2.2 Multi-modal Large Language Model Reasoning Multi-modal Large Language Model reasoning research [ 64 , 56 , 42 , 33 , 39 , 28 , 27 ] has emerged following the success of text-only reasoning models [ 36 , 6 , 12 , 51 ] , focusing on both effective multi-modal chain-of-thought structures [ 57 , 52 , 50 , 20 ] and high-quality training data construction methods [ 13 , 46 , 4 ] . Mainstream approaches have adapted text-based reasoning paradigms to multi-modal contexts, as seen in Virgo [ 13 ] , which demonstrated that text-only reasoning data can activate certain multi-modal reasoning capabilities, and more structured frameworks like LLaVA-CoT’s [ 57 ] four-stage reasoning process and MM-Verify’s [ 50 ] verification-enhanced approach. However, these methods largely inherit