iVGR: Internalizing Visually Grounded Reasoning for MLLMs with Reinforcement Learning

Chang-Bin Zhang, Yujie Zhong, Qiang Zhang, Kai Han · 2026

Read iVGR: Internalizing Visually Grounded Reasoning for MLLMs with Reinforcement Learning by Chang-Bin Zhang, Yujie Zhong, Qiang Zhang, Kai Han (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-31096.txt.

Introduction

Zhang and colleagues bring grounded thought inside language models with reinforcement learning. Grounded chain-of-thought helps fine detail, but forcing object boxes often hurts results versus plain text thought. They guess that place skills move inside text thought, since forced grounding hurts answer guesses. Their iVGR method moves place skills into text thought via dual-stream training. It lines up text and grounded streams through a shared reward, letting place skills work without grounding. Tests show iVGR beats rivals on fine-grained benchmarks, while open to tool use.

Catalog record

CREATORChang-Bin Zhang, Yujie Zhong, Qiang Zhang, Kai Han
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55184
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.31096

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

iVGR: Internalizing Visually Grounded Reasoning for MLLMs with Reinforcement Learning Chang-Bin Zhang, Yujie Zhong, Qiang Zhang, Kai Han Abstract While visually grounded Chain-of-Thought (CoT) has emerged as a promising paradigm to enhance fine-grained perception in multimodal large language models (MLLMs), its efficacy during the inference phase remains underexplored. In this work, we empirically find that mandating explicit object boxes in visually grounded CoT during inference often degrades performance compared to standard textual CoT, which reasons without explicit visual grounding. We hypothesize that the visual localization capability can be internalized into the textual CoT and that the mandatory explicit grounding introduces unnecessary interference with the model’s primary objective of answer prediction. To address this problem, we propose Internalizing Visually Grounded Reasoning ( iVGR ), a novel reinforcement learning framework that transfers localization capabilities into the textual reasoning process. We employ a dual-stream training strategy, where a textual stream is aligned with a high-quality visually grounded stream via a proposed consistency reward, enabling the model to localize accurately without explicit grounding during inference. Extensive experiments demonstrate that our method significantly outperforms existing baselines on fine-grained benchmarks, while maintaining the flexibility to support tool-assisted inference workflows. Project Page: https://visual-ai.github.io/ivgr/ 1 Introduction Figure 1 : Paradigms of visually grounded reasoning. (a) Tool-based approaches rely on dynamically invoking crop tools to acquire fine-grained visual details. (b) Explicit grounding approaches mandate the generation of bounding boxes interleaved within the CoT, eliminating external tools. (c) iVGR (ours) introduces a dual-stream training strategy. By utilizing a consistency reward to align the textual stream with a high-quality grounded stream, we explicitly internalize localization capabilities into the textual reasoning process. Multimodal large language models (MLLMs) ( 25 ; 35 ; 48 ) have achieved remarkable progress in recent years. While post-training techniques, such as those based on reinforcement learning (RL) ( 32 ; 51 ; 56 ) , significantly enhance their general reasoning capabilities, these models still encounter substantial challenges when processing high-resolution images or complex visual scenes ( 58 ; 43 ) . In such fine-grained scenarios, standard supervision with textual Chain-of-Thought (CoT) often fails to guide MLLMs to localize and focus on critical visual details accurately. To address this, recent studies introduce visually grounded CoT ( 41 ; 58 ) , which explicitly anchors the reasoning process using dynamic crops (see Figure 1 (a)) or bounding boxes (see Figure 1 (b)), aiming to enhance precise visual perception. Specifically, crop-based methods (e.g., DeepEyes ( 58 ) ), as illustrated in Figure 1 (a), require the MLLM to invoke an external tool to crop the target region, subsequently integrating the cropped view with the original context for downstream reasoning. Conversely, box-based methods (e.g., TreeVGR ( 38 ) ), as shown in Figure 1 (b), require the MLLM to generate bounding boxes when referring to objects in the CoT, thereby eliminating the need for external crop tools. However, the actual efficacy of such visually grounded CoT with crops or bounding boxes during inference lacks rigorous scrutiny. To investigate the role of explicit crops or grounding in CoT, we conduct a comparative study using representative SOTA models, DeepEyes ( 58 ) and TreeVGR ( 38 ) . Specifically, we take off-the-shelf models originally trained with visually grounded CoT and evaluate them on multiple Visual Question Answering (VQA) benchmarks. Without any re-training, as reported in Table 1 , we compare their default grounded CoT against a typical textual CoT, which is elicited simply by modifying the inference prompts (see Appendix A ). For both models, inferring with visually grounded CoT outperforms the baseline Qwen2.5-VL-7B model ( 1 ) . However, textual CoT achieves superior performance across most benchmarks, despite the models not being trained for text-only CoT. These results demonstrate that the capability of visually grounded reasoning can be implicitly transferred into typical textual CoT, obviating the need for explicit grounding in CoT. Table 1: Performance comparison between visually grounded CoT and textual CoT. We evaluate off-the-shelf models, DeepEyes ( 58 ) and TreeVGR ( 38 ) , across two reasoning modes: standard textual CoT (‘T’) and visually grounded CoT (‘G’). Benchmarks Qwen2.5-VL-7B DeepEyes-7B TreeVGR-7B CoT T G T G T V* 78.5 82.7 81.7 83.8 84.3 HR4K 69.0 75.1 74.9 77.1 76.9 HR8K 65.1 72.6 73.1 73.1 74.7 MME-RW-Lite 44.5 53.2 53.5 54.9 54.7 POPE 86.3 87.7 89.2 87.3 88.4 RealWorldQA 68.1 69.4 69.7 67.3 69.5 CV-Bench-2D 75.7 75.0 77.9 76.6 77.7 CV-Bench-3D 73.6 77.3 80.8 77.2 79.3 Avg. 70.1 74.1 75.1 74.7 75.7 To better understand the mechanism behind this observation, we examine the relationship between answer accuracy and localization quality (IoU, Intersection-over-Union) of the grounded CoT. Specifically, we manually label target bounding boxes for HR8K ( 43 ) and group the questions based on the IoU of the generated grounded CoT. We then calculate the accuracy for each IoU interval, as illustrated in Figure 2 . For DeepEyes ( 58 ) (see Figure 2(a) ), visually grounded CoT with crops outperforms textual CoT when the crops are of high quality (i.e., IoU > 0.5). Conversely, when crop localization is poor, grounded CoT underperforms textual CoT. We hypothesize that inaccurate crops introduce visual noise, which negatively impacts downstream answer prediction. For TreeVGR ( 38 ) (see Figure 2(b) ), however, visually grounded CoT consistently underperforms textual CoT across all IoU intervals. We hypothesize that the visual localization capability can be internalized into the textual CoT and that the mandatory task of explicit grounding imposes unnecessary interference, which detracts from the model’s primary focus on answer prediction. (a) DeepEyes (b) TreeVGR Figure 2 : Relationship between accuracy and localization quality (IoU). Using HR8K ( 43 ) , questions are grouped based on the IoU of the generated grounded CoT, and accuracy is calculated for each IoU interval. Building upon this insight, we propose iVGR , a reinforcement learning-based dual-stream training strategy . For each training query, the policy MLLM generates two distinct streams of rollouts: a grounded stream that incorporates explicit box predictions, and a textual stream that performs standard reasoning. To transfer the grounding capability across streams, we introduce a novel consistency reward that aligns the textual CoT with high-quality grounded reasoning trajectories selected from the grounded stream. In this way, the model internalizes localization into its textual reasoning without producing explicit coordinates at inference. Moreover, iVGR remains compatible with explicit crop tools at test time, and we further design a test-time workflow that exploits this synergy on fine-grained VQA. Extensive experiments on Qwen2.5-VL and Qwen3-VL demonstrate that iVGR yields significant improvements across multiple benchmarks. Our contributions are summarized as follows: • We analyze the efficacy of visually grounded reasoning, revealing that this capability can be implicitly transferred into typical textual CoT, thereby obviating the need for explicit grounding outputs during test time. • Building on this insight, we propose iVGR , a reinforcement-learning-based dual-stream training strategy. By introducing a novel consistency reward, our method explicitly transfers the visually grounded reasoning capability into the textual reasoning process. • We conduct extensive experiments on Qwen2.5-VL and Qwen3-VL models. The results demonstrate that our models achieve significant improvements across multiple benchmarks and maintain compatibility with tool-assisted workflows during test time. 2 Related Work Reasoning MLLMs. Chain-of-Thought (CoT) ( 45 ; 47 ; 6

Read the complete text