GRIT: Teaching MLLMs to Think with Images
Yue Fan, Xuehai He, Diji Yang, Kaizhi Zheng, Ching-Chen Kuo, Yuting Zheng, Sravana Jyothi Narayanaraju, Xinze Guan, Xin Eric Wang · 2025
Read GRIT: Teaching MLLMs to Think with Images by Yue Fan, Xuehai He, Diji Yang, Kaizhi Zheng, Ching-Chen Kuo, Yuting Zheng, Sravana Jyothi Narayanaraju, Xinze Guan, Xin Eric Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15879.txt.
Introduction
Yue Fan and co-authors present GRIT, a method that teaches vision-language models to reason with images, not just words, in a 2025 paper. Most such models write their reasoning as plain text, which can drift away from what the image shows. GRIT has the model point to image regions as it thinks.
Reasoning steps mix sentences with box coordinates that mark the parts of the image being used. Training uses reinforcement learning with rewards for correct answers and proper format, so no hand-labeled reasoning chains are needed. The method is strikingly cheap in data, working with as few as twenty examples. Tests show models trained this way give clearer reasoning that stays tied to the picture.
Catalog record
| CREATOR | Yue Fan, Xuehai He, Diji Yang, Kaizhi Zheng, Ching-Chen Kuo, Yuting Zheng, Sravana Jyothi Narayanaraju, Xinze Guan, Xin Eric Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-15879
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GRIT: Teaching MLLMs to Think with Images Yue Fan, Xuehai He, Diji Yang, Kaizhi Zheng, Ching-Chen Kuo, Yuting Zheng, Sravana Jyothi Narayanaraju, Xinze Guan, Xin Eric Wang Abstract Recent studies have demonstrated the efficacy of using Reinforcement Learning (RL) in building reasoning models that articulate chains of thoughts prior to producing final answers. However, despite ongoing advances that aim at enabling reasoning for vision-language tasks, existing open-source visual reasoning models typically generate reasoning content with pure natural language, lacking explicit integration of visual information. This limits their ability to produce clearly articulated and visually grounded reasoning chains. To this end, we propose Grounded Reasoning with Images and Texts (GRIT), a novel method for training MLLMs to think with images. GRIT introduces a grounded reasoning paradigm, in which models generate reasoning chains that interleave natural language and explicit bounding box coordinates. These coordinates point to regions of the input image that the model consults during its reasoning process. Additionally, GRIT is equipped with a reinforcement learning approach, GRPO-GR, built upon the GRPO algorithm. GRPO-GR employs robust rewards focused on the final answer accuracy and format of the grounded reasoning output, which eliminates the need for data with reasoning chain annotations or explicit bounding box labels. As a result, GRIT achieves exceptional data efficiency, requiring as few as 20 image-question-answer triplets from existing datasets. Comprehensive evaluations demonstrate that GRIT effectively trains MLLMs to produce coherent and visually grounded reasoning chains, showing a successful unification of reasoning and grounding abilities. Figure 1 : Comparison of reasoning with pure natural language and grounded reasoning from GRIT that mixes explicit bounding boxes for image regions with a chain of natural language thoughts. Our GRIT method enables MLLMs to perform grounded reasoning with only 20 training samples, realizing a clear and reliable process of thinking with images. 1 Introduction Reasoning models openai2024o1 ; guo2025deepseek ; yang2024qwen2 ; qwq32b ; dubey2024llama are trained to articulate their problem-solving process through a "reasoning chain" which comprises a detailed, step-by-step deliberation and a final answer. Recent studies have shown that such trained reasoning models are superior in reasoning than models directly prompted in the zero-shot Chain-of-Thought (CoT) wei2022chain manner, where off-the-shelf models are simply guided to "think aloud" at inference time, often reflecting little of their true internal processes chen2025reasoning . Reasoning models first became particularly prominent in the language domain guo2025deepseek ; wei2022chain ; dubey2024llama ; bai2023qwen ; coconut , with models like DeepSeek-R1 guo2025deepseek showcasing capabilities for complex tasks, and subsequently extended to the vision-language field shen2025vlm ; huang2025vision ; yang2025r1 . Current open-source vision reasoning models yield reasoning chains consisting only of natural language when faced with multimodal inputs. This results in reasoning that is less clear and insufficiently grounded in the details of the visual input. To overcome this, a crucial question is how to empower models to think with images, mirroring how humans refer to visual input in their thoughts. However, realizing this capability presents significant technical challenges. Current MLLMs are designed to generate language tokens, lacking the inherent mechanism to generate images directly within a reasoning chain. Furthermore, processing and understanding reasoning sequences that interleave multiple visual elements poses a substantial hurdle, as many MLLMs struggle with maintaining context across numerous images in one input. Beyond these technical challenges, data also presents a major hurdle. In most cases, there is no unique correct reasoning path for complex multimodal questions, and human-annotated data explicitly incorporating visual evidence in reasoning steps is severely scarce. To address these challenges and enable more grounded visual reasoning in MLLMs, we propose the Grounded Reasoning with Images and Text (GRIT) method. GRIT introduces a novel grounded reasoning paradigm where the model generates natural language reasoning chains that freely mix bounding box coordinates pinpointing relevant regions from the input image. These bounding boxes serve to indicate the specific visual information that the model is consulting in its reasoning process. To simplify the inference process, after the generation of bounding box coordinates, the model does not receive additional pixel inputs in the proposed grounded reasoning paradigm; instead, the model comprehends and utilizes the visual information indicated by these coordinates based on its understanding of the original input image. By cropping the input image with the generated bounding boxes, the resulting reasoning chain can be visualized as interleaved text and highlighted regions from the input image, as illustrated in Figure 1 . To train MLLMs to produce reasoning chains in the grounded reasoning paradigm, GRIT employs GRPO-GR, a reinforcement learning method built upon the GRPO algorithm. It is equipped with novel rewards specifically focused on the format of not only reasoning but also grounding, in addition to answer accuracy. Specifically, such format reward encourages reasoning outputs structured by a thinking token pair (e.g., <think> and </think>) and a rethink token pair (e.g., <rethink> and </rethink>); it also rewards the inclusion of syntactically valid bounding boxes within the generated sequence. As a result, the rewards in GRPO-GR do not constrain the specific textual content of the reasoning steps or the semantic accuracy of the grounded regions, thus eliminating the need for data with reasoning chain annotations or explicit bounding box labels. As a result, we find that the GRIT method is extremely data efficient: it enables MLLMs to acquire the grounded reasoning ability with very few data samples sourced from existing VQA datasets using only image-query-answer triplets. With the GRIT method, we train state-of-the-art MLLMs—Qwen 2.5-VL yang2024qwen2 and InternVL 3 zhu2025internvl3 using only 20 image–question–answer triplets drawn from existing object-relation and counting VQA datasets, VSR liu2023visual and TallyQA acharya2019tallyqa . A significant outcome of GRIT is that the trained models preserve their broad versatility, effectively handling not only visual question answering but also grounding-heavy referring expression comprehension tasks. In our experiments with a variety of testing data collected from benchmarks for both VQA and referring expression comprehension, we reveal several key observations. Firstly, the trained models effectively unify the grounding and reasoning abilities—which were originally inherent but disconnected in the base MLLMs—within their grounded reasoning output. Secondly, through both qualitative and quantitative analysis, we reveal a high correlation between the image regions referenced and the accompanying text in the reasoning chain produced by GRIT-trained models. Furthermore, we demonstrate that the generation of bounding boxes boosts the subsequent model reasoning to attend more effectively to the input visual information. Finally, we observe that as training data increases, models trained with GRIT show improved performance but it also reveals challenges for boosting generalizability. Our contributions are as follows: • We propose Grounded Reasoning with Images and Text (GRIT), a novel method that teaches MLLMs to think with images through a grounded reasoning paradigm where models generate reasoning chains interleaving natural language with explicit bounding box coordinates. • We develop GRPO-GR, a reinforcement learning algorithm, which employs novel rewards that enable the grounded reasoning ability of MLLMs efficiently, using only image-question-answer triplets without requiring dense reasoning chains or bounding box annotations. • Through comprehensive evaluations, we demonstrate that MLLMs trained with GRIT, such as Qwen 2.5-VL and InternVL 3, successfully unify their grounding and reasoning abilities to produce accurate and coherent grounded reasoning. 2 Related Work 2.1 Reinforcement Learning for Vision-Language Reasoning Recent studies have applied reinforcement