VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use

Mingyuan Wu, Jingcheng Yang, Jize Jiang, Meitang Li, Kaizhuo Yan, Hanchao Yu, Minjia Zhang, Chengxiang Zhai, Klara Nahrstedt · 2025

Read VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use by Mingyuan Wu, Jingcheng Yang, Jize Jiang, Meitang Li, Kaizhuo Yan, Hanchao Yu, Minjia Zhang, Chengxiang Zhai, Klara Nahrstedt (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19255.txt.

Introduction

Mingyuan Wu and co-authors train vision-language models to reason with pictures, not just about them, by letting the model edit images mid-thought. Standard training produces text-only reasoning over a fixed image. Their framework weaves Python image-editing tools into reinforcement learning, so the model learns when a visual step helps.

Rewards depend only on final answer accuracy, with no supervision of the reasoning steps themselves. On chart and table question answering, the trained models learned to generate intermediate visual steps that improved their answers. The authors call these multimodal chains of thought and release their code. They hope it seeds research on multi-step visual reasoning.

Catalog record

CREATORMingyuan Wu, Jingcheng Yang, Jize Jiang, Meitang Li, Kaizhuo Yan, Hanchao Yu, Minjia Zhang, Chengxiang Zhai, Klara Nahrstedt
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-19255

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use Mingyuan Wu, Jingcheng Yang, Jize Jiang, Meitang Li, Kaizhuo Yan, Hanchao Yu, Minjia Zhang, Chengxiang Zhai, Klara Nahrstedt Abstract Reinforcement learning finetuning (RFT) has significantly advanced the reasoning capabilities of large language models (LLMs) by enabling long chains of thought, multi-turn self-correction, and effective tool use. While recent works attempt to extend RFT to vision-language models (VLMs), these efforts largely focus on text-only reasoning conditioned on original image inputs, and do not incorporate visual reasoning in the response. In contrast, test-time methods like Visual Sketchpad incorporate visual steps but lack training mechanisms. We introduce VTool-R1 , the first RFT framework that trains VLMs to generate multimodal chains of thought by interleaving text and intermediate visual reasoning steps. VTool-R1 integrates Python-based visual editing tools into the RFT process, enabling VLMs to learn when and how to generate visual reasoning steps that enhance the final output quality. Trained with outcome-based rewards, our approach elicits strategic visual tool use for multi-modal reasoning without relying on process-based supervision. Extensive experiments on structured visual reasoning over charts and tables show that VTool-R1 enhances reasoning performance by teaching VLMs to "think with images" and generate multimodal chain of thoughts with tools. To support future research in multi-turn multi-modal reasoning, we open-source our code at https://github.com/VTOOL-R1/vtool-r1 . 1 Introduction Recent large language models (LLMs), notably DeepSeekR1 ( DeepSeek-AI, 2025 ) and the GPT4o series ( OpenAI, 2024 ) , have demonstrated remarkable capabilities in text-based reasoning. Central to this advancement is Reinforcement learning finetuning (RFT), which enables these models to learn to generate long chains of thought and engage in self-correction and verification for complex reasoning tasks ( Kumar et al., 2025 ; Zeng et al., 2025 ) . Beyond boosting intrinsic reasoning capabilities, RFT has also shown promising results in effectively integrating external tool use, such as search engines ( Jin et al., 2025 ; Chen et al., 2025b ) and code interpreters ( Feng et al., 2025 ) , into the reasoning process of LLMs. Despite this rapid progress in LLM reasoning brought by RFT, there has not yet been a well-recognized breakthrough in improving multimodal reasoning capabilities of vision-language models (VLMs). Modern VLMs ( Bai et al., 2025 ) typically consist of a strongly language-aligned image encoder that maps visual inputs into feature space, then a connector module further projects these visual features into the token space of a decoder-only LLM. On top of these architectures, prior works have attempted to replicate post-training strategies first developed on LLMs, like visual instruction tuning ( Xu et al., 2024 ) , to improve VLMs’ ability to follow textual instructions. However, similar attempts by recent works ( Zhou et al., 2025 ; Chen et al., 2025a ; Zhang et al., 2025 ; Huang et al., 2025 ; Liu et al., 2025 ; Deng et al., 2025 ) to replicate the success of RFT on LLMs onto the VLM domain to enhance multimodal reasoning fall short: these reasoning approaches remain fundamentally text-driven : they handle images only during the initial encoding stage and generate reasoning chains purely in text conditioned on fixed input image tokens, without involving any intermediate visual reasoning steps. Why is text-dominant reasoning insufficient? Even the most advanced VLMs may rely on language shortcuts if text dominates reasoning. For example, consider a prompt where a user shows GPT-5 ( OpenAI, 2025 ) an image of a hand with six fingers and asks, “How many fingers are in this hand?” The model may respond with “five” with high chance, based on the pure text reasoning path —“a hand has five fingers”. This illustrates a critical failure of purely language-based reasoning in multimodal settings: the model can decode a more plausible answer through wrong textual chain of thoughts shortcuts. To go beyond text-based reasoning, early inference frameworks like Visual Sketchpad ( Hu et al., 2024 ) and Refocus ( Fu et al., 2025 ) demonstrate that incorporating intermediate visual reasoning steps during inference can improve multimodal reasoning performance. However, these inference-time methods rely on the usage of highly capable models such as GPT-4o to produce meaningful visual steps, and underperform when used with open-source, less capable models. In this paper, we present the first RFT frameworks that directly enables VLMs to learn to think in a combination of images and texts and to be trained to generate multimodal chains of thoughts. Our framework, called VTool-R1 , teaches VLMs to generate intermediate visual reasoning steps through interactions with external image editing tools implemented in Python code. These visual reasoning steps are interleaved with textual chains of thought, resulting in multi-turn multimodal reasoning steps during the generation of the response. Following the design of DeepSeekR1, VTool-R1 uses only outcome-based rewards—the model is not explicitly rewarded for generating visual steps, but instead learns when and how to incorporate visual reasoning steps in order to improve final task performance. We demonstrate the effectiveness of VTool-R1 on challenging structured image reasoning tasks, focusing on table and chart-based reasoning. Our experiments use a well-curated dataset and a visual editing tool set from Refocus. This toolset, which is directly callable by the VLM, enables selective attention on relevant regions in the table and chart - simulating how humans process visual information through attention before forming final reasoning results. Through VTool-R1, the model learns to use these tools strategically to guide its multimodal chain of thought and enhance final reasoning performance. Our key contributions can be summarized as follows: • We present VTool-R1, a novel RFT framework that supports VLM multimodal reasoning with external visual editing tool use. We demonstrate that RFT with outcome based reward design can incentive generation of visual reasoning steps for final reasoning accuracy. • To the best of our knowledge, our work is the first work that successfully enables VLMs to learn to integrate intermediate visual reasoning steps into text-based chains of thoughts in the generated response (i.e. thinking with a combination of images and text). • We validate our approach through extensive controlled experiments on challenging and well-established structured image reasoning datasets, using a predefined visual toolset in Refocus dataset. 2 Related Works 2.1 Visual Chain of Thought Reasoning Early works have demonstrated that incorporating visual intermediate steps—often generated via external tools or Python scripts—can benefit a wide range of visual question answering (VQA) tasks, without any training. Pioneering efforts such as ViperGPT ( Surís et al., 2023 ) and Visual Programming without Training ( Gupta & Kembhavi, 2023 ) utilize Python-based visual tools to manipulate images during inference. Cache of Thought ( Wu et al., 2025 ) further improves multimodal reasoning by retrieving high-quality rationales from similar queries. Recently, Visual Sketchpad ( Hu et al., 2024 ) introduced a framework that equips multimodal language models with a sketchpad and drawing tools. The model is prompted to generate visual artifacts during inference and uses them for iterative planning and reasoning. While this approach successfully introduces visual information into the reasoning process, it operates solely at inference time, with no training involved. Refocus ( Fu et al., 2025 ) takes a step forward by prompting the VLM to invoke visual editing tools for selective attention over

Read the complete text