MedVistaGym : A Scalable Training Environment for Thinking with Medical Images via Tool-Integrated Reinforcement Learning
Meng Lu, Yuxing Lu, Yuchen Zhuang, Megan Mullins, Yang Xie, Guanghua Xiao, Charles Fleming, Wenqi Shi, Xuan Wang, [0.5em] Virginia Tech · 2026
Read MedVistaGym : A Scalable Training Environment for Thinking with Medical Images via Tool-Integrated Reinforcement Learning by Meng Lu, Yuxing Lu, Yuchen Zhuang, Megan Mullins, Yang Xie, Guanghua Xiao, Charles Fleming, Wenqi Shi, Xuan Wang, [0.5em] Virginia Tech (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-07107.txt.
Introduction
Meng Lu and co-authors posted a 2026 arXiv preprint. The preprint deals with MedVistaGym. It came out on arXiv in 2026 as a PAPERS preprint. Vision language models (VLMs) achieve strong performance on general image understanding but struggle to think with medical images.
Especially when performing multi-step reasoning through iterative visual interaction. Medical VLMs often rely on static visual embeddings and single-pass inference, preventing models from re-examining, verifying, or refining visual evidence during reasoning. Readers in the field get a short, focused account of the problem and the steps the authors take.
Catalog record
| CREATOR | Meng Lu, Yuxing Lu, Yuchen Zhuang, Megan Mullins, Yang Xie, Guanghua Xiao, Charles Fleming, Wenqi Shi, Xuan Wang, [0.5em] Virginia Tech |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 92 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-07107
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MedVistaGym : A Scalable Training Environment for Thinking with Medical Images via Tool-Integrated Reinforcement Learning Meng Lu, Yuxing Lu, Yuchen Zhuang, Megan Mullins, Yang Xie, Guanghua Xiao, Charles Fleming, Wenqi Shi, Xuan Wang, [0.5em] Virginia Tech Abstract Vision language models (VLMs) achieve strong performance on general image understanding but struggle to think with medical images , especially when performing multi-step reasoning through iterative visual interaction. Medical VLMs often rely on static visual embeddings and single-pass inference, preventing models from re-examining, verifying, or refining visual evidence during reasoning. While tool-integrated reasoning offers a promising path forward, open-source VLMs lack the training infrastructure to learn effective tool selection, invocation, and coordination in multi-modal medical reasoning. We introduce MedVistaGym , a scalable and interactive training environment that incentivizes tool-integrated visual reasoning for medical image analysis. MedVistaGym equips VLMs to determine when and which tools to invoke, localize task-relevant image regions, and integrate single or multiple sub-image evidence into interleaved multimodal reasoning within a unified, executable interface for agentic training. Using MedVistaGym , we train MedVista-R1 to interleave tool use with agentic reasoning through trajectory sampling and end-to-end reinforcement learning. Across six medical VQA benchmarks, MedVista-R1 -8B exceeds comparably sized tool-augmented baselines by 19.10 to 24.21%, demonstrating that structured agentic training—not tool access alone—unlocks effective tool-integrated reasoning for medical image analysis. 1 Introduction Vision language models (VLMs) have achieved remarkable progress on medical image understanding, demonstrating strong performance across visual question answering (VQA) 7 , disease diagnosis 27 , diagnostic report generation 14 ; 45 , and agentic medical visual analysis 16 . Much of this progress can be attributed to text-based reasoning paradigms such as Chain-of-Thought (CoT) ( 43 ) and reinforcement learning (RL) ( 15 ) , which decompose complex problems into intermediate reasoning steps and improve reasoning quality through outcome-based optimization ( 28 ; 19 ; 49 ) . Together, these advances have moved medical VLMs beyond direct prediction toward more structured, step-by-step clinical reasoning. Despite these advances, current medical VLMs remain limited in interactive visual reasoning, i.e., the ability to iteratively acquire, verify, and refine visual evidence during multi-step inference. Existing approaches largely rely on static visual embeddings and shallow cross-modal alignment, causing models to attend to irrelevant anatomy while overlooking fine-grained diagnostic cues such as blurred lesion boundaries, low-contrast abnormalities, and subtle tissue textures 50 ; 10 . These limitations stem not from model capacity but from static, single-pass reasoning frameworks that lack mechanisms for continuous image interaction. As perception and decision-making occur in one forward pass, models cannot re-observe, verify, or refine visual evidence during reasoning, preventing dynamic visual exploration. Consequently, existing approaches fall short of true “ thinking-with-images ” 36 , where reasoning is tightly coupled with iterative visual perception in complex clinical scenarios. To bridge this gap, recent work has explored tool-integrated reasoning (TIR) ( 17 ; 50 ; 29 ; 46 ) , which augments VLMs with external tools to support fine-grained visual reasoning. However, current open-source VLMs still struggle to leverage these tools effectively for medical reasoning tasks, largely due to the absence of a training environment that allows agents to learn dynamic tool selection and coordinated multi-step interaction through experience. Although some studies explore tool use in specific medical settings ( 24 ; 11 ) , systematic simulation and training of tool-integrated thinking for medical visual reasoning remain largely unexplored. Motivated by these challenges, we introduce MedVistaGym ( Med ical Vis ual-centric T ool-integrated A gentic training environment), a scalable and interactive training environment designed to operationalize tool-integrated thinking for medical visual reasoning. MedVistaGym encapsulates diverse visual tool operations and provides structured textual feedback, enabling closed-loop interaction between reasoning and perception (Figure 1 ). It supports (i) multimodal medical reasoning across six public datasets, (ii) a unified and extensible interface to fifteen visual and medical knowledge tools, and (iii) scalable infrastructure for efficient agent training and evaluation. Building upon MedVistaGym , we develop MedVista-R1 , a VLM-based agent for robust tool-augmented medical image reasoning. Across three in-domain and three out-of-domain medical VQA benchmarks, MedVista-R1 -8B delivers consistent gains over comparably sized open-source baselines under both tool-enabled and tool-free settings, validating MedVistaGym as a scalable environment for training agentic VLMs. 2 MedVistaGym Figure 1: Overview of MedVistaGym , which contains a comprehensive suite of reasoning-intensive medical image analysis tasks and tools in an interactive execution environment, scaling visual-centric tool-integrated agentic reinforcement learning for VLM agents. 2.1 Problem Formulation We formulate tool-integrated medical image analysis as a Partially Observable Markov Decision Process (POMDP). Given a medical question with associated image(s), an agent interacts with an executable environment under instruction space \mathcal{I} and receives partial observations \mathcal{O} . Here \mathcal{S} denotes latent environment states, \mathcal{A} the set of executable tool actions, and \mathcal{P}:\mathcal{S}\times\mathcal{A}\rightarrow\mathcal{S} the deterministic state transition function. At each time step, the agent generates an explicit reasoning step g_{t} to guide decision-making, then selects a tool action a_{t}\in\mathcal{A} . The environment executes the action and returns a new observation as external evidence. This process yields an interaction trajectory U=(g_{0},a_{0},\ldots,g_{T},\hat{y}) , where \hat{y} is the final answer after T interaction steps. 2.2 Environment Design Following the formulation in § 2.1 , we instantiate the agent-environment interaction as an executable system. MedVistaGym is a scalable environment for multi-turn, tool-integrated medical visual reasoning. It defines the agent’s operating world, i.e., tasks, tools, interaction interfaces, and execution infrastructure, independent of any specific training algorithm. A key property of MedVistaGym is its support for online and adaptive tool invocation : each tool call at time t returns a real observation that conditions the agent’s next action at time t+1 . As a result, tool execution emerges dynamically from agent-environment interaction, rather than being scripted in advance. 2.3 Tasks and Tool Sets Medical VQA Tasks. MedVistaGym comprises a set of verifiable medical VQA tasks that demand grounded, multi-step reasoning over visual inputs and intermediate evidence. These tasks span diverse diagnostic scenarios, including clinical perception, lesion-level evidence localization, subtle abnormality detection, and diagnosis-oriented evidence aggregation, where generating reliable answers requires calling external tool support. The training data in MedVistaGym is organized along two complementary axes. (1) Radiology VQA , covering cross-sectional and projection imaging, includes VQA-RAD 22 , which focuses on anatomy and finding recognition in X-ray, CT, and MRI images, and SLAKE 26 , a knowledge-aware dataset with clinically grounded questions over diverse radiology images. (2) Pathology VQA , covering microscopy and histopathology, includes PathVQA 18 , which emphasizes cellular morphology and tissue patterns. More details are provided in Appendix A.2 . Visual-Centric Tools Sets. To support grounded medical image verifiable analysis, MedVistaGym exposes a standardized and extensible suite of executable tools that enable agents to offload perception, localization, segmentation, and knowledge retrieval. These tools return structured outputs that can be directly used as intermediate evidence during reasoning. The tools used in MedVistaGym are organized into four complementary families: (1) Resolution and Region Refinement , which enable focused inspection of image regions (e.g., agent4k , zoom-in ); (2) Medical Localization and Segmentation support the detection and delineation of anatomical or pathological regions (e.g., groundingdino , medsam2 ); (3) Medical Visual Understanding and Parsing , providing a structured interpretation of medical images (e.g., biomedclip , biomedparse ); (4) External Biomedical Knowledge Retrieval