Step-CoT: Stepwise Visual Chain-of-Thought for Medical Visual Question Answering
Lin Fan, Yafei Ou, Zhipeng Deng, Pengyu Dai, Hou Chongxian, Jiale Yan, Yaqian Li, Kaiwen Long, Xun Gong, Masayuki Ikebe, Yefeng Zheng · 2026
Read Step-CoT: Stepwise Visual Chain-of-Thought for Medical Visual Question Answering by Lin Fan, Yafei Ou, Zhipeng Deng, Pengyu Dai, Hou Chongxian, Jiale Yan, Yaqian Li, Kaiwen Long, Xun Gong, Masayuki Ikebe, Yefeng Zheng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13878.txt.
Introduction
Fan and co-authors build Step-CoT, a dataset that teaches medical vision models to reason in clear steps. Most chain-of-thought data for medical questions is free-form and skips the structure clinicians use. Step-CoT covers over 10,000 real cases and 70,000 question pairs, laid out along diagnostic workflows. A teacher-student setup with a dynamic graph focus picks out the most useful steps. Experiments show gains in both answer accuracy and how readable the reasoning is.
Catalog record
| CREATOR | Lin Fan, Yafei Ou, Zhipeng Deng, Pengyu Dai, Hou Chongxian, Jiale Yan, Yaqian Li, Kaiwen Long, Xun Gong, Masayuki Ikebe, Yefeng Zheng |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 86428 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.13878
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Step-CoT: Stepwise Visual Chain-of-Thought for Medical Visual Question Answering Lin Fan, Yafei Ou, Zhipeng Deng, Pengyu Dai, Hou Chongxian, Jiale Yan, Yaqian Li, Kaiwen Long, Xun Gong, Masayuki Ikebe, Yefeng Zheng Abstract Chain-of-thought (CoT) reasoning has advanced medical visual question answering (VQA), yet most existing CoT rationales are free-form and fail to capture the structured reasoning process clinicians actually follow. This work asks: Can traceable, multi-step reasoning supervision improve reasoning accuracy and the interpretability of Medical VQA? To this end, we introduce Step-CoT, a large-scale medical reasoning dataset with expert-curated, structured multi-step CoT aligned to clinical diagnostic workflows, implicitly grounding the model’s reasoning in radiographic evidence. Step-CoT comprises more than 10K real clinical cases and 70K VQA pairs organized around diagnostic workflows, providing supervised intermediate steps that guide models to follow valid reasoning trajectories. To effectively learn from Step-CoT, we further introduce a teacher-student framework with a dynamic graph-structured focusing mechanism that prioritizes diagnostically informative steps while filtering out less relevant contexts. Our experiments show that using Step-CoT can improve reasoning accuracy and interpretability. Benchmark: github.com/hahaha111111/Step-CoT Dataset Card: huggingface.co/datasets/fl-15o/Step-CoT Table 1 : Comparison of recent medical reasoning datasets with Step-CoT. Dataset Instances CoT Construction Expert Involvement Clinical Workflow PMC-VQA 47 227K \times No CoT \checkmark Yes \times No Med-GRIT-270k 19 270k QA pairs \checkmark GPT-generated \times No experts \times No MedCoT 26 50k Extends Med-VQA \checkmark Expert validation \checkmark Yes \checkmark Partial ReasonMed 3 370k reasoning samples \checkmark Agent-generated \checkmark Multi-agent validation \times No HVCR 11 31k QA pairs Expert validation \checkmark Yes \times No V2T-CoT 41 39k examples \checkmark GPT-generated CoTs \times No experts \times No MedThink 15 50k Extends Med-VQA \checkmark Decision-making rationales \checkmark Semi-auto + human validation \times No Step-CoT (Ours) 70K QA pairs with reasoning \checkmark Structured multi-step \checkmark Semi-auto + human validation \checkmark Yes 1 Introduction Medical Visual Question Answering (Med-VQA) has emerged as a critical topic in healthcare AI, leveraging multi-modal deep learning to answer clinical questions related to images posed in natural language 25 ; 7 ; 13 ; 14 ; 6 ; 12 . Leveraging the ability to generate coherent responses while incorporating extensive medical domain knowledge, Med-VQA has demonstrated practical utility across diverse tasks such as computer-aided diagnosis and fine-grained tumor attribute recognition 25 ; 45 ; 16 . Recent advances in Med-VQA models have evolved from focusing on performance enhancement through scaling model architectures and expanding pre-training datasets 44 , such as ExGra-Med 29 , LLaVA-Med 20 , MedGemma 34 , and LLaVA-Tri 42 , to improving interpretability by decomposing complex diagnostic tasks into step-by-step reasoning processes through CoT mechanisms 48 ; 43 , including ReasonMed 3 , HVCR 11 and MedCoT 26 . By explicitly revealing their intermediate reasoning steps, CoT methods enhance both predictive accuracy and interpretability, rendering them particularly valuable for high-stakes domains such as healthcare. Recent studies have explored automatic generation of CoT reasoning data for Large vision language models (LVLMs) to improve reasoning accuracy and enhance interpretability. Approaches such as MedCoT 26 , MedThink 15 , ReasonMed 36 , and the HVCR 11 provide textual reasoning traces that enable models to produce rationales alongside predictions. To further connect reasoning with visual evidence, datasets like V2T-CoT 41 , Med-GRIT-270k 19 , and MedTrinity-25M 42 pair CoT statements with image annotations (as shown in Table 1 ). Although these approaches improve data availability, their effectiveness in modeling clinical reasoning remains limited by two major issues: (i) These datasets lack a structured, stepwise diagnostic protocol. They either provide free-form rationales or automatically generated reasoning chains that fail to align with real clinical workflows and omit intermediate diagnostic states reflecting radiologists’ sequential decision-making. (ii) Most CoT datasets rely heavily on GPT-4.1-based synthetic rationales derived from existing image-text pairs, raising serious concerns about factual inconsistency. Beyond dataset construction, training paradigms for CoT in LVLMs further enlarge this gap. Most CoT training paradigms for LVLMs, which rely on Supervised Fine-Tuning (SFT) or Reinforcement Learning (RL), are inherently non-interactive and perceptually static 50 ; 5 , primarily using a static image plus question input and reporting final-answer outputs without explicit action-level reasoning chains 20 , limiting the model to the initial input image and preventing it from actively gathering new information or refining its perception. For example, BLIP - 2 21 has achieved strong open-domain VQA performance but lacks mechanisms for interactive perceptual refinement or multi-step visual reasoning, while emerging specialized medical LVLMs such as LLaVA - Med 20 demonstrate domain adaptation but still without action-level reasoning chains 20 . Similarly, the recent MedVLM - R1 30 framework uses RL to incentivize reasoning paths in medical image analysis. Yet, the perceptual input remains static, and the model cannot execute intermediate actions that change what it perceives. These limitations underscore the need for a structured reasoning annotation framework that not only integrates visual reasoning and aligns multi-step inference with multi-modal clinical evidence, but also enables traceable, stepwise reasoning interactions with visual data, allowing each reasoning step to dynamically update and refine diagnostic understanding in a clinically consistent manner. This leads to the central question of this work: Can traceable, multi-step reasoning supervision improve reasoning accuracy and the interpretability of Medical VQA? To solve this problem, we introduce Step-CoT, a novel medical dataset for multi-step vision-language reasoning. Step-CoT organizes clinical diagnostic workflows with clinician-curated intermediate reasoning steps, providing the necessary supervision for models to learn dynamic, multi-step reasoning through actionable sequences that can alter perceptual input. By structuring reasoning as a series of clinical actions, our dataset facilitates a transition from static reasoning to a multi-step problem-solving paradigm. Along with the dataset, we release benchmark evaluations, pretrained baselines, and a teacher-student framework to support supervised intermediate reasoning and context-conditioned inference. To summarize, this paper makes the following contributions: • We present Step-CoT, a dedicated medical visual CoT dataset comprising more than 10K real clinical cases and 70K question-answer pairs. Each instance contains clinically grounded reasoning chains. Our experiments show that Step-CoT progressively aligns multi-step reasoning with visual evidence, guides models to follow valid and traceable diagnostic trajectories, enhancing the accuracy, interpretability, and clinical relevance of Med-VQA. • To effectively learn from Step-CoT, we propose an innovative teacher-student CoT reasoning framework that distills complex clinical reasoning knowledge from the dataset into a lightweight student model, thereby improving generalization and adaptability to diverse diagnostic tasks. • We introduce a visual CoT benchmark for Med-VQA that evaluates performance in scenarios requiring clinically faithful reasoning chains and evidence-based justifications for supporting diagnosis. Figure 1 : Overview of the Step-CoT dataset. (A) Conventional Med-VQA approaches, where models take an image and a question as input, perform multi-modal feature fusion and output a diagnostic answer. Although leveraging multi-modal knowledge, this paradigm lacks interpretability and often yields limited diagnostic accuracy. (B) Enhances interpretability by integrating large language models with CoT reasoning to generate intermediate explanations; however, such reasoning is often unreliable. (C) Our proposed Step-CoT dataset and training framework, which introduces explicit intermediate supervision. By guiding the model to learn structured clinical reasoning steps, Step-CoT not only improves interpretability through trustworthy intermediate reasoning but also enhances diagnostic accuracy. 2 Step-CoT Dataset We constructed the medical visual CoT dataset Step-CoT (see Fig. 1 ). Step-CoT formalizes the clinical diagnostic trajectory as a