S-Chain : Structured Visual Chain-of-Thought for Medicine

Khai Le-Duc, Duy M. H. Nguyen, Phuong T. H. Trinh, Tien-Phat Nguyen, Nghiem T. Diep, An Ngo, Tung Vu, Trinh Vuong, Anh-Tien Nguyen, Mau Nguyen, Van Trung Hoang, Khai-Nguyen Nguyen, Hy Nguyen Chris Ngo, Anji Liu, Nhat Ho, Anne-Christin Hauschild, Khanh Xuan Nguyen, Thanh Nguyen-Tang, Pengtao Xie, Dan · 2025

Read S-Chain : Structured Visual Chain-of-Thought for Medicine by Khai Le-Duc, Duy M. H. Nguyen, Phuong T. H. Trinh, Tien-Phat Nguyen, Nghiem T. Diep, An Ngo, Tung Vu, Trinh Vuong, Anh-Tien Nguyen, Mau Nguyen, Van Trung Hoang, Khai-Nguyen Nguyen, Hy Nguyen Chris Ngo, Anji Liu, Nhat Ho, Anne-Christin Hauschild, Khanh Xuan Nguyen, Thanh Nguyen-Tang, Pengtao Xie, Dan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22728.txt.

Introduction

Medical image models must show their work, not just give answers. Khai Le-Duc leads a large team with twenty three co-authors. In 2025 they release S-Chain. It is a dataset for medical visual question answering. Each case links reasoning steps to image regions.

The set holds twelve thousand expert marked medical images. Boxes mark the exact regions used in each step. It supports sixteen languages. Total question pairs pass seven hundred thousand. Tests show this training improves grounding and robustness. The team also studies how retrieval interacts with visual proof. A new mechanism tightens the link between proof and reasoning.

Catalog record

CREATORKhai Le-Duc, Duy M. H. Nguyen, Phuong T. H. Trinh, Tien-Phat Nguyen, Nghiem T. Diep, An Ngo, Tung Vu, Trinh Vuong, Anh-Tien Nguyen, Mau Nguyen, Van Trung Hoang, Khai-Nguyen Nguyen, Hy Nguyen Chris Ngo, Anji Liu, Nhat Ho, Anne-Christin Hauschild, Khanh Xuan Nguyen, Thanh Nguyen-Tang, Pengtao Xie, Dan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE81 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-22728

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

S-Chain : Structured Visual Chain-of-Thought for Medicine Khai Le-Duc, Duy M. H. Nguyen, Phuong T. H. Trinh, Tien-Phat Nguyen, Nghiem T. Diep, An Ngo, Tung Vu, Trinh Vuong, Anh-Tien Nguyen, Mau Nguyen, Van Trung Hoang, Khai-Nguyen Nguyen, Hy Nguyen Chris Ngo, Anji Liu, Nhat Ho, Anne-Christin Hauschild, Khanh Xuan Nguyen, Thanh Nguyen-Tang, Pengtao Xie, Daniel Sonntag, James Zou, Mathias Niepert, Anh Totti Nguyen Abstract Faithful reasoning in medical vision–language models (VLMs) requires not only accurate predictions but also transparent alignment between textual rationales and visual evidence. While Chain-of-Thought (CoT) prompting has shown promise in medical visual question answering (VQA), no large-scale expert-level dataset has captured stepwise reasoning with precise visual grounding. We introduce S-Chain , the first large-scale dataset of 12,000 expert-annotated medical images with bounding boxes and structured visual CoT (SV-CoT), explicitly linking visual regions to reasoning steps. The dataset further supports 16 languages, totaling over 700k VQA pairs for broad multilingual applicability. Using S-Chain , we benchmark state-of-the-art medical VLMs (ExGra-Med, LLaVA-Med) and general-purpose VLMs (Qwen2.5-VL, InternVL2.5), showing that SV-CoT supervision significantly improves interpretability, grounding fidelity, and robustness. Beyond benchmarking, we study its synergy with retrieval-augmented generation, revealing how domain knowledge and visual grounding interact during autoregressive reasoning. Finally, we propose a new mechanism that strengthens the alignment between visual evidence and reasoning, improving both reliability and efficiency. S-Chain establishes a new benchmark for grounded medical reasoning and paves the way toward more trustworthy and explainable medical VLMs. 1 Introduction Large Language Models (LLMs) and Vision Language Models (VLMs) have shown strong capabilities in problem solving, planning, and decision making by learning deductive and inductive reasoning from large-scale data. A key driver is Chain-of-Thought (CoT) reasoning, which breaks complex tasks into step-by-step inferences before reaching a final answer. This paradigm improves performance across domains, from arithmetic and commonsense reasoning in LLM ( Wei et al. 2022 ; Kojima et al. 2022 ) to Visual Question Answering (VQA) and multimodal reasoning in VLM ( Zhang et al. 2023c ; Chen et al. 2024a ) . By externalizing their reasoning process, CoT not only boosts accuracy but also adds interpretability, making them especially promising for high-stakes fields like healthcare. Despite recent progress, training models with strong CoT reasoning still demands large amounts of annotated data, as models must learn to align intermediate reasoning steps with input evidence ( Zelikman et al. 2022 ; Wang et al. 2022 ) . In general Natural Language Processing (NLP) , such supervision can be scaled through crowdsourcing or distillation ( Magister et al. 2022 ; Ho et al. 2023 ) , but in medicine, it is far more costly: annotations must be expert-verified, multimodal, and clinically valid ( Moor et al. 2023a ; Huang et al. 2024 ) . Beyond this, medical reasoning requires visual grounding, i.e., explicitly linking reasoning steps to Region of Interest (ROI) , which adds substantial complexity. As a result, large-scale expert datasets with grounded CoT remain scarce, limiting the training and evaluation of trustworthy medical VLMs . To mitigate the high cost of expert annotation, recent work has explored auto-generation of CoT data for VLM reasoning. For example, MC-CoT ( Wei et al. 2024 ) leverages modular pipelines where LLMs generate reasoning steps that are loosely aligned with multimodal inputs in zero-shot settings, while MedCoT ( Liu et al. 2024 ) introduces hierarchical expert verification to refine automatically produced rationales. Similarly, large medical VQA datasets such as PMC-VQA ( Zhang et al. 2023a ) rely on template-based or synthetic Question Answering (QA) generation to scale supervision. While such approaches improve data availability, their effectiveness is limited for clinical reasoning due to two key issues: (i) auto-generated CoTs often lack structure, providing free-text explanations without explicit correspondence to specific image regions, which weakens visual grounding; and (ii) they are prone to factual mistakes and hallucinations, frequently introducing redundant or clinically irrelevant content that is difficult to filter out ( Gu et al. ; Cheng et al. 2025 ) . These limitations highlight the need for high-quality, structured, and expert-grounded CoT annotations in the medical domain. To address these challenges, we propose a new expert-annotated dataset that provides visually grounded CoTs explicitly linking step-by-step reasoning to visual evidence, which we term Structured Visual Chain-of-Thought (SV-CoT) . Our dataset contains 12,000 medical images with bounding-box annotations of ROI , paired with structured rationales that are decomposed into four clinically meaningful stages: (i) object localization, (ii) image captioning, (iii) multiple-choice reasoning, and (iv) image classification. Unlike auto-generated CoTs , each rationale is carefully annotated and verified by medical experts, ensuring both factual accuracy and strong correspondence between reasoning steps and visual features. To enhance accessibility and global applicability, the dataset further supports 16 languages , resulting in over 700,000 QA pairs . By combining structured reasoning, explicit grounding, multilingual coverage, and expert verification, this resource overcomes the key limitations of existing synthetic CoT approaches and establishes a reliable foundation for training and benchmarking medical VLMs . With this dataset in place, we systematically investigate its impact on the performance of multiple model families, including both domain-specific medical VLMs (e.g., ExGra-Med ( Nguyen et al. 2025 ) , LLaVA-Med ( Li et al. 2023a ) ) and general-purpose VLMs (e.g., Qwen2.5-VL ( Wang et al. 2024 ) , InternVL2.5 ( Chen et al. 2024b ) ), and compare them against baselines trained with synthetic CoTs generated by GPT-4.1. Beyond standard evaluation, we further assess the integration of our SV-CoT supervision with Retrieval-augmented Generation (RAG) ( Zhao et al. 2025 ; Zheng et al. 2025 ) , examining how external domain-specific knowledge interacts with structured reasoning and visual grounding. A key focus of our analysis is the faithfulness of CoT reasoning and grounding during autoregressive training, where we uncover important discrepancies between textual reasoning steps and the visual evidence they reference. These findings motivate the development of new learning strategies that explicitly reinforce the correlation between grounded visual cues and CoT reasoning, leading to more reliable, interpretable, and clinically trustworthy medical VLMs . In summary, we make the key contributions as: • Dataset innovation : We build the first large-scale dataset, S-Chain , that couples 12k medical images with expert-verified bounding-box annotations and visually grounded reasoning traces, extended to 700k multilingual QA pairs across 16 languages, structured into a four-stage reasoning pipeline to enhance clarity and consistency. • Extensive evaluation : We conduct a broad comparative study of specialized medical VLMs and general-purpose VLMs , against baselines using GPT-4.1–generated rationales, highlighting the distinctive gains from expert-grounded supervision. • Analytical insights : We examine how structured visual chain-of-thought reasoning interacts with RAG and probe the faithfulness of CoT alignment with visual grounding during autoregressive training, from which we derive some insights for new learning strategies to tightly couple visual evidence and reasoning. Figure 1: Overview of the S-Chain dataset with SV-CoT annotations . Each image is paired with (Q1) ROI localization via bounding boxes, (Q2) lesion descriptions, and (Q3) lesion grading using standardized scales (e.g., Koedam, GCA, MTA). These stepwise annotations ground reasoning in visual evidence, enabling interpretable and reliable medical VQA . 2 Problem Formulation and Key Challenges We study the problem of grounded medical VQA , where the input is a medical image (e.g.,

Read the complete text