CheXthought: A global multimodal dataset of clinical chain-of-thought reasoning and visual attention for chest X-ray interpretation

2026

Read CheXthought: A global multimodal dataset of clinical chain-of-thought reasoning and visual attention for chest X-ray interpretation (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-26288.txt.

Introduction

Sharma and colleagues released CheXthought, a global chest X-ray resource pairing expert reasoning with eye-tracking data. It holds over 100,000 reasoning traces and more than 6.6 million attention marks across 50,312 scans read by 501 radiologists in 71 countries. Models trained on it classified disease better, grounded claims in image regions, and reasoned about time and uncertainty. Attention hints at run time also recovered missed findings and cut invented ones.

Catalog record

YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE94399
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.26288

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CheXthought: A global multimodal dataset of clinical chain-of-thought reasoning and visual attention for chest X-ray interpretation Abstract Chest X-ray interpretation is one of the most frequently performed diagnostic tasks in medicine and a primary target for AI development, yet current vision–language models are primarily trained on datasets of paired images and reports, not the cognitive processes and visual attention that underlie clinical reasoning. Here, we present CheXthought, a global, multimodal resource containing 103,592 chain-of-thought reasoning traces and 6,609,082 synchronized visual attention annotations across 50,312 multi-read chest X-rays from 501 radiologists in 71 countries. Our analysis reveals clinical reasoning patterns in how experts deploy distinct visual search strategies, integrate clinical context, and communicate uncertainty. We demonstrate the clinical utility of CheXthought across four dimensions. First, CheXthought reasoning significantly outperforms state-of-the-art vision–language model chain-of-thought in factual accuracy and spatial grounding. Second, visual attention data used as an inference-time hint recovers missed findings and significantly reduces hallucinations. Third, models trained on CheXthought data achieve significantly higher accuracy in pathology classification, visual faithfulness, temporal reasoning and uncertainty communication. Fourth, leveraging CheXthought’s multi-reader annotations, we predict both human–human and human–AI disagreement directly from an image, enabling transparent communication of case difficulty, uncertainty and model reliability. These findings establish CheXthought as a resource for advancing multimodal clinical reasoning and the development of more transparent, interpretable vision–language models. Sonali Sharma 1,2 , Jin Long 3 , George Shih 4 , Sarah Eid 5 , Christian Bluethgen 1,2 , Francine L. Jacobson 5 , Emily B. Tsai 2 , *Global Radiology Consortium, Ahmed M. Alaa 6,7 , Curtis P. Langlotz 1,2,8 1 Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Stanford, CA, USA 2 Department of Radiology, Stanford University, Stanford, CA, USA 3 Department of Pediatrics, Stanford University School of Medicine, Stanford, California, USA 4 Department of Radiology, Weill Cornell Medicine, New York, NY, USA 5 Department of Radiology, Brigham and Women’s Hospital, Boston, MA, USA 6 University of California, Berkeley, CA, USA 7 University of California, San Francisco, CA, USA 8 Department of Biomedical Data Science, Stanford University, Palo Alto, CA, USA [email protected] *Global Radiology Consortium Grant MacKinnon, Mustafa Ege Seker, Stephanie Nougaret, Suvrankar Datta, Yoojin Nam, Seong Ho Park, Yousif Al-Naser, Aram Can Yuce, Raif Yarol, Emre Ruhat Avci, Furkan Uncuoglu, Ekin Tunga, Mushtaq Omid, Onur Taydas, Rabia Aksoylu, Burak Yangoz, Kaan Kucukdere, Esra Aktufan, Gizem Karagol, Zeynep Nilufer Tekin, Zeynep Kazci, Ahmet Kasim Karabulut, Caner Öztürk, Fatma Can, Eren Çamur, Gorkem Durak, Yasin Karkas, Cleiton Aquino de Freitas, Klenam Dzefi-Tettey, Delaedem Tamakloe, Eyram Mawulolo Dagadu, Gideon Gawu-Mensah, George Hadjidekov, Mohamad Monif Assker, Munther Abduljalil, Hwajin Lee, Dong Yeong Kim, Jeong Min Song, Songsoo Kim, Seunghyun Song, Hee Sang Oh, Myung-Jean Lee, Ji Won Jeong, Nuri Saraç, Onat Guner, Ismet Cayir, Alexander V. Ortiz, Calvin Qian, Arne Michalson, Munkhbaatar Dagvasumberel, Ali Canturk, Betul Uzunoglu, Abidin Kılınçer, Ayşe Arı, Selman Mert, Elif Kamalak, Ahmet Baytok, Gulsum Kilickap, Esra Yurduseven, Yunus Ysr, Gurhan Abdi, Nurper Denizoğlu, Yasin Celal Gunes, Lili Koteterova, Ainara Darbayeva, Liliana De Giorgi, Giulia Ronca, Francesca Fiorella, Ersilia Salsano, Antonio Salerno, Antonello L.L. Marcigliano, Donato Bressi, Antonio Rizzo, Laura Sommella, Maria Gerardina Freda, Eugenia Amelia Lettieri, Renato Cuocolo, Andrea Ponsiglione, Kursat Karaman, Nur Betul Unal Ozdemir, Elif Yaşar, Elizabeth Zamora Endara, Jasmin Maria-Theresia Happe, Stefanus Imanuel Setiawan, Sunghyun Jung, Won Jung Kim, Andrew Refalo, Simon P. Woodbridge, Naimish Adroja, Sabeeh Syed, Faysal Al-Ma’ayah, Faris Haddad, Mikel Isla-Jover, Martin Santamaria Boado, Prashansa Gupta, Min-Fang Chao, Cheng-Hsuan Juan, Meltem Emine, Carola Brussaard, Calvin T. John, Fırat Atak, Fatih Erdem, Asli Tanrivermis Sayit, Gülten Taşkın, Busra Has, Mohamed R. Nouh, Roqiah Fatmawati, Abdul Kadir, Shyamal Vijai K, Vlastimil Valek Jr., Erencan Karakoç, Chloe DesRoche, Monish Ahluwalia, Jane Burns, Sasa Golubovic, Şerife Nur Ulusan, Mariola Gutiérrez Gallardo, Zaid Hussain Khan, Yohan Joo, Devyani Singh, Khushboo Jain, Alessandra Farchione, Lukas Müller, Maximilian Moos, VenkataGanesan Ponnalagu, Vuk Matić, Ebru Hasbay, Ryeong Ah Kim, Atinuke M Agunloye, Dhananjay Sahastrabudhe, Warissara Jutidamrongphan, Deepak Justine Viswanathan, Wen-yuan Yang, Anurag Goyal, JanJan Chai, Avnish D. Goyani, Paulina Kalinowska, Enes Deger, Joseph Ortolani, Lieve Morbée, Martin T Halicek, Mohammed Shakeebuddin Kashif, Reece J. Goiffon, Sanjana Hebbar, José Ignacio Debes, Clemente García-Hidalgo, Jamal Bani Issa, Aishwarya Yogiraj Gosavi, Kevin Anton, Tejas Phaterpekar, Ricky Hu, Navjit Singh, Asma Al Hatmi, Jason Yao, Courtney Cheng, Tahreem Anwar, Yuki Arita, Rania Refaat, Shreyas Reddy Kankara, Suyash Gunjal. 1 Introduction Chest X-ray (CXR) interpretation is one of the most commonly performed diagnostic imaging tasks across emergency, inpatient, and outpatient settings globally and a primary target for artificial intelligence (AI). As of early 2026, over 35 FDA-cleared AI devices and 10 open-source vision-language models (VLMs) targeting chest radiographs are available [ 1 , 2 ] . As these models are increasingly deployed in clinical settings, understanding how they arrive at a final answer is essential for patient safety, clinician trust and regulatory oversight. Reasoning models with chain-of-thought (CoT) capability have emerged as a mechanism for improving transparency, offering an interpretable window into model decision-making that supports auditing and patient safety [ 3 ] . However, previous work has demonstrated that current CoT outputs are often unfaithful, producing plausible-sounding rationales that are disconnected from the model’s actual internal computations [ 4 ] , and frontier VLMs can achieve high benchmark accuracy without any image input at all, calling into question whether these models truly reason about what they see [ 5 ] . Currently, the majority of VLMs are trained on images paired with diagnostic labels or radiology reports, neither of which characterizes the underlying expert reasoning process or the visual attention patterns that guide interpretation. In practice, clinical reasoning extends beyond pattern recognition; it requires probabilistic reasoning under uncertainty, integration of clinical context with visually grounded evidence, and the bridging of population-based knowledge to individual cases [ 6 ] . In radiology, this reasoning unfolds alongside directed visual attention, as radiologists systematically search across anatomical regions, prioritizing structures, revisiting areas of concern, and spatially anchoring their observations to specific image features. While a small number of prior studies have leveraged eye-tracking to approximate visual attention during image interpretation, existing datasets are limited to small numbers of readers within single institutions, lack inter-reader variability, and do not capture the entire step-by-step verbalized reasoning process that accompanies visual search [ 7 , 8 ] . These constraints stem largely from the logistical and financial challenges of eye-tracking itself, which requires specialized hardware with physically co-located radiologists, limiting the geographic and demographic diversity of annotator cohorts and contributing to broader biases in medical imaging datasets [ 9 ] . To address these challenges, we introduce CheXthought, the first large-scale global human chain-of-thought reasoning and visual attention dataset for chest X-ray interpretation. CheXthought contains 103,592 chain-of-thought reasoning traces paired with 6,609,082 synchronized visual attention coordinates across 50,312 chest X-rays drawn from the CheXpert Plus dataset [ 10 ] . Each reasoning trace captures step-by-step verbalized clinical reasoning while annotators simultaneously traced their visual attention across the image. Using this resource, we first analyze CheXthought data to characterize patterns in how experts clinically reason. We then perform blinded expert evaluation comparing human-authored reasoning to CoTs generated by state-of-the-art vision–language models. We further investigate whether visual attention can be used as an inference-time hint to improve

Read the complete text