Seeing Through Experts’ Eyes: A Foundational Vision-Language Model Trained on Radiologists’ Gaze and Reasoning

Kinhei Lee, Peiyuan Jing ∗, Zhenxuan Zhang ∗, Yue Yang, Tao Wang, Dominic C Marshall, Yingying Fang, Guang Yang · 2026

Read Seeing Through Experts’ Eyes: A Foundational Vision-Language Model Trained on Radiologists’ Gaze and Reasoning by Kinhei Lee, Peiyuan Jing ∗, Zhenxuan Zhang ∗, Yue Yang, Tao Wang, Dominic C Marshall, Yingying Fang, Guang Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14316.txt.

Introduction

Lee and seven colleagues build GazeX, a vision-language model that learns from radiologists' eye-tracking data. Radiologists follow set steps such as ABCDEF, checking clinically relevant regions in order to cut missed findings. GazeX folds gaze paths and focus patterns into training, learning expert attention and integrating observations in clinically meaningful order. Trained on 30,000-plus gaze key frames from five radiologists plus hundreds of thousands of studies and question-answer pairs, it produces more accurate, interpretable, expert-consistent outputs. Verifiable evidence artifacts support efficient human checking.

Catalog record

CREATORKinhei Lee, Peiyuan Jing ∗, Zhenxuan Zhang ∗, Yue Yang, Tao Wang, Dominic C Marshall, Yingying Fang, Guang Yang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60414
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.14316

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Seeing Through Experts’ Eyes: A Foundational Vision-Language Model Trained on Radiologists’ Gaze and Reasoning Kinhei Lee, Peiyuan Jing ∗, Zhenxuan Zhang ∗, Yue Yang, Tao Wang, Dominic C Marshall, Yingying Fang, Guang Yang Abstract Large-scale vision–language models have shown promise in automating chest X-ray interpretation. However, their clinical utility remains limited by a fundamental gap between model outputs and the reasoning processes of radiologists. Most systems optimize for semantic information without emulating how experts visually examine and interpret medical images. As a result, current models often overlook critical findings, misrepresent anatomical context, or generate reports that diverge from established diagnostic workflows. Radiologists typically follow structured protocols (e.g., the ABCDEF approach that sequentially assesses the airways, breathing, circulation, diaphragm, external and foreign material). This standardized workflow ensures that all clinically relevant regions are examined in a consistent and systematic manner. It reduces the risk of missed findings, supports reliable diagnostic reasoning and facilitates clear communication between radiologists and referring clinicians. Emulating such expert strategies is essential for improving the trustworthiness and interpretability of automated report generation systems. Therefore, we introduce Gaze-X, a vision–language model that leverages radiologists’ eye-tracking data as a behavioral prior to model expert diagnostic reasoning. By incorporating gaze trajectories and fixation patterns into pretraining, Gaze-X learns to follow the spatial and temporal structure of radiologist attention and integrates visual observations in a clinically meaningful sequence. This approach enables the model to align its focus with diagnostically relevant regions and to emulate the interpretive logic underlying expert reports. Using a curated dataset of gaze recordings, comprising over 30,000 key frames across diverse disease categories, from five radiologists interpreting chest X‑rays, we demonstrate that Gaze‑X produces more accurate, interpretable, and expert‑consistent outputs across a range of clinically relevant tasks and downstream applications. These include radiology report generation, disease grounding, and visual question answering, utilizing data from 231,835 radiographic studies with associated reports, 780,014 question–answer pairs, and 1,162 image–sentence pairs containing bounding boxes and corresponding phrases. Rather than solely improving output fluency or task performance, Gaze-X addresses a key barrier to clinical uptake by aligning model behavior with established radiological workflows. Specifically, unlike autonomous reporting systems, GazeX produces verifiable evidence artifacts, including inspection trajectories and finding-linked localized regions, enabling efficient human verification and safe human–AI collaboration. Learning “through expert eyes” thus provides a practical route toward more trustworthy, explainable, and diagnostically robust AI systems for radiology and beyond. 1 Introduction With the exponential growth in medical imaging volume and recent advances in multimodal generative artificial intelligence (AI), automated report generation from chest X-rays has become an increasingly important direction in clinical AI [ 42 , 33 , 30 ] . Such systems have the potential to reduce radiologists’ workload and improve the efficiency of medical reporting [ 42 , 47 , 17 , 33 ] . While recent models demonstrate promising fluency in free-text generation, radiology reports are not merely descriptive narratives. They are structured diagnostic documents that reflect a step-by-step reasoning process grounded in anatomical and pathological interpretation. To ensure clinical reliability and interpretability, generated reports should follow established conventions in how radiologists observe, organize, and describe findings [ 36 , 20 ] . This means that clinically useful and trustworthy report generation goes beyond image–text alignment. It must also follow standardized diagnostic protocols of clinicians [ 40 ] . As shown in Fig. 1 , standardized guidelines for interpreting medical images are essential in clinical practice, as they ensure consistency across clinicians, support effective communication, and reduce the risk of overlooking critical findings [ 35 , 2 ] . In real practice, radiologists employ a systematic approach when interpreting chest X-ray images, often following the classic ABCDEF guideline (i.e., Airways, Breathing, Circulation, Diaphragm, External and Foreign material) to (1) minimize missed findings, (2) ensure diagnostic consistency, and (3) identify the patient’s most life-threatening problems first [ 8 , 25 ] . Within this framework, specific observation patterns are followed for each area. For example, the lungs are divided into three zones (upper, middle, and lower), where each occupies about one-third of the lung’s height. Each zone is inspected from top to bottom for lung markings, which appear as small, faint white lines. Inspecting each zone in sequence helps radiologists systematically check for irregularities, such as missing markings, which may indicate a pneumothorax. More importantly, the sequential viewing approach, progressing from the entire lung to smaller subsections, allows radiologists to first assess whether the lung is uniformly expanded. When abnormalities are detected, radiologists can examine smaller anatomical regions to determine the underlying cause and confirm how extensive the pathology is. These examinations provide essential visual clues to verify and consolidate the diagnosis [ 8 ] . These viewing patterns reflect how radiologists allocate visual attention during image interpretation [ 31 ] . Recent studies have demonstrated that eye gaze trajectories are not random but closely follow clinical reasoning frameworks [ 1 , 40 ] . However, current AI models lack this attention alignment. Incorporating radiologist eye gaze into training may help AI systems better emulate human diagnostic workflows and generate more clinically aligned reports. Figure 1: Radiologist-guided diagnostic reasoning patterns for structured chest X-ray interpretation. a , Radiologists’ gaze behaviors demonstrate structured and reproducible diagnostic workflows. Eye-tracking captures fixation trajectories from multiple experts, which reveal common attention patterns aligned with established reading protocols of clinicians. Statistical analysis of inter-expert trajectories shows consistent correlations, highlighting shared visual strategies during chest X-ray interpretation. Enabling large vision–language models to learn such standardized reading workflows has the potential to enhance the performance across diverse clinical downstream tasks. b , Attention-guided visual observations align with clinically defined categories (i.e., airways, breathing, circulation, diaphragm, external devices, and foreign materials), thereby providing a semantic mapping between image features (visual clues from different regions within the examined area) and diagnostic reasoning. c , Visual clues from key anatomical regions are integrated during multi-step diagnostic processes to support structured report generation. d , Downstream tasks such as segmentation, visual question answering (VQA), and report generation benefit from the model’s alignment with expert gaze. This figure illustrates how expert eye movement data capture the temporal and spatial logic of radiological decision-making, serving as supervision signals to guide AI models toward clinically grounded and interpretable report generation. e , Clinician-in-the-loop workflow enabled by GazeX. GazeX provides inspection cues and finding-linked regions to support systematic review and human verification, allowing radiologists to verify or edit model suggestions before finalizing the report. Although many automated chest X-ray interpretation systems have emerged in recent years, they still face critical challenges in achieving the level of clinical reliability and consistency expected in real-world settings [ 40 , 14 , 17 , 41 , 32 , 50 ] . These challenges largely stem from the mismatch between how radiologists interpret medical images and how current AI models are designed [ 14 , 28 , 19 ] . One major limit is the lack of clear alignment between the model’s diagnostic outputs and the corresponding anatomical regions in the image (e.g., when asked to identify pneumothorax, a system might return a positive response without indicating any visual evidence). The system’s conclusions remain opaque and difficult to verify without grounding its answers in observable image features [ 7

Read the complete text