RadEyeVideo: Enhancing general-domain Large Vision Language Model for chest X-ray analysis with video representations of eye gaze
Yunsoo Kim, Jinge Wu, Honghan Wu · 2025
Read RadEyeVideo: Enhancing general-domain Large Vision Language Model for chest X-ray analysis with video representations of eye gaze by Yunsoo Kim, Jinge Wu, Honghan Wu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-09097.txt.
Introduction
Yunsoo Kim, Jinge Wu, and Honghan Wu present RadEyeVideo, a way to help vision-language models read chest X-rays by using radiologists' eye gaze as video. Their 2025 paper notes that gaze data can aid human-computer interaction in medical imaging.
Past work used heatmaps or text prompts, which lose the order in which a radiologist's eyes move. RadEyeVideo instead feeds eye-fixation data as a video sequence, keeping both space and time. The team tested it on report generation and disease diagnosis with three open-source models that take video input. Adding gaze videos raised report generation scores by up to about a quarter in their tests.
Catalog record
| CREATOR | Yunsoo Kim, Jinge Wu, Honghan Wu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-09097
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RadEyeVideo: Enhancing general-domain Large Vision Language Model for chest X-ray analysis with video representations of eye gaze Yunsoo Kim, Jinge Wu, Honghan Wu Abstract Large Vision-Language Models (LVLMs) have demonstrated promising performance in chest X-ray (CXR) analysis. To enhance human-computer interaction, several studies have incorporated radiologists’ eye gaze, typically through heatmaps or textual prompts. However, these methods often overlook the sequential order of eye movements, which could provide valuable insights by highlighting both the areas of interest and the order in which they are examined. In this work, we propose a novel approach called RadEyeVideo that integrates radiologists’ eye-fixation data as a video sequence, capturing both the temporal and spatial dynamics of their gaze. We evaluate this method in CXR report generation and disease diagnosis using three general-domain, open-source LVLMs with video input capabilities. When prompted with eye-gaze videos, model performance improves by up to 24.6% in the report generation task and on average 15.2% for both tasks using scaled evaluation metrics. Notably, RadEyeVideo enhanced an open-domain LVLM model, LLaVA-OneVision, to surpass task-specific medical LVLMs such as MAIRA-2 and CheXagent, trained on large Chest X-ray data. This work highlights that domain expert’s knowledge (eye-gaze information in this case), when effectively integrated with LVLMs, can significantly enhance general-domain models’ capabilities in clinical tasks. RadEyeVideo is a step toward a scalable human-centered approach of utilizing LVLMs in medical image analytics. 1 Introduction Large Vision-Language Models (LVLMs) have emerged as a potentially transformative approach for various chest X-ray (CXR) analysis, including visual question answering, automated report generation, and error detection within the reports [ 2 , 7 , 14 , 17 , 23 , 24 ] . While these models offer the potential to streamline clinical workflows and enhance diagnostic efficiency, the reliability of LVLMs in real-world clinical environments remains a challenge. A key limitation is the variability and accuracy of the outputs generated by these models [ 1 , 4 , 25 , 26 ] . A promising solution to enhance LVLMs in clinical contexts is human-in-the-loop integration, particularly via expert cues. Recent work demonstrates that combining human expertise with automated analysis can exceed the performance of either radiologists or AI models working alone [ 3 , 16 ] . One such expert cue is eye-tracking data, which captures both where and how long radiologists look at relevant regions. Several studies have demonstrated that integrating eye-gaze information into AI models enhances diagnostic accuracy by providing insights into the areas radiologists focus on during image interpretation [ 8 , 15 , 21 , 22 , 30 ] . Recent work has expanded the use of radiologists’ eye-tracking data in LVLMs for multimodal tasks like report generation and visual question answering [ 19 , 20 ] . These models typically integrate eye-gaze information through either simplified static textual prompts or heatmaps. However, these recent studies generally overlook the sequential order of eye movements, which could provide valuable additional context. The sequence in which a radiologist scans an image offers insight into how they prioritize different regions, potentially contributing to more nuanced interpretations and improving performance on downstream tasks. Figure 1 : Comparison of the eye gaze prompting methods. Heat Map: previous work that used the static eye gaze information laid over the chest X-ray image. Fixation Text: previous work that implemented eye gaze information as textual prompt in the order of duration. RadEyeVideo: Our prompting method implements a video to capture a dynamic representation of eye gaze. To address this gap, we propose a novel prompting approach, RadEyeVideo , that integrates eye-tracking data as a video, capturing both the temporal and spatial order of eye movements (Figure 1 ). This richer representation preserves the dynamic process of how radiologists navigate and prioritize different regions in an image. By incorporating this sequential flow, our approach offers deeper insights into the decision-making process, highlighting both the areas of interest and the order in which they are examined. Since radiologists often follow a structured approach during interpretation, capturing this sequence provides critical context for the model’s understanding. Related Works Figure 1 shows how our approach distinguishes itself from previous prompting methods for LVLMs in CXR analysis. For “Heat Map” method, red dots are overlayed on the CXR images to visualize spatial patterns of eye gaze information [ 19 ] . These dots indicate the areas of focus, with darker dots representing longer gaze durations, thereby incorporating a temporal dimension to the data. However, it fails to convey any information about their sequential order, making the prompting method a simplified snapshot of gaze movements. Although the textual prompt can contain some sequential temporal information regarding fixations, existing methods do not fully leverage this capability; instead, gaze data is typically organized by duration, limiting its representation of sequential order [ 20 ] . Although the textual format has the potential to convey a correct sequence, it ultimately represents all spatial and temporal information in plain text. The X and Y coordinates, represented in relative width and height, offer only an indirect representation. The relative position of the gaze in the text has to be used to infer the gaze point on the image, and LVLMs capability of doing this remains unexplored. In contrast, RadEyeVideo captures the dynamic nature of gaze patterns, providing a more comprehensive understanding of radiologist behavior during image interpretation. Contributions This work aims to advance human-centered AI research in AI-assisted diagnostics through innovative integration of eye-tracking data. It establishes a generalizable prompting strategy with a video that can be applied across various medical and non-medical fields. The contributions offer significant improvements in both the accuracy of report generation and diagnosis and the potential for broader human-AI collaboration. This multimodal expert-in-the-loop approach aims to advance human-centered AI research in medical image computing by effectively combining artificial and human intelligence. The proposed work will evaluate the impact of radiologists’ expertise or perceptual cues in the form of eye gaze on model performance, aiming to enhance the accuracy and clinical relevance of AI-driven solutions for various medical image analysis tasks. Our approach offers several key contributions to the fields of AI-driven diagnostics and human-AI collaboration, summarized as follows: • RadEyeVideo - Dynamic Eye Gaze with Video for CXR Analysis: A novel prompting method that inputs radiologists’ eye-gaze as short video sequences, capturing the spatial and temporal structure of scan paths. This prompting improved by 24.6% in the report generation and 15.2% in the overall performance, outperforming even certain task-specific medical LVLMs. • Comprehensive Benchmark of Eye-Gaze Prompting: We systematically compare our video-based approach against heatmap-based and textual-fixation-based prompts. This is the first holistic evaluation of multiple gaze-integration methods, demonstrating RadEyeVideo’s clear advantages in diagnostic accuracy and clinical relevance. • MIMIC-Eye-Video Dataset: We introduce a novel dataset derivation pipeline, yielding eye-gaze videos for 2,298 CXR images from MIMIC-CXR. Due to policy restrictions, only the code and protocols are shared, but this enables future research to replicate our pipeline and further advance medical AI with expert gaze data. 2 RadEyeVideo - Dynamic Eye Gaze Video Prompting We introduce RadEyeVideo, a dynamic eye-gaze video prompting technique, designed to incorporate radiologists’ eye-gaze patterns into LVLMs for chest X-ray report generation and diagnosis. The key motivation behind this method is to enhance the interpretative