MedicalNarratives : Connecting Medical Vision and Language with Localized Narratives
Wisdom O. Ikezogwo Kevin Zhang, Mehmet Saygin Seyfioglu Fatemeh Ghezloo Linda Shapiro Ranjay Krishna · 2025
Read MedicalNarratives : Connecting Medical Vision and Language with Localized Narratives by Wisdom O. Ikezogwo Kevin Zhang, Mehmet Saygin Seyfioglu Fatemeh Ghezloo Linda Shapiro Ranjay Krishna (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-04184.txt.
Introduction
Wisdom Ikezogwo and co-authors build MedicalNarratives, a large data set that pairs medical images with spoken teaching text, in work from 2025. Models that join vision and language need huge data. Medical images cannot match the supply of natural images, so new sources are needed to train at scale.
The team mined teaching videos from YouTube. Instructors narrate while moving a cursor over the key region, like a think aloud study. That gives a natural link between words and pixels over time. The set holds millions of image text pairs. About a million carry mouse traces and boxes, and a smaller set carries video clips aligned to the trace event.
Catalog record
| CREATOR | Wisdom O. Ikezogwo Kevin Zhang, Mehmet Saygin Seyfioglu Fatemeh Ghezloo Linda Shapiro Ranjay Krishna |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-04184
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MedicalNarratives : Connecting Medical Vision and Language with Localized Narratives Wisdom O. Ikezogwo Kevin Zhang, Mehmet Saygin Seyfioglu Fatemeh Ghezloo Linda Shapiro Ranjay Krishna Abstract Multi-modal models are data hungry. While datasets with natural images are abundant, medical image datasets can not afford the same luxury. To enable representation learning for medical images at scale, we turn to YouTube, a platform with a large reservoir of open-source medical pedagogical videos. We curate MedicalNarratives, a dataset 4.7M medical image-text pairs, with 1M samples containing dense annotations in the form of spatial traces (and bounding boxes), and 118K videos centered on the trace event (with aligned text), enabling spatiotemporal grounding beyond single frames. Similar to think-aloud studies where instructors speak while hovering their mouse cursor movements over relevant image regions, 1M images in MedicalNarratives contains localized mouse traces in image pixels, creating a spatial and temporal association between the text and pixels. To evaluate the utility of MedicalNarratives, we train GenMedCLIP with a CLIP-like objective using our dataset spanning 12 medical domains. GenMedCLIP outperforms previous state-of-the-art models on all 12 domains on a newly constructed medical imaging benchmark. [Data] 1 Introduction Figure 1 : MedicalNarratives :Examples from our medical imaging modalities, excluding surgery, endoscopy, and general medical images due to their graphic nature. These samples are selected from interleaved video samples, with each sample showing the image, denoised text, and spatial traces & bbox aligned in-time on 4 domains. See section E in the Appendix for more examples and raw input text. Analyzing medical images requires simultaneous spatial localization and semantic understanding 81 . An expert has to extract visual clues from image regions and combine them with retrieved knowledge from memory, arriving at a diagnosis. This process requires connecting individual spatial image regions to clinical concepts, often utilizing a segmental approach to avoid errors. [ 81 ] . In medical image analysis, typically semantic tasks like classification, captioning, and retrieval are explored exclusively from spatial tasks like detection [ 96 , 129 , 113 ] , or segmentation [ 127 , 23 ] . This can be attributed to the lack of large grounded multimodal datasets to train such models. Recent work like MedTrinity-25M [ 128 ] , attempts to address this by releasing a multimodal dataset with spatial annotations, but relies on sub-optimally pretrained models to generate text descriptions and Regions of Interests (ROIs) for medical images lacking ground truth annotations, potentially propagating model biases and errors. While data collection costs are steep, certain protocols balance ease of collection and training utility. Specifically, Localized Narratives [ 96 , 122 ] proposes a dataset of image, text, and grounding traces, curated by leveraging human annotators to describe an image vocally while simultaneously moving a computer mouse to the regions they describe, resulting in holistic grounded descriptions. This protocol of collecting grounded vision-language (VL) datasets does not have strict spatial annotations, yet, it captures strong spatial correlations to the description with every trace point, making the protocol uniquely easier to undertake and capture data en-mass as it appeals to the human nature to point while describing a scene [ 59 , 123 , 44 ] . Localized narratives have been used to train models on semantic tasks [ 96 , 122 , 133 ] , and spatially-aware multimodal language models (MLM) like PixelLM [ 129 ] , and Molmo [ 29 ] and other generative image models [ 67 , 131 ] . To address these limitations, we draw inspiration from how medical experts naturally communicate and teach. In the joint field of cognitive psychology and medical imaging, studying how medical experts analyze patient data, studies leverage the think-aloud protocol [ 32 ] to capture data for various types of analysis, where experts verbalize their thoughts as they perform a task, and some studies capture their eye gaze/cursor localizing the image regions they focus on [ 73 , 47 ] . This protocol has been used to collect medical datasets [ 92 , 80 ] , including the Tufts dental x-ray database [ 92 ] , which captures a multimodal dataset incorporating radiologist expertise through eye-tracking and the think-aloud protocol. We propose MedicalNarratives a dataset that leverages pedagogical medical videos where instructors narrate descriptions while pointing to relevant regions with their cursor, closely mimicking the think-aloud protocol used in clinical practice and the Localized Narratives protocol. Our dataset contains 4.7 million image-text pairs across 11 medical modalities and 1 pseudo-medical domain, with interleaving samples between varying modalities (e.g., X-ray and CT for the same patient), which we argue improves downstream performance as these samples connect multiple visual and textual concepts. Importantly, 1M of these samples are grounded in expert traces that can be reformatted into bounding boxes or masks (see Figure 5 ), serving as pretraining data for various tasks. To test the base utility of our dataset, we train a vision-language model ( GenMedCLIP ) on our dataset and evaluate it on a new benchmark of datasets that cut across 11 medical modalities. On both classification and retrieval, we see our trained GenMedCLIP model outperform prior SOTA models like BiomedCLIP in both tasks with an average of 3% and 14% respectively. While the proposed dataset is a combination of data from A. Temporal unstructured sources like video, and B. Static structured sources like scientific articles, unlike prior work that solely leverage one source, our experiments show that the utility of the dataset increases with more data from video, with a net difference of 11.65% on classification tasks and 53.6% on zeroshot retrieval tasks. Finally, we show the utility of traces with qualitative examples, converting traces into segmentation using pretrained interactive medical image segmentation (IMIS) models like ScribblePrompt [ 127 , 23 ] . We hope future works leverage the dataset to train more grounded generative models similar to Quilt-LLaVA [ 113 ] , LLaVA-Med++ [ 128 ] , and PixelLM [ 129 ] as well as spatially-controlled medical image captioning models [ 96 ] . To bolster other use cases, we also release the constituting video clip IDs (useful for obtaining the videos) and many other metadata, including modality type and UMLS entities. In addition to the centered still images with traces, we provide paired videos (temporal windows around trace start/end), preserving narration alignment to the pointing behavior. This addition allows models to learn spatiotemporal grounding (e.g., cursor trajectories across frames) rather than static spatial associations alone. Figure 2 : Breakdown of MedicalNarratives in size by modalities across both video and article subsets. Dataset Size Source Medical Only Domains Open Source Data/Code Novel Images Video Text Spatial Annot. PMC-15M [ 134 ] 15M A ✗ 30 ✗/✓ ✓ ✗ Captions ✗ PMC-FG-64M [ 134 ] 46M A ✗ 30 ✗/✗ ✓ ✗ Captions ✗ PMC-CLIP [ 72 ] 15M A ✗ 12 ✓/✗ ✓ ✗ Captions ✗ MedTrinity-25M [ 128 ] 25M P ✓ 10 ✓/✗ ✗ ✗ Captions/labels Seg. mask MedicalNarratives 4.7M V+A ✓ 12 ✓/✓ ✓ ✓ Expert Traces Table 1 : Comparison with large-scale medical datasets . In the table, A: Articles, V: Videos, and P: pre-published datasets. Open-Source column is formatted data/pipeline . 2 Related work 2.1 Vision Language representation Vision-language (multi-modal) models have evolved over time