CAG‐VLM : Fine tuning of a large-scale model to recognize angiographic images for next-generation diagnostic systems
Yuto Nakamura, Satoshi Kodera, Haruki Settai, Hiroki Shinohara, Masatsugu Tamura, [0.8ex] Tomohiro Noguchi, Tatsuki Furusawa, Ryo Takizawa, Tempei Kabayama, Norihiko Takeda, [1.5ex] The University of Tokyo The University of Tokyo Hospital · 2025
Read CAG‐VLM : Fine tuning of a large-scale model to recognize angiographic images for next-generation diagnostic systems by Yuto Nakamura, Satoshi Kodera, Haruki Settai, Hiroki Shinohara, Masatsugu Tamura, [0.8ex] Tomohiro Noguchi, Tatsuki Furusawa, Ryo Takizawa, Tempei Kabayama, Norihiko Takeda, [1.5ex] The University of Tokyo The University of Tokyo Hospital (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-04964.txt.
Introduction
Yuto Nakamura and co-authors teach a vision and language model to read heart artery images. Their twenty twenty-five paper uses coronary angiography, the standard scan for artery disease. Reading these scans well takes expert doctors, so the team built support tools.
They first sampled near fifteen thousand frames from hundreds of exams and marked key frames and side. A trained image network sorted left from right with high scores, even on faint frames. Next they paired key frames with expert-checked reports in Japanese and English. The result is a data set and pipeline for the next wave of heart diagnosis aids.
Catalog record
| CREATOR | Yuto Nakamura, Satoshi Kodera, Haruki Settai, Hiroki Shinohara, Masatsugu Tamura, [0.8ex] Tomohiro Noguchi, Tatsuki Furusawa, Ryo Takizawa, Tempei Kabayama, Norihiko Takeda, [1.5ex] The University of Tokyo The University of Tokyo Hospital |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-04964
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CAG‐VLM : Fine tuning of a large-scale model to recognize angiographic images for next-generation diagnostic systems Yuto Nakamura, Satoshi Kodera, Haruki Settai, Hiroki Shinohara, Masatsugu Tamura, [0.8ex] Tomohiro Noguchi, Tatsuki Furusawa, Ryo Takizawa, Tempei Kabayama, Norihiko Takeda, [1.5ex] The University of Tokyo The University of Tokyo Hospital Abstract Coronary angiography (CAG) is the gold‐standard imaging modality for evaluating coronary artery disease, but its interpretation and subsequent treatment planning rely heavily on expert cardiologists. To enable AI‐based decision support, we introduce a two‐stage, physician‐curated pipeline and a bilingual (Japanese/English) CAG image–report dataset. First, we sample 14,686 frames from 539 exams and annotate them for key‐frame detection and left/right laterality; a ConvNeXt‐Base CNN trained on this data achieves 0.96 F1 on laterality classification, even on low‐contrast frames. Second, we apply the CNN to 243 independent exams, extract 1,114 key frames, and pair each with its pre‐procedure report and expert‐validated diagnostic and treatment summary, yielding a parallel corpus. We then fine‐tune three open‐source VLMs (PaliGemma2, Gemma3, and ConceptCLIP‐enhanced Gemma3) via LoRA and evaluate them using VLScore and cardiologist review. Although PaliGemma2 w/LoRA attains the highest VLScore, Gemma3 w/LoRA achieves the top clinician rating (mean 7.20/10); we designate this best‐performing model as CAG‐VLM. These results demonstrate that specialized, fine‐tuned VLMs can effectively assist cardiologists in generating clinical reports and treatment recommendations from CAG images. 1 Introduction Coronary artery disease (CAD) is a leading cause of morbidity and mortality worldwide of Death Collaborators 2018 , and coronary angiography (CAG) is widely employed as the gold standard for its diagnosis Vrints et al. 2024 . In CAG, a contrast agent is injected into the coronary vessels and X-ray imaging is used to directly assess the presence and severity of stenoses or occlusions. Typically, each exam comprises multiple videos (lasting several to ten seconds) of both the left and right coronary arteries from various viewing angles. Physicians must review all these cine sequences to formulate their findings and decide on an optimal treatment plan. Accurately interpreting monochrome angiograms thus requires extensive expertise and considerable effort Shivaie et al. 2024 Leape et al. 2000 , driving strong demand for automated support Riggs et al. 2025 . Figure 1 : Overview of our two‐stage pipeline. Recent advances in vision-language models (VLMs) have spurred efforts to fine-tune foundation models with medical data, as exemplified by systems like Med-Gemini Saab et al. 2024 and Llava-Med Li et al. 2023 . Moreover, state-of-the-art foundation models have begun to exhibit a certain degree of medical knowledge. For instance, PaliGemma2 has been reported to generate appropriate diagnostic reports for X-ray images Steiner et al. 2024 . The emergence of such open-source multi-modal models with embedded medical expertise suggests new possibilities for AI-driven diagnosis of coronary artery conditions. Deep learning–based stenosis detection Avram et al. 2023 Labrecque Langlais et al. 2024 and Vision-Language Models (VLMs) for report generation in CT and X-ray domains have seen rapid progress Baba et al. 2024 Steiner et al. 2024 . However, two critical challenges remain unaddressed for invasive CAG: 1. Data scarcity No public dataset links CAG frames with patients’ clinical histories and expert-validated diagnostic plus treatment summaries. 2. Delayed VLM adoption Compared to noninvasive modalities (CT, X-ray), there are virtually no applications of large-scale multi-modal VLMs to CAG, leaving their potential largely unexplored. To overcome these gaps, we propose a physician‐curated, two‐phase pipeline. First, we sampled 14,686 frames from 539 CAG exams and annotated each for laterality (LCA/RCA) and key-frame category (better/bad/others). A ConvNeXt‐based CNN fine-tuned on this data achieves automatic key-frame extraction and label assignment. Second, we applied this CNN to 243 independent exams to extract 1,114 frames, pairing each with the patient’s pre-examination report and expert-validated diagnostic and treatment plan summaries. We then fine-tuned three VLM variants—PaliGemma2, Gemma3, and a ConceptCLIP‐enhanced Gemma3—via LoRA. Finally, quantitative evaluation using the VLSCore Dawidowicz et al. 2024 framework and qualitative assessment by cardiologists confirm that our fine-tuned models surpass both closed-source baselines and their own pre-fine-tuned versions in clinical expressiveness. 2 Related Work 2.1 Vision-Language Models for Medical Imaging Vision–Language Models (VLMs) have seen rapid advances in medical image analysis. Early work such as CheXbert combined hand‐crafted and auto‐generated rules to extract findings from chest X-ray reports Smit et al. 2020 , while RadGraph automatically identifies clinical entities and their relationships within radiology reports Jain et al. 2021 . More recently, large‐scale foundation models like Llava-Med Li et al. 2023 and Med-Gemini Saab et al. 2024 have been fine-tuned across multiple healthcare tasks, demonstrating versatility beyond single‐purpose applications. Pretrained multimodal models have also emerged: PMC-CLIP leverages millions of image–caption pairs from PubMed Central to boost performance on medical image retrieval and classification benchmarks Lin et al. 2023 ; BiomedCLIP Zhang et al. 2025 , MedCLIP Wang et al. 2022 , and ConceptCLIP Nie et al. 2025 similarly learn from massive biomedical image–text corpora to achieve state-of-the-art results on chest X-ray, MRI, and pathology tasks. In contrast to these embedding-based approaches, generative VLMs such as PaliGemma2 employ an encoder–decoder transformer to produce full-length radiology reports; when fine-tuned on the MIMIC-CXR dataset, PaliGemma2 achieved a RadGraph F1 score of 29.5%, outperforming prior methods in both report quality and clinical consistency Steiner et al. 2024 . However, these efforts have focused almost exclusively on noninvasive modalities (CT, X-ray, MRI), and applications of VLMs to invasive procedures such as coronary angiography remain largely unexplored. 2.2 Deep Learning for Coronary Angiography: Analysis & Datasets Deep learning in coronary angiography (CAG) has concentrated on stenosis detection, laterality classification, and vessel segmentation using CNN-based architectures. Eschen et al. 2022 achieved a laterality classification F1 of 0.99 on 3,500 CAG videos with a 3D-CNN Eschen et al. 2022 . CathAI integrates view selection, laterality labeling, stenosis localization, and severity estimation into a single pipeline, delivering high quantification and localization accuracy Avram et al. 2023 . Cong et al. combined CNN + LSTM for key-frame selection and FPN for multi-class stenosis classification (mild, moderate, occlusion), reaching an AUC of 0.86 and sensitivity of 0.96 Cong et al. 2023 .More recently, DeepCoro Labrecque Langlais et al. 2024 introduced an eight-stage, video-based analysis pipeline trained on 182,418 CAG cine loops, achieving an AUC of approximately 0.83 for significant stenosis detection—surpassing earlier methods in end-to-end clinical performance Labrecque Langlais et al. 2024 . Several CAG-specific datasets have been released for benchmarking deep learning algorithms. ARCADE comprises two 1,200-image cohorts—one for vessel segmentation and one for stenosis detection—each split into 1,000 training images annotated in COCO format and 200 validation images; the segmentation cohort labels 26 coronary regions according to the SYNTAX score methodology, while the stenosis cohort provides atherosclerotic plaque annotations Popov et al. 2023 . CoronaryDominance includes 1,574 invasive angiography videos labeled for right- versus left-dominant coronary circulation under varied image-quality conditions, enabling evaluation of domain generalization Kruzhilov et al. 2025 . CADICA contains 42 CAG cases with expert-reviewed stenosis annotations Jiménez‐Partinen et al. 2024 . While these resources are invaluable, they face challenges: domain shifts from different imaging systems, potential data leakage when adjacent frames are highly similar, and predominantly English reports that limit non-English clinical contexts. In contrast, our work builds both a laterality/key-frame detection set and a Japanese multimodal corpus pairing pre-procedure reports with expert‐validated diagnostic and treatment