Vision-Language Model-Based Semantic-Guided Imaging Biomarker for Lung Nodule Malignancy Prediction

Luoting Zhuang, Seyed Mohammad Hossein Tabatabaei, Ramin Salehi-Rad, Linh M. Tran, Denise R. Aberle, Ashley E. Prosper, William Hsu · 2025

Read Vision-Language Model-Based Semantic-Guided Imaging Biomarker for Lung Nodule Malignancy Prediction by Luoting Zhuang, Seyed Mohammad Hossein Tabatabaei, Ramin Salehi-Rad, Linh M. Tran, Denise R. Aberle, Ashley E. Prosper, William Hsu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-21344.txt.

Introduction

Zhuang and co-authors build a vision-language model that judges whether a lung nodule is likely to turn cancerous, in this 2025 paper. Earlier models leaned on notes that radiologists must type in by hand, which slows real clinic use. The new model learns from those expert descriptions during training instead.

The team fine-tuned a pretrained image-text model on nearly a thousand low-dose CT scans from a national screening trial. It beat leading rival models at predicting cancer within a year. Tests on three outside hospital datasets also held up well. The model can even estimate nodule traits, like margin and consistency, without direct labels. That mix of accuracy and openness could help screening programs.

Catalog record

CREATORLuoting Zhuang, Seyed Mohammad Hossein Tabatabaei, Ramin Salehi-Rad, Linh M. Tran, Denise R. Aberle, Ashley E. Prosper, William Hsu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-21344

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Vision-Language Model-Based Semantic-Guided Imaging Biomarker for Lung Nodule Malignancy Prediction Luoting Zhuang, Seyed Mohammad Hossein Tabatabaei, Ramin Salehi-Rad, Linh M. Tran, Denise R. Aberle, Ashley E. Prosper, William Hsu Abstract Objective: Machine learning models have utilized semantic features, deep features, or both to assess lung nodule malignancy. However, their reliance on manual annotation during inference, limited interpretability, and sensitivity to imaging variations hinder their application in real-world clinical settings. Thus, this research aims to integrate semantic features derived from radiologists’ assessments of nodules, guiding the model to learn clinically relevant, robust, and explainable imaging features for predicting lung cancer. Methods: We obtained 938 low-dose CT scans from the National Lung Screening Trial (NLST) with 1,261 nodules and semantic features. Additionally, the Lung Image Database Consortium dataset contains 1,018 CT scans, with 2,625 lesions annotated for nodule characteristics. Three external datasets were obtained from UCLA Health, the LUNGx Challenge, and the Duke Lung Cancer Screening. For imaging input, we obtained 2D nodule slices in nine directions from 50 × 50 × 50 50\times 50\times 50 mm nodule crop. We converted structured semantic features into sentences using Gemini. We fine-tuned a pretrained Contrastive Language-Image Pretraining (CLIP) model with a parameter-efficient fine-tuning approach to align imaging and semantic text features and predict the one-year lung cancer diagnosis. Results: Our model outperformed the state-of-the-art (SOTA) models in the NLST test set with an AUROC of 0.901 and AUPRC of 0.776. It also showed robust results in external datasets. Using CLIP, we also obtained predictions on semantic features through zero-shot inference, such as nodule margin (AUROC: 0.807), nodule consistency (0.812), and pleural attachment (0.840). Conclusion: By incorporating semantic features into the vision-language model, our approach surpasses the SOTA models in predicting lung cancer from CT scans collected from diverse clinical settings. It provides explainable outputs, aiding clinicians in comprehending the underlying meaning of model predictions. The code is available at https://github.com/luotingzhuang/CLIP_nodule . Journal: Journal of Biomedical Informatics † † graphicalabstract: 1 Introduction Lung cancer remains the leading cause of cancer-related deaths [ 1 , 2 ] . Computed tomography (CT) has demonstrated effectiveness in detecting lung cancer and decreasing cancer-related mortality during lung cancer screening and also in routine healthcare settings [ 3 , 4 ] . However, radiologists are experiencing burnout due to the growing number of detected nodules resulting from the increasing application of CT. Computer-aided diagnostic systems have been proposed to alleviate the workload of radiologists by providing automatic and accurate predictions of lung cancer based on patient data. Traditional machine learning models in lung nodule risk assessment focus on semantic features, deep features, or both. Semantic features refer to the descriptive terms radiologists use to characterize regions of interest, such as the shape, margin, and vascularity. Models trained on semantic features are easier to interpret, but they also introduce difficulties when scaled up for clinical use, as they require manual annotation from radiologists [ 5 , 6 ] . Recently, deep learning (DL) has become a powerful tool due to its strong capability of extracting complex features from CT without any manual input [ 7 , 8 ] . Such imaging-based models still encounter numerous obstacles. For instance, imaging features are sensitive to variations in the acquisition and reconstruction parameters of CT scans, including dose levels, slice thicknesses, and reconstruction kernels [ 9 , 10 ] . Moreover, research has indicated that DL models learn shortcuts, which are characteristics highly correlated with outcomes but lack clinical significance [ 11 , 12 ] . This phenomenon could adversely impact the reproducibility of features and the generalization of models. Additionally, the lack of explainability remains one of the significant challenges related to deep features. Heatmaps or attribution maps generated by explainability methods, such as GradCAM [ 13 ] , can only indicate the regions on which the model focuses, but do not reveal the specific features utilized. Combining these features has become more popular for enhancing lung cancer prediction performance. However, directly merging two features often reduces applicability in clinical settings, as manual annotations are still required. Alternative approaches tackle this problem through co-learning and multi-task learning [ 14 , 15 , 16 ] , but these models face challenges regarding explainability or training difficulties when dealing with numerous semantic features. An emerging approach in vision-language models (VLM), Contrastive Language-Image Pretraining (CLIP) [ 17 ] , fills the gap by learning the alignment between imaging features and descriptive text. Guided by text, these models allow deep visual features to capture a richer and more robust representation. Therefore, in this study, we have made the following key contributions: 1. We employed CLIP to incorporate semantic features, directing the model in obtaining clinically significant and robust features for predicting lung cancer. 2. We curated a unique dataset comprising CT scans and their corresponding semantic features, which were annotated by radiologists. The semantic features encompass characteristics of both nodules and their surrounding environment. 3. We benchmarked our CLIP model against several state-of-the-art (SOTA) lung cancer prediction models using a comprehensive set of metrics. Our model demonstrated better and more robust results in external datasets collected from different clinical scenarios, such as screening and routine clinical care, and different types of CT scans, such as low-dose CT (LDCT), diagnostic CT, and contrast-enhanced CT (CECT). 4. We explored the zero-shot inference, a feature of CLIP-based models that generates semantic features despite not being explicitly trained to do so. 5. Several adjustments were introduced to accommodate CLIP fine-tuning on 3D images with limited data. By combining these strategies with extensive domain knowledge, our model shows comparable and better performance than imaging-based models trained on tens of thousands of cases using an order of magnitude fewer cases. Statement of Significance Problem or Issue Imaging-based lung nodule malignancy prediction models lack explainability and tend to learn shortcuts, limiting their generalizability and applicability in diverse clinical settings. What is Already Known One of the VLMs, CLIP, aligns image and text features in a shared space to integrate visual and textual information effectively. What this Paper Adds We utilize CLIP to incorporate both internal and external semantic features to guide the imaging model to learn clinically relevant and reliable imaging biomarkers, aiming to improve lung cancer classification across diverse patient populations and CT acquisition protocols. This model offers explainability through zero-shot inference, helping radiologists and end users understand model predictions with semantic features. 2 Related Work 2.1 Imaging-based Lung Cancer Prediction Model DL models trained on CT scans have been made available online for potential implementation in real-world clinical settings. Sybil [ 7 ] employs ResNet18 to predict lung cancer risk scores for up to six years using LDCT scans. Sybil was trained using the National Lung Screening Trial (NLST) dataset, with around 15,000 CT scans, and has consistently demonstrated robust performance during external validation using screening scans from two distinct sites. In another study, Venkadesh et al. [ 8 ] developed a model to predict the malignancy risk of pulmonary nodules by integrating ResNet18 for 2D nodule crops and Inception-V1 for 3D nodule crops. This model was trained using 16,077 nodules from the NLST dataset and has shown robust performance in external cancer screening datasets. Although these models have shown effectiveness in

Read the complete text