Can Large Language Models Challenge CNNs in Medical Image Analysis?
2025
Read Can Large Language Models Challenge CNNs in Medical Image Analysis? (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23503.txt.
Introduction
Shibbir Ahmed, Shahnewaz Karim Sakib, and Anindya Bijoy Das compare large language models with classic networks on medical image reading in this 2025 paper. Convolutional networks are the standard tool for scan classification. Multimodal language models now claim image skills too. The team benchmarked both camps on public diagnostic sets. Accuracy and F1 scores measured diagnostic quality. Speed, energy use, and carbon cost were also counted.
Classic networks still won on raw performance in their tests. Filtering steps on top of language models closed much of the gap. Green costs matter as hospitals weigh AI adoption. The study frames model choice as a three-way trade. It is a comparative evaluation paper. Its datasets are public, so others can recheck it.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-23503
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Can Large Language Models Challenge CNNs in Medical Image Analysis? Abstract This study presents a multimodal AI framework designed for precisely classifying medical diagnostic images. Utilizing publicly available datasets, the proposed system compares the strengths of convolutional neural networks (CNNs) and different large language models (LLMs). This in-depth comparative analysis highlights key differences in diagnostic performance, execution efficiency, and environmental impacts. Model evaluation was based on accuracy, F1-score, average execution time, average energy consumption, and estimated C O 2 CO_{2} emission. The findings indicate that although CNN-based models can outperform various multimodal techniques that incorporate both images and contextual information, applying additional filtering on top of LLMs can lead to substantial performance gains. These findings highlight the transformative potential of multimodal AI systems to enhance the reliability, efficiency, and scalability of medical diagnostics in clinical settings. \threeauthors Shibbir AhmedTexas State University San Marcos, TX, USA Shahnewaz Karim Sakib University of Tennessee at Chattanooga Chattanooga, TN, USA Anindya Bijoy Das The University of Akron Akron, OH, USA 1 Introduction The rapid evolution of artificial intelligence (AI) has ushered in transformative approaches for medical diagnostics, particularly in the analysis of diagnostic imaging data [ 1 ] . Over the past decade, machine learning techniques have been applied successfully to tasks such as image classification, segmentation, and anomaly detection in modalities including X-ray [ 2 ] , computed tomography (CT) [ 3 ] , and magnetic resonance imaging (MRI) [ 4 ] . Early breakthroughs using convolutional neural networks (CNNs) [ 2 ] , recurrent neural networks (RNNs), and autoencoders demonstrated the potential of deep architectures to learn hierarchical representations directly from raw images. Recent advancements in deep learning have introduced alternative architectures such as transformers [ 5 ] , capsule networks [ 6 ] , and diffusion models [ 7 ] , significantly enhancing the capabilities of medical image analysis. These innovative models excel in capturing complex patterns and dependencies in medical data, which is crucial for accurate diagnosis and effective treatment planning. Their successful application across various clinical scenarios demonstrates their potential in improving the precision and efficiency of healthcare interventions. In parallel with these imaging advances, large language models (LLMs) [ 8 ] have emerged as powerful tools for processing and interpreting clinical narratives, radiology reports, and other unstructured data sources [ 8 , 9 , 10 , 11 ] . LLMs complement image-based predictions to boost diagnostic accuracy by extracting and contextualizing patient information. For example, integrating LLMs with imaging architectures enables automated report generation and decision support, i.e., cross-referencing clinical findings with current literature, to reduce clinician workload and diagnostic errors [ 9 ] . Furthermore, the synergy between LLMs and advanced machine learning techniques enhances AI interpretability and explainability [ 12 ] , improving trust among healthcare professionals and enabling personalized treatments. Incorporating LLMs augments imaging model performance, streamlines diagnosis, and improves clinical workflows, ultimately enhancing patient care. In this study, we investigate the reliability and performance of multimodal AI approaches for diagnostic image classification by comparing the effectiveness of CNN-based architectures with that of several benchmark large language models [ 13 , 14 ] , including OpenAI’s GPT-4o, and Meta’s LLaMA. We employ publicly available datasets comprising X-ray, CT, and MRI modalities (as shown in Fig. 1 ), which are meticulously partitioned into training, validation, and unseen testing sets. In addition, we explore these multiple LLMs to assess their potential in diagnostics, where few studies [ 15 ] have concurrently addressed both utility and reliability. (a) Chest X-ray (b) Chest CT scan (c) Brain MRI Figure 1 : Different Medical Imaging Modalities and Pathologies: (a) Chest X-rays depicting normal lung anatomy, COVID-19 infection, lung opacity, and viral pneumonia. (b) Chest CT scans for normal lung, squamous cell carcinoma, adenocarcinoma, and large cell carcinoma. (c) Brain MRI scans highlighting conditions such as no tumor, glioma, meningioma, and pituitary abnormalities. These exemplify typical radiological features used for training and evaluating AI-driven diagnostic models in medical imaging. Our systematic evaluation employs performance metrics such as accuracy, F1-score, average execution time, average energy consumption, and estimated CO_{2} emission to provide detailed insights into the strengths and limitations of CNNs and LLMs in real-world clinical scenarios. Additionally, we explore confidence scores as a measure of reliability, offering a comparative analysis of the trustworthiness of these models in diagnostic decision-making. This study contributes by: • Benchmarking CNNs and LLMs on classifying diverse medical imaging datasets, including image analysis by the LLM models for future study. • Analyzing computational efficiency, including runtime, energy consumption, and associated CO 2 emissions. • Highlighting trade-offs between model accuracy and resource demands, emphasizing considerations critical for practical and sustainable AI adoption in healthcare. In summary, our research findings provide useful benchmarks for researchers and valuable insights for medical professionals seeking to adopt AI-driven diagnosis, ultimately improving clinical workflow reliability and efficiency. 2 Datasets and Related Works In this section, we detail the datasets utilized in this study and review existing literature relevant to the classification of medical images. 2.1 Dataset Selection To ensure a comprehensive evaluation, we select the following publicly available datasets covering a range of imaging modalities and diagnostic tasks. These datasets were curated to maintain class balance, diversity, and representativeness. Chest X-Ray Dataset: The COVID-19 Radiography Database [ 16 ] contains X-ray images labeled as COVID-19, pneumonia, lung opacity, and normal. This dataset is commonly used to benchmark AI models for classifying chest X-rays. [ 17 ] . Brain Tumor MRI Dataset: The Brain Tumor MRI Dataset [ 18 ] , available on Kaggle, includes T1-weighted MRI images labeled as glioma, meningioma, pituitary tumor, or no tumor. This dataset enables the evaluation of models in brain tumor classification tasks. Chest CT Scan Dataset: This dataset [ 19 ] , also available on Kaggle, includes chest CT scans labeled as normal, adenocarcinoma, large cell carcinoma and squamous cell carcinoma. This enables us to work on different chest cancer detection. Sample images are illustrated in Fig. 1 , which demonstrates the distinctions between different types of medical imaging: chest X-ray in Fig. 1 (a), chest CT scan in Fig. 1 (b) and brain MRI in Fig. 1 (c). 2.2 Related Works Next, we explore relevant studies focused on the analysis and classification of medical images, such as X-rays, CT scans, and MRI data, utilizing various LLM-based methodologies. Transfer Learning and Pre-trained Models : Recent studies have adapted pre-trained language models, like BERT and GPT, to extract and contextualize features from medical images. For example, [ 20 ] developed MMBERT explores on two visual question answering (VQA) datasets for radiology images, outperforms different previous ensemble models and provides attention maps which help in model interpretability. Multi-modal Learning Approaches : Multi-modal learning techniques have shown great promise in enhancing the analysis of medical images by integrating models trained on different types of data [ 21 ] . For instance, incorporating neural networks with LLMs can enable a synergistic approach where the model benefits from the spatial recognition capabilities of neural networks while leveraging the contextual processing strengths of LLMs. The work in [ 22 ] demonstrates the effectiveness of large visual language models in analyzing biomedical images such as brain MRIs, microscopic images of blood cells, and