ChestGPT: Integrating Large Language Models and Vision Transformers for Disease Detection and Localization in Chest X-Rays Conference: 16th Indian Conference on Computer Vision, Graphics and Image Processing; December 2025; Mandi, India Price: 15.00 Note: Produces the permission block, and copyright

Shehroz S. Khan, Petar Przulj, Ahmed Ashraf, Ali Abedi · 2025

Read ChestGPT: Integrating Large Language Models and Vision Transformers for Disease Detection and Localization in Chest X-Rays Conference: 16th Indian Conference on Computer Vision, Graphics and Image Processing; December 2025; Mandi, India Price: 15.00 Note: Produces the permission block, and copyright by Shehroz S. Khan, Petar Przulj, Ahmed Ashraf, Ali Abedi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-03739.txt.

Introduction

Shehroz Khan, Petar Przulj, Ahmed Ashraf, and Ali Abedi present ChestGPT in a 2025 paper on disease detection in chest X-rays. The world needs more radiologists as imaging use grows, and supply is not keeping pace, so computer tools that aid diagnosis have clear worth.

ChestGPT joins a vision transformer with a large language model. The vision part turns the X-ray into a form the language model can work with. The pair then classifies disease and points to where it sits in the image. The paper frames the system as a way to lift the reach and accuracy of human readers.

Catalog record

CREATORShehroz S. Khan, Petar Przulj, Ahmed Ashraf, Ali Abedi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-03739

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ChestGPT: Integrating Large Language Models and Vision Transformers for Disease Detection and Localization in Chest X-Rays Conference: 16th Indian Conference on Computer Vision, Graphics and Image Processing; December 2025; Mandi, India Price: 15.00 Note: Produces the permission block, and copyright information CCS: Computing methodologies Computer vision CCS: Computing methodologies Machine learning CCS: Computing methodologies Machine learning approaches CCS: Applied computing Health informatics CCS: Computing methodologies Natural language processing Shehroz S. Khan, Petar Przulj, Ahmed Ashraf, Ali Abedi Abstract . The global demand for radiologists is increasing rapidly due to a growing reliance on medical imaging services, while the supply of radiologists is not keeping pace. Advances in computer vision and image processing technologies present significant potential to address this gap by enhancing radiologists’ capabilities and improving diagnostic accuracy. Large language models (LLMs), particularly generative pre-trained transformers (GPTs), have become the primary approach for understanding and generating textual data. In parallel, vision transformers (ViTs) have proven effective at converting visual data into a format that LLMs can process efficiently. In this paper, we present ChestGPT, a deep-learning framework that integrates the EVA ViT with the Llama 2 LLM to classify diseases and localize regions of interest in chest X-ray images. The ViT converts X-ray images into tokens, which are then fed, together with engineered prompts, into the LLM, enabling joint classification and localization of diseases. This approach incorporates transfer learning techniques to enhance both explainability and performance. The proposed method achieved strong global disease classification performance on the VinDr-CXR dataset, with an F1 score of 0.76, and successfully localized pathologies by generating bounding boxes around the regions of interest. We also outline several task-specific prompts, in addition to general-purpose prompts, for scenarios radiologists might encounter. Overall, this framework offers an assistive tool that can lighten radiologists’ workload by providing preliminary findings and regions of interest to facilitate their diagnostic process. 1. Introduction The field of radiology is under strain as medical imaging demand increases faster than the supply of available radiologists ( Cao et al. 2023 ) . This imbalance results in heavier workloads for radiologists and potential delays in patient diagnoses, which can have serious consequences ( Limb 2022 ) . To address this issue, recent advancements in computer vision and deep learning offer promising avenues for automated support systems that could enhance diagnostic accuracy ( Akhter et al. 2023 ; Hosny et al. 2018 ) . Large language models (LLMs) have recently gained significant attention due to their ability to process and understand text at an advanced level. Models such as generative pre-trained transformers (GPTs) ( Radford et al. 2018 ) are widely applied across various domains, including healthcare, for tasks such as text generation, question answering, and managing complex information ( Thirunavukarasu et al. 2023 ) . In parallel, vision transformers (ViTs) ( Fang et al. 2023 ; Han et al. 2022 ) have shown substantial progress in image analysis. ViTs can process visual data, such as X-rays, and convert it into a format that LLMs can utilize, enabling integrated image-text processing. This capability significantly improves the interpretation and analysis of medical images, particularly in radiology ( Nerella et al. 2024 ; Yan et al. 2022 ) . Recent studies have explored the integration of LLMs with vision models for medical imaging, especially in the context of chest X-rays ( Wang et al. 2021 ; Monajatipoor et al. 2021 ; Wiehe et al. 2022 ; Akhter et al. 2023 ; Lee et al. 2023 ; Li et al. 2024 ; Thawakar et al. 2024 ; Cho et al. 2024 ; Keicher et al. 2024 ) . These works have demonstrated that ViTs effectively extract salient features from X-ray images, which LLMs can then leverage to generate diagnostic reports or predict disease outcomes. However, several challenges remain, particularly in accurately localizing pathological regions and improving model interpretability. Moreover, the development of an end-to-end framework that combines both classification and localization within a unified system remains underexplored, indicating a gap in current research. In this paper, we introduce ChestGPT, a novel deep learning framework that integrates the EVA ViT ( Fang et al. 2023 ) with the LLaMA 2 LLM developed by Meta AI ( Touvron et al. 2023 ) . In this framework, chest X-ray images are first transformed into visual tokens using the ViT, which are then processed by the LLM along with carefully engineered prompts. ChestGPT performs both disease classification and localization, identifying the presence of conditions and highlighting their locations within the image. The framework further employs prompt engineering techniques to enhance interpretability and performance, making it a viable tool for clinical use. We evaluate ChestGPT on the VinDr-CXR dataset ( Nguyen et al. 2022 ) , which is designed for disease classification and localization in chest radiographs. This work makes the following contributions: • By combining advanced vision and language models, a new deep-learning framework was introduced to jointly detect diseases and localize regions of interest in chest X-ray images. • Well-crafted prompts and transfer learning techniques were incorporated to enhance the framework’s performance and extend its applicability. • Extensive experiments were conducted on a public dataset showcasing the success of the proposed method for chest X-ray-based disease detection and localization with a strong F1 score. The structure of this paper is organized as follows. Section 2 offers an overview of the relevant literature. This is succeeded by Section 3 , which details the proposed methodology. Following this, Section 4 describes the experimental setup and discusses the results obtained with the proposed method. Lastly, Section 5 concludes the paper and proposes directions for future research. 2. Related Work In this section, previous works that have integrated vision and language models for the analysis of chest X-rays are reviewed. The focus is on examining how these approaches have utilized the strengths of both model types to analyze visual and textual data modalities to improve disease detection, diagnosis, and report generation from chest X-ray images. Wang et al. ( Wang et al. 2021 ) used paired chest X-ray images and text reports, as well as chest X-ray images and text reports individually from multiple institutions, to pre-train a transformer-based deep learning model in a self-supervised way. The pre-trained models were then applied to various downstream tasks, including X-ray classification, retrieval, and image regeneration. Their experimental results on these tasks showed that the proposed pre-training techniques led to more generalizable models. To extract effective representations of textual and visual data in the medical domain, Monajatipoor et al. ( Monajatipoor et al. 2021 ) proposed BERTHop, a transformer-based model built upon PixelHop ( Chen and Kuo 2019 ) and VisualBERT ( Li et al. 2019 ) . BERTHop enhances the ability to capture associations between these two modalities by first encoding the image and text, then extracting potential features from both. A transformer-based model then learns the relationships between these modalities. By utilizing appropriate vision and text extractors, BERTHop is capable of identifying abnormalities and associating them with textual labels. The model achieved high values of Area Under the Curve of the Receiver Operating Characteristic curve on imbalanced disease diagnosis datasets. Wiehe et al. ( Wiehe et al. 2022 ) investigated the adaptation of CLIP-based models ( Radford et

Read the complete text