LungGPT: A unified multimodal system for interpretable diagnosis and clinical decision support of respiratory diseases Authors: Jun Shao, Xingting Liu, Zhihan Zhang, Shu Liao, Jiaojiao Wu, Haibo Yang, Zi-Hao Zhao, Liwen Wang, Shu-Fan Liang, Xinglie Wang, Junyao Tang, Yuan Liu, Feng Shi, Dinggang Shen, Weimin Li, Chengdi Wang Published: 2026 ABSTRACT SummaryRespiratory diseases cause significant morbidity, yet diagnosis remains labor intensive and dependent on physician expertise. Here, we present LungGPT, a unified multimodal system trained on 147 million tokens of domain-specific electronic health records from 125,917 participants. LungGPT comprises two modules: LungGPT-Dx for respiratory disease diagnosis and early warning of critical illness, and LungGPT-Ex for interpretable diagnostic reasoning and treatment recommendations. In large-scale evaluations, LungGPT-Dx achieves a macro-average area under the curve (AUC) of 0.852 (95% confidence interval [CI]: 0.839–0.865) across 22 respiratory diseases, with disease-specific AUCs exceeding 0.900 for lung cancer and pulmonary tuberculosis. Crucially, the model further improves early warning of critical illness by incorporating chain-of-thought (CoT) reasoning into textual data and integrating computed tomography (CT) imaging features. LungGPT-Ex generates high-quality, interpretable reasoning that outperforms specialized clinical models and matches advanced general-purpose models such as GPT-4o and DeepSeek-R1 in correctness, completeness, and truthfulness. By bridging precision diagnostics and rapid decision-making, LungGPT provides a standardized framework to enhance clinical workflows and improve patient outcomes in respiratory healthcare.Keywords: respiratory diseases, multimodal model, large language model, clinical diagnosis, decision support Introduction Respiratory diseases, mainly including chronic obstructive pulmonary disease (COPD), lung cancer, pneumonia, and pulmonary tuberculosis (PTB), are characterized by high incidence and mortality rates, imposing a considerable disease burden worldwide.1,2,3,4 These diseases require rapid diagnosis and symptomatic treatment, and progression to critical illness, particularly severe pneumonia, is associated with mortality approaching 50%.5,6 However, many respiratory disorders present with similar clinical manifestations, making differential diagnosis inherently difficult. In clinical practice, the diagnostic process demands that physicians employ logical reasoning to discern correlations within medical histories, physical examinations, and auxiliary tests to deduce potential diagnoses.7,8 Manual review of fragmented clinical records is often labor intensive and time consuming,9 which could lead to physicians overlooking critical nuances and escalating the risk of misdiagnosis. These challenges highlight the urgent need for intelligent systems that support more accurate and efficient diagnosis. Recent advances in artificial intelligence (AI) have enabled a series of breakthroughs in biomedical applications.10,11,12,13,14 The field has evolved substantially from deep learning approaches, which excelled in pattern recognition but often required task-specific architectures and large labeled datasets.15 This trajectory advanced with the emergence of foundation models, particularly large language models (LLMs), which exhibit broad capabilities in knowledge encoding, text comprehension, and clinical reasoning, thus providing intelligent tools to enhance human productivity.16,17,18,19,20,21 Several general-purpose LLMs are demonstrating promise in their initial medical applications. A recent randomized controlled trial demonstrated that GPT-4 assistance in managing complex cases raised the management reasoning scores of physicians.22 Moreover, open-source LLMs such as the DeepSeek model show an enhanced capacity to assist in clinical decision-making for both prevalent and rare diseases.23,24,25 However, most evaluations have relied on public benchmarks, and their effectiveness in real-world clinical settings remains insufficiently characterized.26,27 The application of foundation models in healthcare is rapidly expanding and demonstrates remarkable potential across multiple clinical domains.28,29,30,31,32 The recent text-based generative model Delphi-2M successfully reconstructed longitudinal disease trajectories and predicted the incidence of thousands of diseases, achieving a mean area under the curve (AUC) exceeding 0.70.33 The foundation model Ark+, trained on a public chest radiography dataset, was pretrained by recycling the knowledge of heterogeneous expert labels.34 In addition, MUSK, a multimodal foundation model, leveraged vision-language alignment to fully exploit pathological image information. It showed superior predictive performance for lung cancer immunotherapy response with an AUC of 0.768.35 However, existing models are often applied to isolated clinical tasks and lack the interpretability essential for complex medical decision-making.36,37 It is necessary to develop integrated decision-support systems that can be embedded into clinical workflows, particularly in specialized fields such as respiratory medicine. In this study, an intelligent system for lung diseases powered by a Generative Pre-trained Transformer (LungGPT) was constructed by pretraining on 1.8 billion general medical tokens and fine-tuning on 147 million domain-specific tokens from 125,917 inpatients (Figure 1). LungGPT is designed to address four critical clinical tasks: (1) diagnosing multiple respiratory diseases, (2) providing early warning of critical illness by synergizing text records with computed tomography (CT) imaging features, (3) generating interpretable diagnostic reasoning, and (4) recommending evidence-based treatment via structured decision pathways. This end-to-end clinical decision-support system unifies diagnostic intelligence with generative assistance and enhances both the efficiency and interpretability of clinical workflows. Overview of the LungGPT system (A) The foundational datasets for the system. It included human dialog data with 5.8 billion tokens that were sourced from large-scale open-source repositories including WanJuan-CC and WuDaoCorpora; a medical corpus with 1.8 billion tokens that was derived from research papers, medical textbooks, and clinical guidelines; and 147 million tokens derived from domain-specific electronic health records (EHRs) from 125,917 participants. (B) Architecture of the LungGPT system. Based on the above datasets, LungGPT-Dx was optimized for respiratory disease classification, whereas LungGPT-Ex was optimized for evidence-grounded generation of diagnostic reasoning and decision support. RLHF, reinforcement learning from human feedback. (C) Network structure of the system. The system included text encoding, a multi-head attention mechanism, root mean square layer normalization (RMSNorm), quantized low-rank adaptation (QLoRA) for efficient parametric fine-tuning, a multilayer perceptron (MLP), and a linear layer for data transformation for medical analysis and decision-making. Results Pre-training corpora and clinical cohort For model pre-training, this study utilized a substantial corpus of 5.8 billion tokens of human dialog data including the safe and high-quality English webtext dataset WanJuan-CC and the super large-scale Chinese corpora WuDaoCorpora.38,39 Additionally, a specialized medical corpus of 1.8 billion tokens was assembled, which comprehensively integrated authoritative resources from international databases such as PubMed and MEDLINE, as well as the prominent Chinese academic database China National Knowledge Infrastructure (CNKI). For model fine-tuning, we further incorporated domain-specific data from 125,917 hospitalized participants totaling 147 million tokens. The electronic health record (EHR) data encompassed multi-source records, including demographics, chief complaints, imaging findings, radiological reports, laboratory test descriptions, and follow-up records. Following International Classification of Diseases, 10th revision (ICD-10) standardization, we identified a specific cohort of 29,215 patients whose primary diagnosis was a respiratory disease, encompassing a broad range of ICD-10 codes (Figure 2A; Table S1). These were further categorized into 22 major disease types through in-depth discussion and reviewed by senior physicians. Co-occurrence analysis revealed substantial multimorbidity across respiratory disease categories, with notable co-occurrence of pneumonia with COPD, pleural effusion (PE), and bronchiectasis (BRO) (Figures 2B and S1). Following cohort partitioning, the training and testing datasets showed comparable gender distributions, with male patients accounting for 63% (Table S2). In addition, the age distribution of patients in both the training and testing datasets was similar and the majority of participants were between 40 and 70 years old (Figure 2C). Characteristics of the respiratory disease cohort (A) A network graph illustrating the relationships between 22 respiratory diseases and their associated codes. Each node represents a disease, and the size of the node corresponds to the number of patients. (B) A chord diagram depicting the flow of respiratory comorbidities among participants. The width of each flow line was proportional to the number of participants sharing the corresponding comorbid conditions. (C) Age distribution of participants by gender in the training and testing datasets. ASTH, asthma; BRO, bronchiectasis; CB, chronic bronchitis; COPD, chronic obstructive pulmonary disease; ILD, interstitial lung disease; LA, lung abscess; LBT, lung benign tumor; LC, lung cancer; LI, lung injury; MD, mediastinal disease; PC, pulmonary cyst; PE, pleural effusion; PEMB, pulmonary embolism; PEMP, pulmonary emphysema; PH, pulmonary hypertension; PNC, pneumoconiosis; PNE, pneumonia; PT, pneumothorax; PTB, pulmonary tuberculosis; RF, rib fracture; SAS, sleep apnea syndrome; URTI, upper respiratory tract infection. Diagnostic performance across respiratory diseases LungGPT-Dx, a pretrained model fine-tuned with EHR data, was used to diagnose respiratory diseases. The system performed well in the evaluation of 22 respiratory diseases with AUC values ranging from 0.754 to 0.967 and a macro-average AUC of 0.852 (95% confidence interval [CI]: 0.839–0.865) in the testing dataset (Figure 3A). The diagnostic performance of the LungGPT-Dx model for most diseases reached AUC values greater than 0.800. Notably, the model demonstrated strong discriminative performance for several major diseases, including pulmonary hypertension (AUC 0.948, 95% CI: 0.921-0.970), lung cancer (AUC: 0.921, 95% CI: 0.912-0.929), PTB (AUC: 0.903, 95% CI: 0.872-0.930), asthma (AUC: 0.902, 95% CI: 0.863-0.939), interstitial lung disease (AUC: 0.883, 95% CI: 0.856-0.910), COPD (AUC: 0.863, 95% CI: 0.847-0.881), and pneumonia (AUC: 0.821, 95% CI: 0.810-0.831) (Figure 3B; Table S3). Furthermore, the model demonstrated robust predictive performance across the majority of diseases, achieving high top-1 and top-3 accuracies, alongside a strong F1 score, recall, and precision (Figure 3C). Specifically, the macro-averaged metrics reached 0.719, 0.767, and 0.763 for the top-1 accuracy, top-3 accuracy, and precision. Further subgroup analyses revealed performance variations across gender. LungGPT-Dx showed superior diagnostic performance in female patients, with a macro-average AUC of 0.881, compared with 0.836 in male patients (Figure S2). Diagnostic performance of LungGPT-Dx across respiratory diseases (A) Confusion matrix of the predicted versus actual categories for 22 respiratory diseases. Each cell represented the ratio of predicted to actual numbers of samples, with the color intensity indicating the level of the ratio. (B) Bar plot showing the diagnostic area under the curve (AUC) of LungGPT-Dx for each of the 22 respiratory diseases. (C) Radial bar plot displaying diagnostic metrics of LungGPT-Dx across the 22 diseases: top-1 accuracy (ACC), top-3 ACC, F1 score, recall (REC), and precision (PRE). When compared with state-of-the-art baseline LLMs based on a 349-case evaluation subset, LungGPT-Dx demonstrated superior group-wise performance, with an AUC of 0.862, surpassing comparator models including DeepSeek-R1 (0.829), PULSE (0.834), GPT-4o (0.825), HuaTuoGPT (0.715), and Qwen (0.672) (Figure S3; Table S4). It also demonstrated superior top-1 accuracy (0.756), F1 score (0.758), and precision (0.788). These findings underscore LungGPT-Dx’s exceptional diagnostic precision and its significant advantages over existing models, positioning it as a potential AI-based diagnostic tool for respiratory diseases. Early warning of critical illness In critical care management, particularly for life-threatening respiratory emergencies, rapid and precise assessment is of utmost importance. To address this, LungGPT implemented a hierarchical diagnostic framework for critical illness (Figure 4A). The system initially processed unstructured EHR data for disease detection. Then, it employed prompt engineering with chain-of-thought (CoT) reasoning to convert these textual findings into structured information, which was subsequently fused with CT imaging data through AI models to stratify severe conditions. As illustrated in the CoT schematic for severe pneumonia (SP), the model simulated clinical decision-making by systematically evaluating major and minor criteria derived from both textual and visual modalities (Figure 4B).40 Ultimately, this multimodal paradigm not only enhanced risk stratification but also provided interpretability for its subsequent reasoning and diagnostic decisions. Workflow and performance evaluation of the LungGPT in diagnosing critical illness (A) The hierarchical workflow of the LungGPT for identifying critical illness. In Step 1, LungGPT obtained primary diagnostic results and the corresponding explanatory content by analyzing the electronic health record (EHR) text. In Step 2, the system transformed the textual diagnoses into structured database information using prompt engineering with chain-of-thought (CoT) and introduced an artificial intelligence model for computed tomography (CT) images, thereby achieving the stratification of severe and acute conditions such as severe pneumonia (SP) and acute pulmonary embolism (APE). PNE, pneumonia; PEMB, pulmonary embolism. (B) CoT process for the diagnosis of severe pneumonia. The CoT technique was utilized to transform the EHR text and CT images into structured information, which enabled disease diagnosis based on corresponding clinical criteria. (C) Performance evaluation metrics for identifying SP, rapidly progressive lung cancer (RPLC), APE, and acute exacerbation of chronic obstructive pulmonary disease (AECOPD). Quantitative performance evaluations demonstrated that this multimodal integration significantly enhanced diagnostic accuracy (Figure 4C). By combining CT imaging data with text-based CoT, the model showed substantial improvements in identifying severe conditions, increasing the AUC for SP from 0.776 to 0.802. It also markedly improved the AUC for rapidly progressive lung cancer (RPLC, volume doubling time <400 days) from 0.595 to 0.899, identifying high-risk patients who warrant expedited clinical assessment to avoid delaying treatment. Furthermore, for acute conditions, this integration increased the AUC for acute pulmonary embolism (APE) from 0.714 to 0.802 and for acute exacerbation of chronic obstructive pulmonary disease (AECOPD) from 0.816 to 0.869, facilitating timely clinical interventions for high-risk patients. These improvements in identifying critically ill patients may support more timely triage and inform targeted clinical management. Diagnostic reasoning evaluation Furthermore, LungGPT-Ex summarized medical texts to generate diagnostic reasoning and treatment planning with retrieval-augmented generation (RAG) and prompts (Figure 5A). Three board-certified clinicians (3–5 years of clinical experience) independently evaluated outputs for 600 diagnostic cases. The clinicians assessed the model outputs in terms of three key metrics including correctness, completeness, and truthfulness on a 3-point Likert scale (1: fair, 2: good, and 3: excellent). Clinical assessment of the ability of LungGPT to generate text (A) Study design for clinician evaluation of diagnostic reasoning and decision-generation outputs produced by LungGPT-Ex and five comparator models using correctness, completeness, and truthfulness. (B) Distributions of reader scores for diagnostic reasoning. The heatmaps denoted reader preferences as measured by a 3-point Likert scale. The box indicates the interquartile range (IQR), whiskers extend to 1.5 × IQR (Tukey’s method), dots represent outliers, and + denotes the mean. Statistical significance was determined using Friedman test followed by Dunn’s multiple comparisons test. ∗∗∗p < 0.001. ns, not significant. (C) Distribution of reader scores for decision generation. The heatmaps denoted reader preference as measured by a 3-point Likert scale. The box indicates the IQR, whiskers extend to 1.5 × IQR (Tukey’s method), dots represent outliers, and + denotes the mean. Statistical significance was determined using Friedman test followed by Dunn’s multiple comparisons test. ∗∗p < 0.01, ∗∗∗p < 0.001. ns, not significant. In terms of diagnostic reasoning, LungGPT-Ex outperformed most LLMs (Figures 5B and S4; Table S5). For correctness, LungGPT-Ex achieved an average score of 2.84, reflecting its ability to minimize false or misleading information. This performance significantly exceeded that of PULSE (1.34), HuaTuoGPT (1.38), Qwen (2.31), and DeepSeek-R1 (2.33), while closely matching that of GPT-4o (2.89). Regarding completeness, LungGPT-Ex scored 2.71, highlighting its capacity to provide detailed and comprehensive diagnostic explanations. This score surpassed PULSE (1.25), HuaTuoGPT (1.33), Qwen (1.84), and DeepSeek-R1 (2.27). Its truthfulness score (2.90) was comparable to GPT-4o (2.92) and DeepSeek-R1 (2.87) and significantly outperformed PULSE (1.43), HuaTuoGPT (1.54), and Qwen (2.59). Decision generation evaluation In the task of treatment planning, LungGPT-Ex continued to demonstrate exceptional performance (Figures 5C and S5; Table S6). It achieved a correctness score of 2.63, outperforming PULSE (1.18), HuaTuoGPT (1.27), Qwen (2.44), and DeepSeek-R1 (2.45), and nearly matching GPT-4o (2.72). With respect to completeness, LungGPT-Ex scored 2.65, on par with GPT-4o (2.64) and substantially higher than DeepSeek-R1 (2.32), Qwen (2.27), and HuaTuoGPT (1.30). Its truthfulness score was particularly notable, achieving 2.92 and comparable to GPT-4o, DeepSeek-R1, and Qwen, significantly outperforming HuaTuoGPT (1.57) and PULSE (1.35). Beyond diagnostic explanation, LungGPT-Ex generated patient-specific management suggestions spanning scheduled surveillance and follow-up, pharmacologic therapy, surgical or interventional evaluation when indicated, multidisciplinary team consultation, comprehensive treatment planning, and so on. By organizing diagnostic evidence and potential management options into a structured format, LungGPT-Ex could assist clinicians in interpreting model outputs, thereby improving the efficiency of clinical decision-making. Clinical application of LungGPT In practice, LungGPT used unstructured EHR data as input and then produced concise summaries including diagnoses, evidence-based diagnostic reasoning, and tailored decision recommendations (Figure 6). As illustrated in a representative case of a 57-year-old male presenting with recurrent cough, sputum production, dyspnea, and hemoptysis, the model accurately identified severe pneumonia—supported by clinical manifestations such as acute symptom exacerbation, auscultatory findings of wet rales, and chest CT revealing scattered patches, nodules, and consolidations in both lungs. Additionally, LungGPT recognized acute exacerbation of COPD and chronic pulmonary heart disease, integrating longitudinal history and multimodality data into a coherent diagnostic narrative. The system further provided clear therapeutic guidance for severe cases, including prioritized antibiotic treatment, cardiopulmonary function support, and individualized monitoring plans. The clinical application of LungGPT was facilitated by a streamlined interface with several steps (Figure S6). Specifically, it provided tailored recommendations for key disease categories including infectious, neoplastic, airway, and interstitial lung diseases. This example illustrates how LungGPT automatically generates structured diagnostic reasoning and clinical decision-support recommendations from complex patient records. Application of LungGPT for tailored diagnosis and decision recommendations in respiratory diseases After inputting the prompt command, the system could analyze admission records to diagnose disease. It also generated text summarized to provide clinicians with diagnostic reasoning and decision recommendations. This example highlighted the identification of severe pneumonia, supported by clinical findings such as recurrent cough and imaging results, and delivered clear therapeutic guidance, notably outlining specific interventions like oxygen therapy and anti-infective regimens for critical cases. All generated outputs were reviewed under clinician supervision, and final diagnostic and treatment decisions remained the responsibility of the treating physician. Discussion In this study, a domain-specific multimodal system named LungGPT was developed for respiratory disease diagnosis and clinical decision support. This system was adept at processing information from multiple sources, including chief complaints, imaging findings, and laboratory tests to classify 22 respiratory diseases. Moreover, LungGPT excelled in the timely identification of critical conditions and provided intervention options. Meanwhile, LungGPT was capable of generating diagnostic reasoning and treatment planning based on specific prompts. Its performance was superior to that of most existing models, positioning it as a powerful auxiliary tool in clinical practice. The rapid advancement of AI has fundamentally transformed the execution of healthcare tasks.41,42,43,44 A paradigmatic development in modern AI is the emergence of foundation models.45 Leveraging large-scale datasets for pretraining and fine-tuning, general-purpose AI systems capably serve a broad spectrum of downstream tasks. LungGPT achieved superior diagnostic performance with a notable enhancement after pretraining and fine-tuning. This advantage was largely attributable to the large-scale, high-quality clinical corpus derived from 125,917 inpatient EHRs. This underscores the significant impact of domain-specific fine-tuning on amplifying the capabilities of LLMs.46,47 However, the diagnostic performance remains suboptimal for some diseases. On the one hand, overlapping features and hierarchical disease coding relationships often complicate diagnosis given the complex comorbid relationships. To address this, top-3 accuracy was additionally reported to account for clinically plausible alternative diagnoses in the ranked model outputs. On the other hand, the diagnostic accuracy for certain diseases was hindered by limited data volume, suggesting that zero-shot or few-shot learning may offer promising solutions for further improvement.48 Another paradigm in modern AI focuses on generative models, which integrate high-dimensional data to create content spanning text, images, and molecular designs.45 However, general-purpose LLMs often lack the rigorous precision required for medical applications. Leveraging real-world cases as prompts, LungGPT generated diagnostic reasoning and decision-making schemes whose format and phrasing closely mirror those of authentic inpatient records. The model was comparable to GPT-4o in terms of correctness, completeness, and truthfulness, providing powerful technical support to assist physicians in improving their productivity. Meanwhile, through RAG that draws on authoritative sources including guidelines, expert consensus, and professional textbooks, LungGPT-Ex generated clinical summaries that remain fully aligned with contemporary practice. This ability to update and standardize prompts in a timely manner endowed this model with higher accuracy in the text generation for respiratory diseases.49 Future developments could incorporate real-time dialog functionality, enabling more natural and efficient interactions between physicians and the model.50,51 LLMs leverage encoded medical knowledge to aid clinicians toward precision diagnostic efficiency. Google introduced Med-PaLM, which integrated clinical knowledge with instruction-tuned learning and achieved expert-level performance on the MultiMedQA benchmark.52 Its successor, Med-PaLM 2, further incorporated retrieval and verification strategies, attaining an accuracy of 86.5% on the MedQA dataset, which represented a 19% improvement over the Med-PaLM.53 In respiratory medicine, foundation models cover a wide range of aspects such as disease screening, diagnosis, and treatment. For example, GPT-4o has been used to evaluate malignancy progression of pulmonary nodules to support lung cancer screening by dynamically monitoring the size and morphology.54 While models like text-based Delphi-2M and imaging-based Ark+ can predict respiratory diseases, they lack the capability for treatment recommendation.33,34 In this study, LungGPT unified diagnostic inference and decision support, positioning itself for seamless integration into routine practice. Guided by the clinical demands of respiratory medicine, LungGPT has preliminarily explored the integration of data from various sources and has achieved promising results in the decision support of various diseases. Currently, multimodal generative models have advanced rapidly.55,56,57 The integration of multimodal data enables clinicians to gain a comprehensive understanding of diseases to enhance the accuracy of diagnosis, alleviating the limitations imposed by single-modality data.58,59,60 For instance, BiomedGPT, as a vision-language generalist model, utilizes joint learning of imaging and textual features to achieve cross-modal alignment, which enables it to perform a wide range of biomedical tasks.61 Nevertheless, real-world deployment confronts formidable challenges. Datasets must be watermarked for full traceability,62 while hallucinations, biases, and equity concerns that are inherent to LLMs continue to impede adoption.63 Consequently, clinicians, researchers, health-policy experts, and patient representatives must be integrated from the earliest design and development stages to safeguard fairness and ethical alignment.64 In summary, LungGPT, as a domain-specific LLM trained on large-scale data, excelled in predicting 22 respiratory diseases and risk stratification for critical illness. It furthermore generated interpretable and traceable diagnostic reasoning and decision recommendations. The multimodal system provides an end-to-end framework spanning diagnosis, early warning of critical illness, and decision support. This system was expected to substantially enhance the quality of healthcare services. Limitations of the study The clinical translation of LungGPT confronts several challenges. First, the generalizability and robustness of LungGPT require further external validation across diverse demographic and regional settings. Second, the integration of multimodal and dynamic clinical data is needed to evolve LungGPT into a generalist medical AI agent, thereby operationalizing cross-scale precision medicine and enabling comprehensive exploitation of molecular-to-patient information within an AI framework to deliver truly individualized therapeutic decisions.65 In addition, progression from an experimental architecture to a regulated decision-support tool demands rigorous regulatory scrutiny. To address these issues, prospective and multi-center studies are necessary to iteratively refine the model. Resource availability Lead contact Further information and reasonable requests for resources should be directed to the lead contact, Chengdi Wang (chengdi_wang@scu.edu.cn). Materials availability This study did not generate new reagents. Data and code availability •The pretraining datasets WanJuanCC (https://opendatalab.com/OpenDataLab/WanJuanCC) and WuDaoCorpora (https://openi.pcl.ac.cn/BAAI/WuDao-Data) are publicly available, with relevant links listed in the key resources table. The fine-tuning dataset (de-identified respiratory cases) sourced from West China Hospital of Sichuan University are available on reasonable request by contacting the corresponding authors. The pretraining datasets WanJuanCC (https://opendatalab.com/OpenDataLab/WanJuanCC) and WuDaoCorpora (https://openi.pcl.ac.cn/BAAI/WuDao-Data) are publicly available, with relevant links listed in the key resources table. The fine-tuning dataset (de-identified respiratory cases) sourced from West China Hospital of Sichuan University are available on reasonable request by contacting the corresponding authors. •The custom code for training the LungGPT model was written in Python with PyTorch. The code and related documentation are available publicly: https://github.com/simonsf/LungGPT. The custom code for training the LungGPT model was written in Python with PyTorch. The code and related documentation are available publicly: https://github.com/simonsf/LungGPT. •Any additional information required to reanalyze the data reported in this work is available from the lead contact upon request. Any additional information required to reanalyze the data reported in this work is available from the lead contact upon request. Acknowledgments This research was supported by the Noncommunicable Chronic Diseases-National Science and Technology Major Project (2024ZD0528600/2024ZD0528604 and 2024ZD0529504/2024ZD0529500); National Natural Science Foundation of China (82341083 and 82470109); the Science and Technology Project of Sichuan (2026NSFSCZY0142 and 2026NSFSC1934); the Fundamental Research Funds for the Central Universities (SCU2026D010); the 1·3·5 Project for Disciplines Excellence, West China Hospital, Sichuan University (ZYYC23027); and the 1·3·5 Project of State Key Laboratory of Respiratory Health and Multimorbidity, West China Hospital, Sichuan University (RHM24208). Author contributions C.W., D.S., W.L., and F.S. conceived the study, designed the overall research framework, and supervised the project. C.W., J.S., X.L., S.L., X.W., and H.Y. implemented and performed the experiments. J.S., S.-F.L., Z.-H.Z., J.W., H.Y., L.W., S.L., J.T., and Y.L. contributed to data preparation, computational analysis, model evaluation, and interpretation of the experimental results. C.W., J.S., X.L., Z.Z., J.W., and F.S. wrote the paper. All authors discussed the results, reviewed and approved the final manuscript, and agree to be accountable for the integrity of the work. Declaration of interests S.L., J.W., H.Y., X.W., J.T., Y.L., and F.S. are employees of United Imaging Intelligence. The company had no involvement in the design, execution, surveillance, data analysis, or interpretation and provided no financial support for the study. STAR★Methods Key resources table REAGENT or RESOURCESOURCEIDENTIFIERDeposited dataWanJuanCC for LungGPT pretrainingShanghai AI Lab (Qiu et al.38)https://opendatalab.com/OpenDataLab/WanJuanCCWuDaoCorpora for LungGPT pretrainingBeijing Academy of Artificial Intelligence (Yuan et al.39)https://openi.pcl.ac.cn/BAAI/WuDao-DataInternational Classification of Diseases, 10th Revision (ICD-10)National Health Security Administrationhttps://code.nhsa.gov.cn/jbzd/public/dataWesterSearch.htmlSoftware and algorithmsPython version 3.11.9Python Software Foundationhttps://www.python.org/OpenAI Python library 1.30.1OpenAIhttps://platform.openai.com/docs/librariesTransformers algorithmHugging Face (Wolf et al.66)https://huggingface.co/docs/transformers/PULSE modelOpenMEDLabhttps://github.com/openmedlab/PULSEHuaTuoGPT modelFreedomIntelligence (Zhang et al.67)https://github.com/FreedomIntelligence/HuatuoGPTQwen modelQwen Team, Alibaba Cloud (Bai et al.68)https://qwenlm.github.io/GPT-4o modelOpenAIhttps://openai.com/index/hello-gpt-4o/DeepSeek-R1DeepSeek Inc. (Bi et al.69)https://huggingface.co/deepseek-ai/DeepSeek-R1LungGPT code and modelThis studyhttps://github.com/simonsf/LungGPT Experimental model and study participant details The data used in this study consist of both general medical corpora and domain-specific electronic health record (EHR) data. Specifically, the general medical corpus was employed during pretraining to enable the large language model (LLM) to acquire representations relevant to clinical reasoning. Moreover, we ensured that this pretraining did not compromise the model’s instruction-following and interactive capabilities. To achieve this goal, we incorporated both an online human dialogue corpus and a general medical corpus sourced from research papers, textbooks, and clinical guidelines. Domain-specific EHR data from 125,917 participants were derived from West China Hospital of Sichuan University for the period from 2009 to 2021 and were utilized in the fine-tuning phase. These records included comprehensive information about patient conditions, such as demographic details, chief complaints, imaging findings, radiological reports, laboratory test descriptions, pathological reports, pulmonary function tests, and follow-up records. This study was approved by the Research Ethics Committee in the West China Hospital of Sichuan University, Chengdu, China (No. 2024.1762). Informed consent was waived because of the retrospective nature of the study. Method details Multimodal data processing Data curation To enhance the capabilities of models in both general communication and specialized domains, the pre-training data incorporated 5.8 billion tokens of open-source human dialogue from WanJuan-CC (English) and WuDaoCorpora (Chinese). Furthermore, to ground the model in robust medical knowledge, a dedicated corpus of 1.8 billion tokens was constructed from peer-reviewed research and clinical guidelines indexed in PubMed, MEDLINE, and China National Knowledge Infrastructure (CNKI). This general medical corpus comprised the latest clinical studies, specialized textbooks, and practice guidelines related to respiratory disease management. Domain-specific EHR data were derived from 125,917 hospitalized patients with at least one respiratory disease diagnosis. Eligible patients were those with a confirmed respiratory disease and complete inpatient EHRs. Patients with unclear diagnoses or suspected misdiagnosis, as well as those who could not be clearly matched to a specific respiratory disease, were excluded. Cases deemed unsuitable for inclusion by the researchers were also excluded. A total of 29,215 patients with a primary diagnosis of respiratory disease were included in this study, and the dataset was partitioned into a training set and a testing set at an 8:2 ratio. ICD-10 label encoding To ensure data consistency and comparability, we standardized all patients’ diagnostic information by ICD-10 coding, which was manually annotated (Table S1). Initially, we implemented a fuzzy matching algorithm that compared the disease names written by physicians with the ICD-10 code list to find the most similar matches. This step aimed to reduce the annotation workload. Despite the assistance from fuzzy matching, all the coding was manually verified and annotated by our team of experienced clinical coders. This manual process ensured that each diagnosis was accurately matched to the correct ICD-10 code. By relying on manual annotation, supported by fuzzy matching for efficiency, we maintained a high level of accuracy and consistency in the ICD-10 coding process for our dataset. Chest CT image acquisition Chest computed tomography (CT) images were retrieved in DICOM format from the Picture Archiving and Communication System (PACS). Eligibility criteria included volumetric CT acquisition within 48 hours before or after the index admission, axial slice thickness ≤ 2.5 mm, and complete lung coverage from the lung apices to the costophrenic angles. Patients who met these CT criteria and had complete EHR data were included in the multimodal cohort for critical illness prediction. Severe pneumonia, rapidly progressive lung cancer, AECOPD, and acute pulmonary embolism were selected as the critical illness subtypes for investigation. The role of multimodal information fusion in enabling precise differential diagnosis was assessed across these conditions. LungGPT architecture and workflow Task-specific large language models We developed two distinct LLMs for the tasks of disease diagnosis and decision explanation, named LungGPT-Dx (7B) and LungGPT-Ex (110B), respectively. Both models were based on the Qwen architecture68 and trained using quantized low-rank adaptation (QLoRA).70 The LungGPT-Dx was fine-tuned for ICD-10 annotated disease diagnosis, which significantly enhanced its performance in classification tasks. However, fully fine-tuning an LLM often compromises its conversational and logical reasoning abilities.71 Therefore, LungGPT-Ex underwent supervised fine-tuning (SFT) separately and reinforcement learning from human feedback (RLHF)72 based on Qwen-110B, enabling it to have a nuanced understanding of diagnostic criteria while maintaining robust conversational capabilities. Retrieval-augmented generation The general medical corpus underwent data preprocessing to construct a textual knowledge base, which was then embedded into vectors to form a vectorized knowledge database. The knowledge database was designed for continuous real-time updates. In situations where updating the base model with pretraining was not feasible, the real-time augmentation of medical knowledge through retrieval-augmented generation (RAG) effectively mitigated the model’s limitations in handling medical knowledge. In implementation, retrieved clinically relevant passages from the knowledge base. This could provide reference information for LungGPT-Ex in terms of diagnostic criteria and maintain reliable answers. Integrated pipeline The workflow of our proposed framework included LungGPT-Dx, LungGPT-Ex, and the structured database. In practical applications, the patient’s EHR was first input into LungGPT-Dx, which generated a textual diagnosis of potential respiratory diseases. Next, the patient’s EHR was concatenated with the diagnostic output from LungGPT-Dx to create a query, which was then processed via LangChain73 to retrieve relevant knowledge from a structured database. This integration of EHR data and diagnostic results formed a knowledge-enhanced input. This enhanced input was subsequently fed into LungGPT-Ex, which was responsible for providing a comprehensive explanation of the diagnosis. LungGPT-Ex then delivered a detailed diagnostic explanation and generated treatment planning. To achieve precise identification of critical respiratory conditions, we developed a hierarchical framework that integrated textual clinical data with radiological features. First, a chain-of-thought (CoT) prompting strategy74 was employed to identify key physiological indicators and symptomatic abnormalities within the EHR, mapping these findings to predefined clinical semantic categories. Simultaneously, a deep learning model was utilized to extract high-dimensional features from chest CT images.75 Through this integrated reasoning process, the system generated evidence-based syntheses aligned with established clinical guidelines, enabling the high-fidelity identification of critically ill patients. Model benchmarking and evaluation Comparator models To ensure fair benchmarking between LungGPT and the baseline LLM models, we implemented strict evaluation protocols. All the comparator models were used in their original public releases without domain adaptation in our study. Specifically, PULSE was implemented using the Bloomz architecture; the publicly available LLaMA-based 7B medical version of HuaTuoGPT was utilized, the Qwen-7B general-purpose implementation of Qwen was employed, GPT-4o was accessed via an official API without customization, and the publicly available DeepSeek-R1 was used for DeepSeek. The complete model configurations were detailed in Table S7. Respiratory disease diagnosis For the performance evaluation of LungGPT-Dx, we employed several key metrics including area under the receiver-operating characteristic curve (AUC), the top-1/3 accuracy (ACC), precision (PRE), recall (REC), and the F1 score (F1). These metrics were critical for assessing the effectiveness of our classification approach. The AUC quantified the model’s ability to discriminate between positive and negative cases across all classification thresholds, with higher values indicating better overall diagnostic performance. Top-1/3 ACC measured the frequency with which the correct diagnosis appears within the top 1 or 3 predictions made by the model. They provide a clear picture of the model’s ranking ability, with top-1 accuracy being the strictest measure and top-3 accuracy offering a broader view of the model’s capabilities. PRE was the ratio of the number of correctly predicted positive observations to the total number of predicted positive observations. It is a measure of the model’s accuracy in identifying relevant instances among the retrieved instances. REC or sensitivity, was the ratio of the number of correctly predicted positive observations to the total number of observations in the actual class. It assesses the model’s ability to identify all relevant instances. The F1 score was the harmonic mean of precision and recall, and it provided a single metric that balanced concerns regarding both false positives and false negatives. It is particularly useful when dealing with imbalanced datasets. Diagnostic reasoning and decision support Considering the evaluation of the diagnostic criteria provided by the LungGPT-Ex, we adopted the Likert Scale Scoring system. The Likert scale was a widely recognized psychometric tool used to gauge subjective assessments, where respondents specify their level of agreement or disagreement with a statement. Correctness, completeness, and truthfulness were the key evaluation indicators for the responses of LLMs. Correctness was used to evaluate whether the diagnostic criteria can support the diagnostic result accurately to ensure alignment with established medical knowledge. Completeness measured whether the diagnostic criteria cover all necessary aspects needed to reach a proper diagnosis and was used to assess the thoroughness of the LLM’s output. Truthfulness was used to assess whether the diagnostic criteria are based on real and verifiable information to ensure the reliability and trustworthiness of the information provided by the LLM. Each criterion was assessed on a 3-point scale, where 3 represents the best score and 1 represents the worst score. By combining these quantitative and qualitative evaluation methods, we aimed to thoroughly assess the performance and reliability of this proposed method for respiratory disease diagnosis and to ensure that it meets the high standards required for clinical application. Implementation details LungGPT integrated with hospital information systems (HIS) via a secure application programming interface (API), compliant with data privacy regulations. Clinicians could access a web-based dashboard, input a patient’s identifier to retrieve structured (demographics, medical history) and unstructured (radiology reports, progress notes) data, with support for manual upload of additional lab results or imaging findings. Then LungGPT-Dx generated a ranked list of respiratory diseases, and LungGPT-Ex produced diagnostic rationales and treatment recommendations using a vectorized knowledge base of clinical guidelines. Outputs were reviewed by clinicians, who retained final decision-making authority, with options to adjust suggestions based on bedside observations. The interface underwent iterative usability testing by participating clinicians. Feedback from participating clinicians, including observations on usability and diagnostic alignment, informed iterative refinements to the interface and model outputs. Quantification and statistical analysis To evaluate the diagnostic performance of LungGPT, metrics such as AUC, the top-1/3 ACC, PRE, REC, and the F1 score were adopted. 95% confidence intervals for AUCs were estimated using the DeLong method. To evaluate the quality of the diagnostic explanations and treatment planning generated by LungGPT, three clinicians independently assessed outputs by using a 3-point Likert scale across three distinct dimensions (correctness, completeness, and truthfulness). To compare the clinical scores quantitatively, statistical analyses were performed using the Friedman test followed by Dunn’s multiple comparisons test. All the statistical analyses were performed using SPSS (version 26.0), where ∗ indicating p < 0.05, ∗∗ indicating p < 0.01, and ∗∗∗ indicating p < 0.001. Published: September 3, 2026 Footnotes Supplementary data related to this article can be found online at https://doi.org/10.1016/j.xcrm.2026.102926. Contributor Information Feng Shi, Email: feng.shi@uii-ai.com. Dinggang Shen, Email: dinggang.shen@gmail.com. Weimin Li, Email: weimi003@scu.edu.cn. Chengdi Wang, Email: chengdi_wang@scu.edu.cn. Supplemental information Document S1. Figures S1–S5 and Tables S1–S7 Document S2. Article plus supplemental information References Associated Data Supplementary Materials Document S1. Figures S1–S5 and Tables S1–S7 Document S2. Article plus supplemental information Data Availability Statement •The pretraining datasets WanJuanCC (https://opendatalab.com/OpenDataLab/WanJuanCC) and WuDaoCorpora (https://openi.pcl.ac.cn/BAAI/WuDao-Data) are publicly available, with relevant links listed in the key resources table. The fine-tuning dataset (de-identified respiratory cases) sourced from West China Hospital of Sichuan University are available on reasonable request by contacting the corresponding authors. The pretraining datasets WanJuanCC (https://opendatalab.com/OpenDataLab/WanJuanCC) and WuDaoCorpora (https://openi.pcl.ac.cn/BAAI/WuDao-Data) are publicly available, with relevant links listed in the key resources table. The fine-tuning dataset (de-identified respiratory cases) sourced from West China Hospital of Sichuan University are available on reasonable request by contacting the corresponding authors. •The custom code for training the LungGPT model was written in Python with PyTorch. The code and related documentation are available publicly: https://github.com/simonsf/LungGPT. The custom code for training the LungGPT model was written in Python with PyTorch. The code and related documentation are available publicly: https://github.com/simonsf/LungGPT. •Any additional information required to reanalyze the data reported in this work is available from the lead contact upon request. Any additional information required to reanalyze the data reported in this work is available from the lead contact upon request.