Clinical large language model centered on electronic medical records Authors: Yan Zhuang, Bo Wang, Chengliang Yin, Junyan Zhang, Fanqing Meng, Jianfei Zhao, Qingyong Su, Xuan Zhao, Xiuxing Li, Ping Hu, Shiyuan Liu, Rilige Wu, Yun Hua, Wei Dong, Bing Wei, Li Zhang, Lei Zheng, João Conde, Ge Shi, Chong Feng, Kunlun He Published: 2026 ABSTRACT In the quest to enhance medical consultation, our study introduces AI4Doctor, a sophisticated large-language model (LLM) tailored for the clinical domain. At the heart of AI4Doctor is an innovative integration strategy that synergizes distilled data extracted from electronic medical records (EMR) with empirical insights gathered from practicing physicians during the supervised fine-tuning. Although existing platforms offer informative responses, they fall short of replicating the nuanced decision-making processes of medical professionals, particularly in complex, integrative diagnostic scenarios. Motivated by the need to create a realistic medical practice environment, we propose that a combination of direct knowledge transfer from seasoned doctors and the strategic use of EMR can augment the abilities of LLM, enabling it to more closely mimic the clinical acumen of healthcare practitioners. To navigate the complexities of merging diverse instructional sources, we employ a curriculum learning approach during the fine-tuning process. Moreover, we advance our model’s performance by developing a reward system that incentivizes the alignment of the LLM’s outputs with the valuable attributes inherent in both doctors’ expertise, including diagnostic priors, risk thresholds, and heuristic saliencies accumulated from practice and EMR data. This is achieved through a novel reinforcement-learning approach. Besides, we introduce a new benchmark involving a comparative evaluation. We utilize a subjective evaluation system wherein experts critically assess the responses from a professional perspective as well. Our research underscores the potential of this hybrid model to serve as a robust tool in medical consultations, bridging the gap between artificial intelligence and real-world clinical practice. Introduction Large Language Models (LLMs) have demonstrated remarkable potential in healthcare applications1,2 achieving performance comparable to medical professionals in knowledge-based assessments such as medical licensing examinations3. Despite these advances, LLMs exhibit significant limitations when applied to complex clinical scenarios that require contextual understanding and practical expertise3–8. Most concerning is their tendency to generate hallucinations—confident but factually incorrect information—which poses substantial safety and ethical risks in clinical settings9–12. These limitations highlight the critical gap between theoretical medical knowledge and practical clinical application in current LLM implementations. Recent years have witnessed rapid development in medical LLMs across multiple languages and scales. In English, Med-PaLM 23 has achieved remarkable performance on medical licensing exams, surpassing the passing threshold on both the US Medical Licensing Exam (USMLE) and Multiple Choice Questions (MCQs). GPT-4V Medical demonstrates advanced capabilities in medical image understanding and clinical reasoning. Clinical-BERT and BioBERT1 have shown effectiveness in specific clinical NLP tasks such as named entity recognition and relation extraction. Among Chinese medical LLMs, HuaTuoGPT13, trained on 400B tokens of medical text, exhibits strong performance in medical dialogue and knowledge retrieval. MING and DoctorGLM14, while showing promise in medical question answering, generally lag behind their English counterparts due to limited high-quality Chinese medical corpus. Medical LLMs currently rely primarily on medical literature, textbooks, and online resources for training1,2,15,16, with limited integration of real clinical data. This approach, while providing broad medical knowledge coverage, often falls short in practical clinical scenarios. The disconnect between theoretical knowledge resources and the complex, contextual nature of clinical decision-making contributes significantly to current limitations. This fundamental limitation in training data composition highlights the need for more clinically relevant resources to enhance medical LLMs. Electronic Medical Records (EMRs) represent a valuable yet underutilized solution for addressing these deficiencies17–22. As comprehensive repositories of clinical workflows, diagnostic reasoning, and treatment patterns, EMRs capture the nuanced decision-making processes essential for effective clinical practice13,15,16 Unlike static knowledge bases, EMRs provide dynamic, patient-specific information reflecting real clinical decision-making processes. Although they contain rich, structured clinical evidence that captures relationships between diagnoses, treatments, and outcomes. Incorporating EMR data could potentially bridge the gap between theoretical knowledge and practical application in medical LLMs, though significant challenges exist in processing their structured formats, varying quality, and specialized terminology requirements. Prior EMR‑based systems (e.g., EMR‑conditioned LLMs and temporal decision‑support models) often rely on explicit rule templates, fixed‑interval resampling, or retrieval‑centric pipelines. In contrast, AI4Doctor emphasizes scalable pre-training/fine‑tuning with de‑identified, episode‑structured EMR context and introduces STG to capture multi‑resolution temporal dynamics and missingness. This design prioritizes case‑grounded reasoning over heuristic prompting or post‑hoc retrieval. We introduce AI4Doctor, a unified framework that unlocks the clinical value of electronic medical records (EMRs) through a sequence of methodical innovations. At its core is an ontology-driven standardisation pipeline applied to more than three million multi-centre inpatient records spanning multiple specialties. Coupled with a novel Special Temporal Grid representation, this pipeline resolves long-standing challenges in structuring longitudinal EMR data. To evaluate real-world clinical reasoning, we release 301MedQA, a benchmark that distils EMR-derived scenarios from four representative specialties. Our contributions are threefold: (1) End-to-end EMR processing: an integrative workflow that fuses ontology-based normalisation with fine-grained temporal modelling. (2) 301MedQA benchmark: the first public test bed for EMR-anchored clinical reasoning, mirroring authentic inpatient workflows. (3) AI4Doc-LLM: a large-language model in EMR understanding, clinical decision support, and medical documentation, while retaining alignment with expert-level reasoning. Together, these components establish a scalable path for deploying trustworthy language models in routine clinical practice. By integrating real-world clinical data with novel model and benchmark design, this study lays a foundation for advancing trustworthy medical LLMs from lab settings to practical deployment. Results Our framework utilizes structured EMR data spanning the complete hospitalization period, encompassing Progress Notes, Vital Signs, Diagnostic Records, Medical Orders, Lab Results, and Test Results, as shown in Fig. 1. This comprehensive medical record system is systematically organized into a temporal sequence, starting from admission records through regular progress notes to discharge summaries. The structured EMR data serves as the foundation for all four training stages: continue pre-training uses the complete EMR corpus for domain adaptation; instruction fine-tuning leverages the logical relationships between different medical records; reinforcement preference alignment incorporates expert-reviewed EMR cases; and retrieval integration utilizes indexed EMR knowledge bases. This standardized EMR structure ensures consistency and clinical validity throughout the training pipeline.Fig. 1Overview of EMR timeline and integration during hospitalization period.The upper panel shows the temporal organization of medical records from Day 0 to Day 7. The lower panel illustrates the relationship between different medical elements and the integrated hospitalization period components. Overview of EMR timeline and integration during hospitalization period. The upper panel shows the temporal organization of medical records from Day 0 to Day 7. The lower panel illustrates the relationship between different medical elements and the integrated hospitalization period components. Framework of AI4Docter Figure 2 provides a comprehensive overview of our AI4Doctor framework. In panel (a), we illustrate our ontology-based EMR processing pipeline, which handles diverse clinical records to ensure patient privacy while maintaining clinical validity. Panel (b) demonstrates our domain adaptation approach, where we leverage three primary knowledge sources: medical literature, textbooks, and standardized EMR data. This comprehensive knowledge integration supports effective medical-domain adaptation and curriculum-based instruction tuning, establishing a solid foundation for clinical understanding.Fig. 2Overview of the AI4Doctor framework.A EMR data processing pipeline with privacy protection mechanisms, incorporating diverse clinical records and hospital databases; (B) Domain adaptation and knowledge integration approach, combining medical literature, textbooks, and EMR data for comprehensive medical knowledge acquisition; (C) Temporal knowledge extraction system that constructs 3D medical instructions from clinical pathways and treatment trajectories; (D) Cross-specialty knowledge integration framework featuring a hybrid retrieve-and-inference system across multiple medical departments; (E) Multi-dimensional evaluation system based on senior clinical experts’ assessment of comprehensiveness, professionalism, and safety; (F) Benchmark evaluation framework using GPT-4 for systematic performance assessment on EMR-derived tasks. Overview of the AI4Doctor framework. A EMR data processing pipeline with privacy protection mechanisms, incorporating diverse clinical records and hospital databases; (B) Domain adaptation and knowledge integration approach, combining medical literature, textbooks, and EMR data for comprehensive medical knowledge acquisition; (C) Temporal knowledge extraction system that constructs 3D medical instructions from clinical pathways and treatment trajectories; (D) Cross-specialty knowledge integration framework featuring a hybrid retrieve-and-inference system across multiple medical departments; (E) Multi-dimensional evaluation system based on senior clinical experts’ assessment of comprehensiveness, professionalism, and safety; (F) Benchmark evaluation framework using GPT-4 for systematic performance assessment on EMR-derived tasks. The temporal knowledge extraction process is shown in panel (c), where we construct medical instructions from EMR data. This includes structured representations of diagnosis and treatment plans, admission records, postoperative courses, and discharge summaries, capturing the complete patient care trajectory. Panel (d) illustrates our hybrid retrieve-and-inference system, which encompasses major clinical domains, including internal medicine subspecialties, surgical specialities, critical care, and emergency medicine. The framework provides broad coverage across the continuum of hospital-based healthcare delivery. The evaluation framework is depicted in panel (e), showing our multi-faceted assessment approach through expert reviews across different medical departments. Senior clinical experts evaluate the model’s performance based on comprehensiveness, professionalism, and safety criteria. Finally, panel (f) presents the benchmark evaluation system, where we utilize GPT-4 based assessments on our EMR-derived benchmark datasets to validate the model’s clinical competence. This rigorous evaluation ensures that our model meets the high standards required for medical applications. Our experimental evaluation examines two complementary aspects of AI4Doctor’s performance. First, we analyze the capabilities of our foundation models trained at three parameter scales (7B, 13B, and 32B), focusing on their intrinsic medical knowledge and reasoning abilities acquired through our multi-stage training approach. The detailed results and analyses are presented in the following sections. Evaluation of AI4Docter We conduct comprehensive evaluations of our base models across multiple dimensions to assess their medical expertise and clinical reasoning capabilities. Complex clinical question A rigorous evaluation is conducted by senior physicians across multiple medical specialities. The evaluation employed a validated five-dimensional scoring framework: Overall clinical assessment, Adherence to evidence-based guidelines, Clinical relevance, Potential iatrogenic risk, and Therapeutic value. Each dimension was scored on a 10-point Likert scale and weighted according to clinical significance, calculation methodology detailed in the supplementary materials. For comparative analysis, we selected Qwen-32B, the highest-performing general Chinese language model with comparable parameter count, and HuatuoGPT 2-34B, the state-of-the-art domain-specific medical model, as the competitive baseline. The evaluation framework was designed to comprehensively assess medical language models against human expertise across multiple specialities. The evaluation was conducted at the Chinese PLA General Hospital, a tertiary academic medical centre renowned for its clinical excellence and translational research initiatives. The assessment incorporated practicing clinicians stratified by clinical experience: attending physicians (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$>$$\end{document}>10 years), fellows (5-10 years), and residents (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$<$$\end{document}<5 years), with two practitioners from each stratum. Additionally, nine board-certified specialists from multiple departments, including internal medicine subspecialties, surgical disciplines, and emergency medicine, developed 100 complex clinical vignettes and systematically evaluated responses from both models and participating clinicians. Table 1 presents the performance comparison between AI models and human physicians of varying experience levels on the medical QA task. AI4Doctor achieved the highest score of 80.52, demonstrating a clear advantage over other participants. The performance gap between the highest and lowest scores was approximately 19 points, with AI4Doctor establishing a 6.5-point lead over the second-ranked participant.Table 1Rankings based on comprehensive clinical evaluation criteria for complex medical question-answering tasksRankRespondentScore1AI4Doctor80.522Qwen-34B74.033HuatuoGPT-34B73.074Senior Physicians72.995Junior Physicians67.656Intermediate Physicians64.17Language models were evaluated with zero temperature to ensure deterministic outputs and response consistency across multiple request. Scores for human clinicians represent averaged performance metrics from two physicians at each experience level. Table 1Rankings based on comprehensive clinical evaluation criteria for complex medical question-answering tasksRankRespondentScore1AI4Doctor80.522Qwen-34B74.033HuatuoGPT-34B73.074Senior Physicians72.995Junior Physicians67.656Intermediate Physicians64.17Language models were evaluated with zero temperature to ensure deterministic outputs and response consistency across multiple request. Scores for human clinicians represent averaged performance metrics from two physicians at each experience level. Rankings based on comprehensive clinical evaluation criteria for complex medical question-answering tasks Language models were evaluated with zero temperature to ensure deterministic outputs and response consistency across multiple request. Scores for human clinicians represent averaged performance metrics from two physicians at each experience level. Results on EMR-based clinical question We developed a specialized assessment framework for the medical record generation task. This evaluation system encompasses five critical dimensions: Completeness, Specificity, Fluency, Clinical Utility, and Overall Impression, comprehensively evaluating the clinical value of model-generated medical documentation. For healthcare documentation assessment, we employed a standardized 10-point Likert scale with domain-specific evaluation criteria. The evaluation dataset was curated from a multi-specialty clinical repository of 10,000 records, from which clinical experts selected 40 representative cases. We utilized three large medical language models to generate records for each case and compared them against the reference documentation from the original clinical database. Detailed evaluation for case selection and dimensional weighting coefficients are provided in the Supplementary Materials. Experimental results are presented in Table 2, which presents the performance evaluation of four models across critical medical documentation tasks. AI4Doctor demonstrates consistent performance, with the highest performance in Discharge Summary (68.3) and an overall average of 67.73. Qwen-Chat shows comparable performance with an average score of 67.13, particularly excelling in First Admission Record. HuatuoGPT-II exhibits varied performance across tasks, achieving competitive scores in First Postoperative Record but showing notable weakness in Discharge Summary.Table 2Comparison of different models across medical tasksModelDPFARFPRDSAverage ScoreAI4docter (32B)67.668.266.868.367.73Qwen-Chat (32B)66.269.166.766.567.13HuatuoGPT-II (34B)60.265.567.243.259.03Medical record51.150.267.748.354.33DP diagnosis plan, FAR first admission record, FPR first postoperative record, DS discharge summary. Table 2Comparison of different models across medical tasksModelDPFARFPRDSAverage ScoreAI4docter (32B)67.668.266.868.367.73Qwen-Chat (32B)66.269.166.766.567.13HuatuoGPT-II (34B)60.265.567.243.259.03Medical record51.150.267.748.354.33DP diagnosis plan, FAR first admission record, FPR first postoperative record, DS discharge summary. Comparison of different models across medical tasks DP diagnosis plan, FAR first admission record, FPR first postoperative record, DS discharge summary. Benchmark performance Evaluation on 301MedQA This study utilizes a large-scale, multi-center clinical database encompassing electronic medical records from nine medical centers of the PLA General Hospital. The data spans 46 clinical departments, covering various fields of medicine, and thus possesses broad representativeness and diversity. The data has undergone thorough parsing and governance: content was segmented from medical records, and terms for diagnoses, symptoms, and other medical ontologies were standardized. This improved data interoperability and semantic consistency, facilitating cross-center and cross-disciplinary data integration and comparative analysis. To cover a more comprehensive range of results, we have categorized our benchmark, 301MedQA, into three types: Foundational Medical Queries, Advanced Clinical Queries, and Real-World Clinic Questions. We focused on representative clinical departments, covering common diseases and critical conditions, which allowed us to evaluate the method’s validity and robustness effectively, and detail of the benchmark construction are illustrated in Supplementary Materials B. The data statistics are shown in the Table 3. We use GPT-4 as the reference to review the quality of the model outputs. We prompt it to consider doctor-like language. Given the question and the corresponding two answers from two models, GPT-4 is asked first to compare the advantages of each output and analyze the answer to provide a score to each response. We take the average scores over all the questions and calculate the performance ratio for each compared model. The results of Advanced Clinical Queries (CW) are shown in Fig. 2(\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\boldsymbol{f}}$$\end{document}f). The results underscore AI4Doctor’s effectiveness in handling complex clinical queries, showcasing its robust medical reasoning capabilities and domain expertise. This consistent performance across multiple comparisons validates our comprehensive training approach and specialized medical-domain adaptation strategies.Table 3Benchmark statistics of 301MedQA, CW denotes Clinician-Written, and EX denotes LLM-expandedData DescribeNumberFoundational Medical Queries1200Advanced Clinical Queries (CW)176Advanced Clinical Queries (EX)601Real-World Clinic Questions300 Table 3Benchmark statistics of 301MedQA, CW denotes Clinician-Written, and EX denotes LLM-expandedData DescribeNumberFoundational Medical Queries1200Advanced Clinical Queries (CW)176Advanced Clinical Queries (EX)601Real-World Clinic Questions300 Benchmark statistics of 301MedQA, CW denotes Clinician-Written, and EX denotes LLM-expanded Evaluation on standard medical datasets and medical components in general benchmarks In order to measure the final performance of the model, the medical benchmarks in our assessment include C-Eval23, an extensive Chinese evaluation framework, and CMMLU24, designed to rigorously test the knowledge and reasoning skills of large-scale Chinese language models, whose results are shown in Table 4.Table 4Chinese public benchmark resultsModelCMMLU♢C_Eval♢HuatuoGPT33.2336.53Llama2-7B-Chat31.0226.40Llama2-13B-Chat32.6431.47ChatGLM2-6B43.8747.20ChatGLM3-6B46.9748.80Baichuan2-7B-Chat50.7451.47Baichuan2-13B-Chat52.9558.67Qwen-7B-Chat54.6552.80ChatGPT (API)50.3748.80HuatuoGPT-II (7B)59.0862.40HuatuoGPT-II (13B)61.4564.00AI4Docter (13B)57.3065.43Qwen1.5-Chat(32B)77.6069.30HuatuoGPT-II (34B)76.2067.21AI4Docter (32B)81.2070.39Evaluation was done using validation data for CEVAL and CMMLU. ♢ signifies extraction of only medical-related questions. Table 4Chinese public benchmark resultsModelCMMLU♢C_Eval♢HuatuoGPT33.2336.53Llama2-7B-Chat31.0226.40Llama2-13B-Chat32.6431.47ChatGLM2-6B43.8747.20ChatGLM3-6B46.9748.80Baichuan2-7B-Chat50.7451.47Baichuan2-13B-Chat52.9558.67Qwen-7B-Chat54.6552.80ChatGPT (API)50.3748.80HuatuoGPT-II (7B)59.0862.40HuatuoGPT-II (13B)61.4564.00AI4Docter (13B)57.3065.43Qwen1.5-Chat(32B)77.6069.30HuatuoGPT-II (34B)76.2067.21AI4Docter (32B)81.2070.39Evaluation was done using validation data for CEVAL and CMMLU. ♢ signifies extraction of only medical-related questions. Chinese public benchmark results Evaluation was done using validation data for CEVAL and CMMLU. ♢ signifies extraction of only medical-related questions. We specifically used the CMB-Exam and CMB-Clin25 for evaluation. This approach aims to measure the model’s adaptability and proficiency across varied medical knowledge frameworks and terminologies. The results are shown as Table 5. In addition to GPT-4, we are superior to other common domain models and all medical models.Table 5Zero-shot accuracy in the answer-only and CoT settings across different categories on CMB-Clin datasetsModelOpenPhysicianNursePharmacistTechnicianDisciplinesGraduate ExamAvgGeneralModelsChatGLM2-6B✓40.2048.5040.3438.6737.1933.3739.71+CoT40.2547.5636.0636.5835.5635.0638.51Baichuan-13B-chat✓34.8041.2535.4135.1731.8127.5634.33+CoT37.7044.7541.2234.6737.9432.9438.20ChatGPT✗40.7545.6936.5940.0837.9428.8138.31GPT-4✗59.9069.3152.1961.5059.6954.1959.46Medical ModelsBianQue-2✓4.904.194.283.583.313.253.92+CoT7.856.637.348.336.635.947.12Bentsao✓21.5519.9420.9422.7519.5616.8120.26+CoT21.0020.5620.6622.1719.2516.4420.01ChatGLM-Med✓21.7522.0621.8421.0018.4417.5020.43+CoT15.5516.2517.3416.3312.6312.5615.11ChatMed-Consult✓20.2022.3120.5922.6720.3817.4420.60+CoT19.4021.6920.0022.8318.8818.5620.23MedicalGPT✓26.4030.9424.7227.1725.4421.5026.03+CoT24.8027.1923.0924.5823.7521.0624.08HuatuoGPT✓29.1033.5627.4130.5829.4425.0629.19+CoT29.9034.0029.0630.9227.3825.6929.49DoctorGLM✓2.703.313.843.753.192.253.17+CoT3.153.133.412.503.382.252.97AI4Docter✓56.3163.1852.8152.2656.6455.1755.70 Table 5Zero-shot accuracy in the answer-only and CoT settings across different categories on CMB-Clin datasetsModelOpenPhysicianNursePharmacistTechnicianDisciplinesGraduate ExamAvgGeneralModelsChatGLM2-6B✓40.2048.5040.3438.6737.1933.3739.71+CoT40.2547.5636.0636.5835.5635.0638.51Baichuan-13B-chat✓34.8041.2535.4135.1731.8127.5634.33+CoT37.7044.7541.2234.6737.9432.9438.20ChatGPT✗40.7545.6936.5940.0837.9428.8138.31GPT-4✗59.9069.3152.1961.5059.6954.1959.46Medical ModelsBianQue-2✓4.904.194.283.583.313.253.92+CoT7.856.637.348.336.635.947.12Bentsao✓21.5519.9420.9422.7519.5616.8120.26+CoT21.0020.5620.6622.1719.2516.4420.01ChatGLM-Med✓21.7522.0621.8421.0018.4417.5020.43+CoT15.5516.2517.3416.3312.6312.5615.11ChatMed-Consult✓20.2022.3120.5922.6720.3817.4420.60+CoT19.4021.6920.0022.8318.8818.5620.23MedicalGPT✓26.4030.9424.7227.1725.4421.5026.03+CoT24.8027.1923.0924.5823.7521.0624.08HuatuoGPT✓29.1033.5627.4130.5829.4425.0629.19+CoT29.9034.0029.0630.9227.3825.6929.49DoctorGLM✓2.703.313.843.753.192.253.17+CoT3.153.133.412.503.382.252.97AI4Docter✓56.3163.1852.8152.2656.6455.1755.70 Zero-shot accuracy in the answer-only and CoT settings across different categories on CMB-Clin datasets CMB-Clin includes complex clinical diagnostic problems that evaluate the model’s ability to synthesize knowledge and reasoning. On the one hand, the knowledge aspect implies the need for the model to draw upon its medical knowledge when answering questions. On the other hand, the reasoning facet necessitates the model’s ability to analyze case reports, thus combining its medical knowledge to respond to inquiries. Following the evaluation setting with GPT-4 used in ref. 25, we evaluate the result of our model with other medical models; the result is shown in Table 6, AI4Docter outperforms all the medical models except and GPT-4.Table 6Results of automatic evaluation using GPT-4 on CMB-ClinModelFluencyRelevanceCompletenessProficiencyAvg.ChatGLM-Med3.551.971.612.372.38Bentsao3.882.051.712.582.55DoctorGLM4.742.001.652.302.67MedicalGPT4.482.642.192.893.05ChatMed-Consult4.883.082.673.303.48BianQue-24.863.523.023.603.75HuatuoGPT4.893.753.383.863.97ChatGLM2-6B4.863.763.514.004.03Qwen-Chat (32B)4.833.743.383.983.99HuatuoGPT (34B)4.803.763.414.014.00AI4Docter (13B)4.813.763.654.024.06AI4Docter (32B)4.924.263.924.324.36Avg. represents the average scores of each model across all aspects. Models are displayed in descending order of Avg. in the original table. Table 6Results of automatic evaluation using GPT-4 on CMB-ClinModelFluencyRelevanceCompletenessProficiencyAvg.ChatGLM-Med3.551.971.612.372.38Bentsao3.882.051.712.582.55DoctorGLM4.742.001.652.302.67MedicalGPT4.482.642.192.893.05ChatMed-Consult4.883.082.673.303.48BianQue-24.863.523.023.603.75HuatuoGPT4.893.753.383.863.97ChatGLM2-6B4.863.763.514.004.03Qwen-Chat (32B)4.833.743.383.983.99HuatuoGPT (34B)4.803.763.414.014.00AI4Docter (13B)4.813.763.654.024.06AI4Docter (32B)4.924.263.924.324.36Avg. represents the average scores of each model across all aspects. Models are displayed in descending order of Avg. in the original table. Results of automatic evaluation using GPT-4 on CMB-Clin Avg. represents the average scores of each model across all aspects. Models are displayed in descending order of Avg. in the original table. Impact of odds ratio preference optimization Through detailed ablation studies, we examine the effectiveness of Odds-Ratio Preference Optimization (ORPO) approach. A total of 176 questions in Advanced Clinical Queries (CW) from 301MedQA are collected to verify the result of preference alignment. The results, shown in Fig. 3, demonstrate consistent improvements across eight critical dimensions: comprehensiveness, risks and side effects, professionalism, ethicality, accuracy, comprehensibility, timeliness, and relevance. Experimental results illustrate that ORPO effectively aligns model outputs with professional medical standards while maintaining strong performance across all evaluation criteria. This comprehensive evaluation on the Advanced Clinical Queries component of 301MedQA demonstrates the significant impact of our preference optimization strategy on model performance.Fig. 3Evaluation Results with ORPO result of GPT-4 on Clinician-Written and LLM-expand Clinical Questions. Evaluation Results with ORPO result of GPT-4 on Clinician-Written and LLM-expand Clinical Questions. Retrieval-augmented clinical reasoning Intuitively, we hope that our model can also obtain diagnostic as output based on the patient information as input, through the complex effects of internal parameters, or refer to relevant information acquired from similar diagnostic records through retrieval. Specifically, we built high-quality QA from multiple data sources such as electronic medical records, medical guidelines, and clinical pathways as the external datastore. For Medical Consultation, given patient information as the query, our system retrieves similar medical records and corresponding diagnoses on the datastore according to the vectorized similarity, and prepends them to the query together as input to the language model to obtain more accurate output. Similar methods can be extent to Differential Diagnosis and Diagnosis and Treatement Protocol. We shows that (1) retrieval on our vector base can obtain relevant and consistent information to be further leveraged by the LM, and (2) our LLM takes advantage of retrieval augmentation and its effectiveness for real-world medical tasks. Retrieval accuracy Conclusively, we report the retrieval accuracy automatically and manually evaluated, to verify that retrieval on our vector base can obtain relevant and consistent information to be further leveraged by the model. For manual evaluation, we organize doctors to score the relevance between the query and the retrieved results form 4 (highly related) to 1 (not relevant). All retrieved results on the test set were scored by two clinicians, and those with large differences were additionally evaluated by a third, more professional expert. For automatic evaluation, we report the MRR@(1–5) as shown in Fig. 4.Fig. 4Retrieval & Augmentation Results.The retrieval (a–c) and retrieval augmentation (d–f) results under different settings, where w/sum means the retrieved documents are compressed by a summarization model, and w/ lab means the model output is further augmented by the retrieved laboratory results. Retrieval & Augmentation Results. The retrieval (a–c) and retrieval augmentation (d–f) results under different settings, where w/sum means the retrieved documents are compressed by a summarization model, and w/ lab means the model output is further augmented by the retrieved laboratory results. Augmentation Performance Through manual & automatic evaluation, we illustrate that our LLM takes advantage of retrieval augmentation and its effectiveness for real-world medical tasks such as Medical Consultation. We use 8 different metrics to evaluate the effectiveness of the augmentation. For a detailed introduction to each metric, please see Supplementary Materials Metrics. First, we organize doctors to perform manual evaluation to give model outputs a comprehensive score (Manual Eval, 1–5). In addition, we prompt GPT-4 to score the model outputs (GPT Score, 1–5) using the same rule as instruction, where the results showed high consistency with the manual evaluation. Furthermore, we use ROUGE-L and F1 Score to evaluate the consistency of the generation with the standard answers provided by clinicians. We use Chinese tokenized MAUVE to evaluate the fluency, and calculate 2-gram counts (Distinct Score) to evaluate the diversity of the generated content. For the comprehensive evaluation, shown in Fig. 5, of Medical Consultation on different retrieval tasks (e.g. EMR, LAB result, etc.), scores given by human and GPT-4 improved up to 10.4% and 6.0% respectively. Consistency improved significantly for both ROUGE ( + 3.30) and F1 Score ( + 5.12). For fine-grained metrics, EM increased 4.89 and Hit increased 4.00. Such results on these dimensions illustrate that our LLM do take advantage of retrieval augmentation and is effective for Medical Consultation. Meanwhile, fluency (MAUVE) and diversity (Distinct Score) achieved competitive result or improved slightly as a side effect.Fig. 5Performance on Medical Consultation under different settings.“Base” denotes our base model, “Re-Aug” denotes retrieval augmentation settings. “EMR” and “MG” denotes Electronic Medical Records and Medical Guides, respectively. Performance on Medical Consultation under different settings. “Base” denotes our base model, “Re-Aug” denotes retrieval augmentation settings. “EMR” and “MG” denotes Electronic Medical Records and Medical Guides, respectively. When compared to other popular LLMs on Medical Consultation with Electronic Medical Records and Medical Guides, our models perform the best (Fig. 5, Supplementary Materials Table 4), achieving improvement up to +8.71(32.55 v.s 41.26) for ROUGE-L, +10.92(47.27 v.s 58.19) for F1 Score, and +4.49(58.04 v.s 62.53)/ + 3.00(59.00 v.s. 62.00) for fin-grained metrics including EM/Hit, exceeding those of the other models for both the numerical value and the extent of improvement. Discussion Large language models (LLMs) have progressed from general-purpose knowledge engines to domain-tuned medical assistants2,4,13–16,26–29 yet questions remain about calibration30, guideline adherence31, long-context retrieval9,32–34 and workflow fidelity35. Mid-sized, EHR-centric encoders such as TransformEHR show that judicious scaling plus clinical corpora can rival larger baselines36, while multi-modal works37 and modular RAG systems38–52 illustrate the value of retrieval-aware design. Governance frameworks53 and deployment strategies54 further stress safety and cost-consciousness, while RL alignment alternatives support low-resource clinical use55. Against this backdrop, we present AI4Doctor, a 32B medical LLM combining longitudinal EMR pre-training, curriculum tuning, preference alignment, and hybrid retrieval—offering senior-clinician reasoning within a deployable compute envelope. AI4Doctor demonstrates that a mid-scale, 32-B parameter backbone—when paired with longitudinal EMR pre-training, curriculum-based instruction tuning, Odds-Ratio Preference Optimisation (ORPO) and hybrid retrieval—can achieve senior-clinician competence while remaining deployable on commodity hardware. This trajectory echoes recent calls to translate foundation-model promise into dependable bedside tools without resorting to billion-scale systems56. Continued pre-training on three million anonymised inpatient timelines proved decisive: CMMLU-Med rose by 8.4 absolute points and clinician-rated answer quality improved in infection-related topics. These gains corroborate evidence that domain-specific corpora sharpen probabilistic calibration more effectively than parameter inflation30, and align with the EHR-centred improvements reported in TransformEHR36. Scaling analysis reinforces a “right-sized” principle: our 13 B variant outperforms larger 34 B baselines on CMMLU-Med and C-Eval-Med, further supporting the notion that capacity exceeding data yields diminishing returns36. Expert evaluation across yielded an overall score of 80.52/100, a + 6.5 margin over the strongest general Chinese LLM of comparable size and +7.5 over senior physicians. This human-aligned assessment advances the broader evaluation agenda for clinical LLMs35 and highlights the value of temporal EMR conditioning. Hybrid retrieval offered complementary benefits. Injecting top-\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k$$\end{document}k EMR or guideline snippets raised ROUGE-L by +3.3, F1 by +5.1, and human ratings by +10.4%. These improvements resonate with hepatology-guideline experiments31 and long-context partitioning strategies9, confirming that targeted external memory mitigates context-window limits. ORPO alignment improved all eight clinical dimensions by 3–10% while consuming one-tenth the compute of full RLHF, dovetailing with reinforcement-learning guidance for resource-constrained healthcare settings55. Two limitations remain. Performance dips for rare conditions, and encounters exceeding the current context-window admit only a single retrieval chunk. Future work will explore knowledge-graph reranking and sliding-window decoding, drawing inspiration from multi-modal transformers that marry images and text37. Parallel efforts will embed GREAT-PLEA–style governance53 and balance on-prem versus cloud deployment trade-offs54 before large-scale roll-out. In conclusion, we present AI4Doc-LLM, an advanced medical language model, as a testament to our commitment to harnessing the power of AI for enhancing healthcare. Meticulously crafted through the integration of medical knowledge, expert insights, and clinical data, our model stands as a fusion innovation in the intersection of linguistics, engineering, medicine, and informatics. AI4Doc-LLM’s exceptional performance across various medical benchmarks demonstrates its unparalleled capability in clinical knowledge comprehension and problem-solving. The model’s adeptness at providing reliable clinical decision support, while mitigating issues like subjective hallucinations, exemplifies its practical utility in real-world medical scenarios. Beyond its current achievements, the AI4Doc-LLM’s design philosophy of continuous learning and adaptation mirrors the lifelong learning trajectory of medical professionals, ensuring its relevance and effectiveness in an ever-evolving healthcare landscape. Our research exemplifies a pioneering effort in the field of medical AI, offering a comprehensive solution that not only addresses current challenges but also paves the way for future advancements. The implications and prospects of AI4Doc-LLM extend beyond immediate clinical applications, envisioning a future where AI and human expertise coalesce to revolutionize healthcare delivery and patient care. This journey, grounded in rigorous research and collaborative efforts, heralds a new era in medical technology, promising enhanced healthcare outcomes and a profound impact on the field of medicine. AI4Doctor addresses the gap between exam‑style medical QA and episode‑grounded clinical decision‑making. Our contributions are threefold: (1) an end‑to‑end EMR pipeline with ontology standardization and the STG temporal representation; (2) a clinician‑curated evaluation protocol and 301MedQA benchmark emphasizing closed‑book patient‑episode reasoning; and (3) an alignment recipe that integrates clinician preferences with safety‑aware weighting. Together, these advances strengthen fidelity to clinical workflows and provide a transparent path toward safe, reproducible deployment. Method EMRs offer medical LLMs unparalleled real-world clinical insights—capturing patient journeys, cross-specialty reasoning, and treatment patterns in actual practice. Unlike textbooks, they encode how doctors truly diagnose and treat over time. This enables models to learn authentic medical decision-making, not just textbook knowledge. Medical adaption through temporal coherence EMR Our approach is structured around the seven key processes in EMRs, including consultation, laboratory assessment, diagnosis, treatment planning, medication management, inpatient care, and follow-up. By systematically organizing and filtering these processes, we extract temporally coherent and clinically relevant features from medical records. To ensure privacy, we employ a two-stage de-identification protocol that combines automated methods—medical NER, rule-based matching, and dictionary screening—with expert verification via stratified sampling. This strategy effectively removes protected health information while preserving essential clinical utility across multiple medical centers, time spans, departments, and record lengths. For EMR-specific corpus construction, hospitalization episodes are modelled as temporal graph sequences, and daily medical events are compared to standard clinical pathways. This captures both typical patterns and meaningful variations, while the knowledge structure retains core medical concepts and their clinical relationships, ensuring balanced representation across diseases, treatments, and departments. To further enhance model robustness and mitigate catastrophic forgetting, we augment the dataset with high-quality corpora from multiple sources, including real-world EMRs, Chinese medical textbooks, PubMed literature, and filtered general Chinese data. All data undergoes thorough preprocessing and quality assurance, as detailed in the Supplementary Materials D. Unlike fixed‑interval resampling or purely event‑sequence models, STG aligns multi‑modal EMR signals to multi‑resolution bins anchored to clinical indices (e.g., admission) and separates values, deltas/trends, time‑since‑last‑seen, and explicit missingness into distinct channels, preserving dynamics essential for reasoning. Leveraging EMR for advanced multi-task model optimization and instruction fine-tuning We utilize EMRs as a comprehensive resource for multi-task instruction tuning, optimizing the model’s capacity to answer domain-specific questions. Our pipeline includes: (1) a task cube organizing medical tasks by NLP abilities, specialties, and application domains; (2) task-aware data selection for diverse and high-quality EMR instances; and (3) collaborative medical question construction combining expert knowledge with data-driven filtering. This structured process transforms raw EMRs into high-quality instruction examples, covering diverse clinical scenarios and retaining natural interconnections in medical practice. To maintain general conversational and reasoning abilities, we supplement EMR-based instruction data with both real-world medical Q&A and general-domain dialogue data. Further implementation details appear in Supplementary Materials E. For preference modelling, we integrate implicit supervision from structured EMR data with explicit evaluation from medical professionals. EMRs provide rich clinical decision signals, which are enhanced by systematic expert assessment of model outputs across real-time performance, safety, and comprehensiveness. Candidate responses are generated using multi-model sampling—including both self-sampling and specialized open-source models—and high-scoring outputs are selected as preference data. Training employs distributed computing with ZeRO Stage 3 optimization and two epochs to ensure both computational efficiency and clinical robustness. The overall framework ensures that the model aligns with professional medical standards while retaining real-time performance and safety. Data statistics and further details are available in the Supplementary Materials E. To mitigate catastrophic forgetting and maintain general conversational and reasoning capabilities, we incorporated a mixture of real-world medical question-answering data and general-domain conversational data into our training regimen. The distribution statistics of these datasets are presented in Table 7.Table 7Static of Instruction DatasetsInstruction SourceInstance NumberEMR93,421Clinical Guidelines69,094Question Answer Pair10,162Seed Medical Question1,932Others236,715Total411,324 Table 7Static of Instruction DatasetsInstruction SourceInstance NumberEMR93,421Clinical Guidelines69,094Question Answer Pair10,162Seed Medical Question1,932Others236,715Total411,324 Static of Instruction Datasets Knowledge-enhanced medical dialogue generation via EMR-based retrieval The integration of historical EMR cases and expert knowledge bases significantly enhances the model’s ability to handle complex medical scenarios while reducing hallucination risks. Our approach combines parametric knowledge stored in model weights with non-parametric knowledge accessed through efficient retrieval mechanisms, enabling both memorization and dynamic reference of medical expertise. These details in EMR provide the foundation for constructing various medical tasks, such as diagnostic discussions, treatment plan recommendations, and admission diagnosis assessments. For ontology-enhanced EMR knowledge representation and retrieval, we construct dual representations combining dense vectors and structured medical concepts for each EMR document. The dense embedding is encoded by a medical-domain-pretrained encoder, while sparse ontological features are extracted from medical concepts and attributes. These ontological features derive from structured EMR fields, including chief complaints, medical history, examination results, medications, surgical records, and diagnostic discussions. The knowledge base integrates three primary information sources: EMR abstractions, medical guidelines, and expert dialogues. The approach incorporates three primary information sources: EMR abstractions, medical guidelines, and expert dialogues. EMR abstractions systematically capture essential clinical information through structured core elements. The medical guideline component maintains version-controlled clinical protocols, with standardized frameworks like NCCN Guidelines serving as authoritative references. Expert dialogues are structured as question-answer pairs enriched with medical context metadata, including specialty, difficulty level, and verification status. During inference, the model generates responses by incorporating retrieved knowledge from these integrated sources. For each retrieved document, we construct a context-enriched prompt template. This unified framework enables the model to leverage both structured medical knowledge and practical clinical experience, significantly improving response accuracy and reducing hallucination in complex medical scenarios. Ethics This study was reviewed and approved by the Ethics Review Board Approval No. S2025-302-01. The study used retrospective, de-identified electronic medical records, and the requirement for informed consent. Supplementary information Supplementary Materials Supplementary Materials