Diagnostic accuracy of large language models for emergency department triage: a systematic review and meta-analysis Authors: Limei Cui, Mengting Wang, Ye Xu, Wei Wei, Xuehong Zhao Published: 2026 ABSTRACT ObjectiveLarge language models (LLMs) have shown potential in the field of emergency department (ED) triage. However, a comprehensive evaluation of their diagnostic performance and a comparison with triage nurses have not been extensively explored. To systematically evaluate the comparative triage accuracy of LLMs versus triage nurses in ED, assess their performance in identifying patients assigned to the highest-acuity triage category by the reference standard.MethodsA systematic search was conducted in CNKI, Wanfang, VIP, CBM, PubMed, Embase, Web of Science, and Cochrane Library databases from inception to March 2025. Two researchers independently screened and evaluated the quality of the selected studies and conducted meta-analyses using RevMan 5.4 and Stata 16.0.ResultsA total of 11 studies were included, involving 3088 real or simulated ED cases. The pooled odds ratio (OR) for LLM triage accuracy was 0.65 (95% CI: 0.32–1.34; I² = 97%). For identifying patients assigned to the highest-acuity triage category by the reference standard, LLMs demonstrated a pooled sensitivity of 61% (95% CI: 48%-73%) and specificity of 97% (95% CI: 88%-99%), with statistically significant differences (p < 0.05). The pooled area under the curve (AUC) was 0.82 (95% CI: 0.78–0.85). These estimates reflect concordance with triage category by the reference standard rather than discrimination of outcome-anchored physiological acuity. No significant publication bias was detected by Deeks’ funnel plot asymmetry test.ConclusionCurrent LLMs demonstrate moderate discriminatory performance for identifying patients assigned to the highest-acuity triage category by the reference standard, but have limited sensitivity for these cases. LLMs may have potential as adjunctive support tools, but further prospective evaluation with patient-centered and safety-critical outcomes is required before implementation can be considered.Supplementary InformationThe online version contains supplementary material available at 10.1186/s12873-026-01639-z. Introduction Patients presenting to the emergency department (ED) are typically characterized by acute-onset illness, a wide variety of conditions, and varying degrees of illness severity [1]. As a crucial clinical unit within hospitals, the ED plays a key role in early patient stabilization and patient safety. Triage, the initial step in the ED workflow, enables rapid assessment and prioritization of patients, thereby facilitating timely treatment for patients assigned to the highest-acuity triage category. At present, triage is primarily performed by nurses. The accuracy of triage is directly influenced by subjective factors, such as the nurse’s knowledge base, clinical experience, and individual judgment [2, 3]. Meanwhile, the increasing volume of ED visits and the growing prevalence of ED crowding pose significant challenges for triage nurses [4, 5]. Evidence suggests that increased patient volume substantially elevates workload and occupational stress among triage staff, thereby compromising triage quality [6]. This may result in higher rates of triage delays and missed assessments, as well as reduced triage accuracy, ultimately affecting patient outcomes, medical safety, and the efficient utilization of emergency care resources [2, 7–9]. Therefore, achieving accurate and rapid triage within the constraints of limited medical resources and critical time windows has become a vital component in enhancing ED service quality and ensuring patient safety. In recent years, artificial intelligence (AI) technologies, particularly large language models (LLMs), have rapidly transformed the healthcare sector through advances in machine learning and natural language processing [10, 11]. Trained on massive text corpora, LLMs possess capabilities for multilingual understanding and generation, demonstrating significant potential for supporting clinical decision-making and other healthcare applications [12–14]. Some studies have reported that ChatGPT-4, developed by OpenAI, demonstrates favorable predictive performance in ED triage and may serve as an effective adjunctive tool [15]. However, other studies argue that LLMs exhibit relatively high rates of under-triage and are not yet suitable as standalone replacements for conventional emergency triage systems [16]. To date, multiple studies have examined the accuracy of various LLMs in ED triage [15–20], yet the findings remain inconsistent. Furthermore, evidence specifically evaluating LLMs triage performance in identifying patients assigned to the highest-acuity triage category is relatively limited. Therefore, this meta-analysis aims to quantitatively evaluate the comparative triage classification performance of LLMs in ED triage and to compare their triage performance with that of nurses as reported in the included studies. Importantly, this comparison does not assume nurse triage to represent a definitive gold standard; rather, it reflects current clinical practice and serves as a pragmatic benchmark against which LLM performance can be contextualized. In addition, this study assesses the ability of LLMs to identify patients assigned to the highest-acuity triage category according to the reference standard. It is worth noting that the triage category may not necessarily represent outcome-anchored physiological severity. By synthesizing the currently available evidence, this meta-analysis provides a structured evaluation of the performance, potential, and limitations of LLMs in ED triage. The findings are intended to inform the development of AI-assisted decision-support systems and to support cautious, evidence-based consideration of their role as adjunct tools in emergency triage, rather than as standalone clinical solutions. Materials and methods Protocol registration This systematic review and meta-analysis included 11 observational studies that directly evaluated the comparative triage classification performance of LLMs in ED triage and their performance in triaging patients assigned to the highest-acuity triage category. The review was conducted and reported in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 statement and the PRISMA extension for Diagnostic Test Accuracy (PRISMA-DTA) [21, 22]. The protocol has been registered with PROSPERO under registration number CRD420251001624. Data sources and search strategy We searched electronic databases including CNKI, Wanfang, VIP, CBM, PubMed, Embase, Web of Science, and the Cochrane Library from inception to March 2025. In addition, reference lists of relevant studies and reviews were manually screened to identify further eligible publications. The search strategy combined Medical Subject Headings (MeSH) terms and free-text keywords using Boolean operators. The search strategy is provided in Supplementary Material 1. Eligibility criteria Eligibility criteria were defined with reference to a diagnostic test accuracy framework. Population (P): studies involving real or simulated ED cases. Index test (I): triage performed by a large language model (LLM), including but not limited to ChatGPT, Gemini, DeepSeek, or other comparable models. Reference standard (R): because a universally accepted gold standard for the “correct” ED triage category is difficult to establish, this review used the study-defined reference triage classification reported in the original studies, such as consensus-based judgments by nurses or clinicians. Diagnosis outcomes (D): (1) overall triage accuracy of the LLM compared with the reference standard; and/or (2) diagnostic performance for identifying patients assigned to the highest-acuity triage category, based on data sufficient to construct a 2 × 2 contingency table (true positives, false positives, true negatives, and false negatives). Inclusion criteria Studies were included if they met all of the following criteria: (1) involved real or simulated ED cases; (2) used an observational, retrospective, cross-sectional, or simulation-based comparative study design; (3) evaluated triage performed by an LLM; (4) reported overall triage accuracy, or provided sufficient data to derive diagnostic accuracy measures for identifying patients assigned to the highest-acuity triage category; (5) were published in English or Chinese. Exclusion criteria Studies were excluded if they were: (1) Conference abstracts or studies for which the full text was unavailable; (2) Duplicate publications; (3) Studies with incomplete data that could not be extracted or transformed for analysis; or (4) Studies judged to have critical methodological limitations that precluded meaningful synthesis. Screening and quality assessment All records retrieved from the databases were imported into NoteExpress for deduplication. Two graduate students trained in evidence-based nursing methodology independently screened the retrieved records and assessed study eligibility based on the predefined inclusion and exclusion criteria. Any disagreements were resolved through discussion, and when consensus could not be reached, a third reviewer adjudicated the final decision. The methodological quality of the included studies was evaluated using the Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2) tool [23], which assesses both risk of bias and applicability concerns. The QUADAS-2 tool assesses the risk of bias across four domains: patient selection, index test, reference standard, and flow and timing. It also evaluates concerns regarding applicability for the first three domains. The QUADAS-2 was considered a pragmatic tool for structured appraisal of diagnostic accuracy-related bias. However, QUADAS-2 does not fully capture AI-specific sources of bias, such as possible training-data leakage, prompt instability, model version drift, and limited generalizability across settings. A descriptive AI-specific methodological audit was provided in Supplementary Table 1 to complement QUADAS-2 by summarizing AI-related reporting and methodological concerns across the included studies. In addition, reference-standard domain of QUADAS-2 mainly evaluates whether the reference standard was applied appropriately and interpreted independently, but it does not fully assess whether the reference standard itself is an unbiased measure of true clinical severity. This distinction is particularly important in ED triage, because physician, expert-panel, or prespecified simulated-case triage labels may contain under-triage, over-triage, or disposition-error label noise when they are not anchored to patient-centered outcomes. Data extraction Two master’s-level reviewers independently extracted data according to a prespecified data extraction plan. Discrepancies and issues requiring data transformation were resolved through discussion with a third reviewer. Data were collected using a standardized Excel form, including first author, publication year, country, study design, triage system, type of clinical cases being classified, evaluation setting and the number of scenarios or patients assessed, the type and version of the LLM used, control group, reported outcomes, evaluation mode, real-time workflow, translational interpretation, reference standard category and patient-centered outcomes reported. As triage categories do not fully correspond to the severity of clinical outcomes, and in order to address potential reference-standard bias, we also extracted the type of reference standard used in each included study and whether patient-centred outcome measures were reported, such as admission to Intensive Care Unit (ICU) within a specified time window, in-hospital mortality, readmission within 72 h, or receipt of time-critical interventions. Statistical analysis Meta-analyses were performed using Review Manager (RevMan) version 5.4 and Stata version 16.0. All statistical tests were two-sided, and p < 0.05 was considered statistically significant. For the outcome of overall triage accuracy, studies reporting the numbers of correctly and incorrectly triaged cases in the LLM group and comparator group were synthesized using odds ratio (OR) with 95% confidence intervals (CI). In this analysis, the effect estimate reflected the relative odds of correct triage classification by the LLM compared with the control reported in the original study. In studies that evaluated more than one LLM or LLM version, each model-specific comparison was extracted and entered as a separate effect size in the pooled OR analysis. Statistical heterogeneity was assessed using the chi-square test and the I² statistic. A random-effects model was used when substantial heterogeneity was present; otherwise, a fixed-effects model was applied. For the outcome of identification of patients assigned to the highest-acuity triage category, diagnostic accuracy meta-analysis was conducted when studies provided sufficient data to construct 2 × 2 contingency tables. For this analysis, patients classified as the highest-acuity triage level by the reference standard in the original study were treated as the target-positive group. True positives, false positives, true negatives, and false negatives were extracted or derived accordingly. A bivariate mixed-effects model was fitted in Stata using the midas command to obtain pooled sensitivity and specificity with 95% CI. A summary receiver operating characteristic (SROC) curve was constructed, and the area under the curve (AUC) was calculated to summarize overall discriminatory performance. These two analytic approaches addressed related but distinct questions. The pooled OR for overall triage accuracy reflected agreement in correct classification at the overall study level, whereas pooled sensitivity and specificity reflected discrimination for a prespecified target condition, namely assignment to the highest-acuity triage category. For the diagnostic accuracy meta-analysis, potential publication bias was assessed using Deeks’ funnel plot asymmetry test in Stata, and p < 0.10 was considered suggestive of significant asymmetry. A leave-one-out sensitivity analysis was conducted for the pooled analysis of overall triage accuracy to evaluate the influence of individual studies on the pooled effect estimate and between-study heterogeneity. Results Search results and study selection A total of 979 records were identified through the database search. After removal of duplicates, 603 records remained for screening. Following title and abstract screening and full-text assessment, 11 studies were included in the final review [15–20, 24–28]. The study selection process is presented in Fig. 1. Fig. 1PRISMA flowchart of screening and study selection PRISMA flowchart of screening and study selection Study characteristics and quality assessment The 11 included studies [15–20, 24–28] evaluated a total of 3,088 real or simulated ED cases, of which 2,701 cases were included in the meta-analysis. Studies were conducted between 2023 and 2025 and predominantly used observational designs, focusing on a broad range of ED clinical scenarios. All included studies applied ChatGPT for ED triage; additional LLMs evaluated across studies included Copilot, Gemini, Pi, and Google-based models. Detailed study characteristics are summarized in Table 1. In addition, a reference-standard and outcome-anchoring audit was conducted for all included studies, as presented in Table 2. Across the included studies, the reference standards were primarily based on emergency physician triage, expert panel triage or prespecified correct triage level in simulated cases. One study assessed the predictive ability of LLMs for patient-centred outcomes [17]. As there was only one such study and the corresponding raw data could not be extracted, no subgroup meta-analysis contrasting outcome-anchored ground truth with non-outcome-anchored triage labels could be performed. To further address the translational context of included studies, a structured assessment of translational characteristics was conducted for all included studies, as presented in Table 2. The included studies were retrospective or in-silico evaluations without integration into live emergency department workflows, and none represented prospective real-time clinical deployment. Table 1Characteristics of included studiesFirst authorYearCountryStudy designTriage systemType of casesNu-mberType and version of LLMDiagnostic accuracy of control groupDiagnostic accuracy of LLMArslan, B [18]2024TurkeyObservational StudyEmergency Severity Index (ESI)Real ED cases468ChatGPT-4.0Copilot Pro65.20%ChatGPT-4.0: 66.50%Copilot Pro: 61.80%Tortum, F [16]2024TurkeyObservational StudyThree-level triage systemReal ED cases(aged 18 and above)500Google Gemini 1.0ChatGPT-3.5Pi91.60%Gemini: 52.20%ChatGPT-3.5: 51.60%Pi: 49.20%Zaboli, A [17]2024ItalyObservational StudyManchester Triage System (MTS)Real ED cases(aged 18 and above)30ChatGPT-3.563.33%ChatGPT-3.5: 50.00%Paslı,S [15]2024TurkeyObservational StudyEmergency Severity Index (ESI)Real ED cases758ChatGPT-4.094.06%ChatGPT-4.0: 95.25%Yazaki, M [19]2025JapanObservational StudyEmergency Severity Index Ver.3 Field TriageSimulated ED cases100ChatGPT-3.5ChatGPT-4.035.00%ChatGPT-3.5:47.00%ChatGPT-4.0:48.00%ChatGPT-3.5(RAG-enhanced):70.00%ChatGPT-4.0(RAG-enhanced):55.00%Eraybar, S [20]2024TurkeyObservational StudyThree-level triage systemSimulated ED cases50ChatGPT-3.5Google Bard63.00%ChatGPT-3.5: 48.00%Google Bard: 54.00%Colakca, C [24]2024TurkeyObservational StudyEmergency Severity Index (ESI)Real ED cases(aged 18 and above)745ChatGPT-4.0--ChatGPT-4.0: 76.60%Masanneck, L[25]2024GermanyObservational StudyManchester Triage System (MTS)Real ED cases145ChatGPT-3.5ChatGPT-4.0Llama 3 70BGemini 1.5Mixtral 8 × 7b----Kim, JH [26]2024KoreaObservational StudyKorean Triage and Acuity ScaleSimulated ED cases202ChatGPT-3.5ChatGPT-4.0----Fraser, H [27]2023USAObservational StudyThree-level triage systemReal ED cases(aged 18 and above)40ChatGPT-3.5ChatGPT-4.0--ChatGPT-4.0: 76.00%ChatGPT-3.5: 59.00%Sarbay,İ [28]2023TurkeyObservational StudyEmergency Severity Index (ESI)Simulated ED cases50ChatGPT-4.0--ChatGPT-4.0: 58.00% Table 1Characteristics of included studiesFirst authorYearCountryStudy designTriage systemType of casesNu-mberType and version of LLMDiagnostic accuracy of control groupDiagnostic accuracy of LLMArslan, B [18]2024TurkeyObservational StudyEmergency Severity Index (ESI)Real ED cases468ChatGPT-4.0Copilot Pro65.20%ChatGPT-4.0: 66.50%Copilot Pro: 61.80%Tortum, F [16]2024TurkeyObservational StudyThree-level triage systemReal ED cases(aged 18 and above)500Google Gemini 1.0ChatGPT-3.5Pi91.60%Gemini: 52.20%ChatGPT-3.5: 51.60%Pi: 49.20%Zaboli, A [17]2024ItalyObservational StudyManchester Triage System (MTS)Real ED cases(aged 18 and above)30ChatGPT-3.563.33%ChatGPT-3.5: 50.00%Paslı,S [15]2024TurkeyObservational StudyEmergency Severity Index (ESI)Real ED cases758ChatGPT-4.094.06%ChatGPT-4.0: 95.25%Yazaki, M [19]2025JapanObservational StudyEmergency Severity Index Ver.3 Field TriageSimulated ED cases100ChatGPT-3.5ChatGPT-4.035.00%ChatGPT-3.5:47.00%ChatGPT-4.0:48.00%ChatGPT-3.5(RAG-enhanced):70.00%ChatGPT-4.0(RAG-enhanced):55.00%Eraybar, S [20]2024TurkeyObservational StudyThree-level triage systemSimulated ED cases50ChatGPT-3.5Google Bard63.00%ChatGPT-3.5: 48.00%Google Bard: 54.00%Colakca, C [24]2024TurkeyObservational StudyEmergency Severity Index (ESI)Real ED cases(aged 18 and above)745ChatGPT-4.0--ChatGPT-4.0: 76.60%Masanneck, L[25]2024GermanyObservational StudyManchester Triage System (MTS)Real ED cases145ChatGPT-3.5ChatGPT-4.0Llama 3 70BGemini 1.5Mixtral 8 × 7b----Kim, JH [26]2024KoreaObservational StudyKorean Triage and Acuity ScaleSimulated ED cases202ChatGPT-3.5ChatGPT-4.0----Fraser, H [27]2023USAObservational StudyThree-level triage systemReal ED cases(aged 18 and above)40ChatGPT-3.5ChatGPT-4.0--ChatGPT-4.0: 76.00%ChatGPT-3.5: 59.00%Sarbay,İ [28]2023TurkeyObservational StudyEmergency Severity Index (ESI)Simulated ED cases50ChatGPT-4.0--ChatGPT-4.0: 58.00% Characteristics of included studies ChatGPT-4.0 Copilot Pro ChatGPT-4.0: 66.50% Copilot Pro: 61.80% Real ED cases (aged 18 and above) Google Gemini 1.0 ChatGPT-3.5 Pi Gemini: 52.20% ChatGPT-3.5: 51.60% Pi: 49.20% Real ED cases (aged 18 and above) ChatGPT-3.5 ChatGPT-4.0 ChatGPT-3.5:47.00% ChatGPT-4.0:48.00% ChatGPT-3.5(RAG-enhanced):70.00% ChatGPT-4.0(RAG-enhanced):55.00% ChatGPT-3.5 Google Bard ChatGPT-3.5: 48.00% Google Bard: 54.00% Real ED cases (aged 18 and above) Masanneck, L [25] ChatGPT-3.5 ChatGPT-4.0 Llama 3 70B Gemini 1.5 Mixtral 8 × 7b ChatGPT-3.5 ChatGPT-4.0 Real ED cases (aged 18 and above) ChatGPT-3.5 ChatGPT-4.0 ChatGPT-4.0: 76.00% ChatGPT-3.5: 59.00% Table 2Translational interpretation, reference standards and outcome anchoring of the included studiesFirst authorYearType of casesEvaluation modeReal-time workflowTranslational interpretationReference standard categoryPatient-centered outcomes reportedExtractable 2 × 2 data for outcome-anchored analysisArslan, B [18]2024Real ED casesRetrospective clinical-data evaluationNoPreclinicalTriage level only assigned by an emergency physicianNoNoTortum, F [16]2024Real ED cases(aged 18 and above)Retrospective clinical-data evaluationNoPreclinicalConsensus triage level determined by expert panelNoNoZaboli, A [17]2024Real ED cases(aged 18 and above)Retrospective clinical-data evaluationNoPreclinicalConsensus triage level determined by expert panelYesNoPaslı,S [15]2024Real ED casesRetrospective clinical-data evaluationNoPreclinicalTriage level only assigned by an emergency physicianNoNoYazaki, M [19]2025Simulated ED casesIn-silico evaluationNoPreclinicalPrespecified correct triage level in simulated casesNoNoEraybar, S [20]2024Simulated ED casesIn-silico evaluationNoPreclinicalConsensus triage level determined by expert panelNoNoColakca, C [24]2024Real ED cases(aged 18 and above)Retrospective clinical-data evaluationNoPreclinicalConsensus triage level determined by expert panelNoNoMasanneck, L[25]2024Real ED casesRetrospective clinical-data evaluationNoPreclinicalConsensus triage level determined by expert panelNoNoKim, JH [26]2024Simulated ED casesIn-silico evaluationNoPreclinicalTriage level only assigned by an emergency physicianNoNoFraser, H [27]2023Real ED cases(aged 18 and above)Retrospective clinical-data evaluationNoPreclinicalConsensus triage level determined by expert panelNoNoSarbay,İ [28]2023Simulated ED casesIn-silico evaluationNoPreclinicalConsensus triage level determined by expert panelNoNo Table 2Translational interpretation, reference standards and outcome anchoring of the included studiesFirst authorYearType of casesEvaluation modeReal-time workflowTranslational interpretationReference standard categoryPatient-centered outcomes reportedExtractable 2 × 2 data for outcome-anchored analysisArslan, B [18]2024Real ED casesRetrospective clinical-data evaluationNoPreclinicalTriage level only assigned by an emergency physicianNoNoTortum, F [16]2024Real ED cases(aged 18 and above)Retrospective clinical-data evaluationNoPreclinicalConsensus triage level determined by expert panelNoNoZaboli, A [17]2024Real ED cases(aged 18 and above)Retrospective clinical-data evaluationNoPreclinicalConsensus triage level determined by expert panelYesNoPaslı,S [15]2024Real ED casesRetrospective clinical-data evaluationNoPreclinicalTriage level only assigned by an emergency physicianNoNoYazaki, M [19]2025Simulated ED casesIn-silico evaluationNoPreclinicalPrespecified correct triage level in simulated casesNoNoEraybar, S [20]2024Simulated ED casesIn-silico evaluationNoPreclinicalConsensus triage level determined by expert panelNoNoColakca, C [24]2024Real ED cases(aged 18 and above)Retrospective clinical-data evaluationNoPreclinicalConsensus triage level determined by expert panelNoNoMasanneck, L[25]2024Real ED casesRetrospective clinical-data evaluationNoPreclinicalConsensus triage level determined by expert panelNoNoKim, JH [26]2024Simulated ED casesIn-silico evaluationNoPreclinicalTriage level only assigned by an emergency physicianNoNoFraser, H [27]2023Real ED cases(aged 18 and above)Retrospective clinical-data evaluationNoPreclinicalConsensus triage level determined by expert panelNoNoSarbay,İ [28]2023Simulated ED casesIn-silico evaluationNoPreclinicalConsensus triage level determined by expert panelNoNo Translational interpretation, reference standards and outcome anchoring of the included studies Real ED cases (aged 18 and above) Real ED cases (aged 18 and above) Real ED cases (aged 18 and above) Masanneck, L [25] Real ED cases (aged 18 and above) The QUADAS-2 assessment results shown in Figs. 2 and 3 should be interpreted in light of the reference-standard and outcome-anchoring audit in Table 2. A low risk rating in the QUADAS-2 reference-standard domain indicates that the reference standard was considered appropriately applied within the methodological framework of QUADAS-2, which should not be interpreted as evidence that the reference standard was an unbiased measure of true clinical severity. As shown in Table 2, most included studies used non-outcome-anchored triage labels assigned by emergency physicians, expert panels, or prespecified simulated-case answers. Therefore, these reference standards may still embed label noise related to under-triage, over-triage, local triage practice, and institutional decision-making. Fig. 2Risk of bias summary graph for the included studies. Note: Q1, patient selection; Q2, index test; Q3, reference standard; Q4, flow and timing; Q5, applicability concerns for patient selection; Q6, applicability concerns for the index test; Q7, applicability concerns for the reference standard Risk of bias summary graph for the included studies. Note: Q1, patient selection; Q2, index test; Q3, reference standard; Q4, flow and timing; Q5, applicability concerns for patient selection; Q6, applicability concerns for the index test; Q7, applicability concerns for the reference standard Fig. 3Summary of the risk of bias assessment for the included studies. Note: Q1, patient selection; Q2, index test; Q3, reference standard; Q4, flow and timing; Q5, applicability concerns for patient selection; Q6, applicability concerns for the index test; Q7, applicability concerns for the reference standard Summary of the risk of bias assessment for the included studies. Note: Q1, patient selection; Q2, index test; Q3, reference standard; Q4, flow and timing; Q5, applicability concerns for patient selection; Q6, applicability concerns for the index test; Q7, applicability concerns for the reference standard Outcomes Triage accuracy of large language models Across eight studies evaluating five different LLMs [15–20, 24, 28], a pooled analysis of triage accuracy showed substantial between-study heterogeneity (I² = 97%, p < 0.05), warranting a random-effects model. Overall, when LLM-based triage was compared with nurse triage for real or simulated ED cases, the pooled odds ratio (OR) for triage accuracy was 0.65 (95% CI: 0.32–1.34), as shown in Fig. 4. Given that two studies [15, 19] used an enhanced version of ChatGPT for triage, a separate meta-analysis was conducted for these studies. The pooled OR was 2.08 (95% CI: 0.87–4.98). However, this result did not reach statistical significance, as illustrated in Fig. 5. These findings suggest that, compared with nurse triage as performed in current clinical practice, LLM-based triage did not demonstrate statistically significant superiority in overall triage accuracy. Nevertheless, retrieval-augmented generation (RAG) suggests a positive trend and significant potential for LLMs in ED triage applications. Notably, the pooled OR should be interpreted cautiously. In the context of ED triage, a universally valid gold standard for the “correct” triage category is difficult to define. Accordingly, this comparison reflects the relative triage performance of LLMs versus nurse triage as observed in current practice, rather than accuracy against a definitive gold standard of true clinical severity. Fig. 4Forest plot of odds ratio (OR) for the LLMs. Note: Arslan, B① denotes data related to Copilot; Arslan, B② denotes data related to ChatGPT; Eraybar, S① denotes data related to Google Bard; Eraybar, S② denotes data related to ChatGPT; Tortum, F① denotes data related to Pi; Tortum, F② denotes data related to Google Gemini; Tortum, F③ denotes data related to ChatGPT; Yazaki, M① denotes data related to ChatGPT-3.5 (RAG-enhanced); Yazaki, M② denotes data related to ChatGPT-4 (RAG-enhanced); Yazaki, M③ denotes data related to ChatGPT-3.5; Yazaki, M④ denotes data related to ChatGPT-4 Forest plot of odds ratio (OR) for the LLMs. Note: Arslan, B① denotes data related to Copilot; Arslan, B② denotes data related to ChatGPT; Eraybar, S① denotes data related to Google Bard; Eraybar, S② denotes data related to ChatGPT; Tortum, F① denotes data related to Pi; Tortum, F② denotes data related to Google Gemini; Tortum, F③ denotes data related to ChatGPT; Yazaki, M① denotes data related to ChatGPT-3.5 (RAG-enhanced); Yazaki, M② denotes data related to ChatGPT-4 (RAG-enhanced); Yazaki, M③ denotes data related to ChatGPT-3.5; Yazaki, M④ denotes data related to ChatGPT-4 Fig. 5Forest plot of odds ratio (OR) for the RAG-enhanced LLMs Forest plot of odds ratio (OR) for the RAG-enhanced LLMs Subgroup analysis based on case type and translational context To explore whether the translational context of evaluation influenced the comparative triage performance of LLMs, a subgroup analysis was conducted according to case type, distinguishing studies using real ED cases from those using simulated ED cases. In this review, case type was considered an important observable indicator of translational context: studies using simulated ED cases generally represented in-silico evaluations, whereas studies using real ED cases represented retrospective evaluations based on existing clinical data. Among studies based on real ED cases, LLM-based triage showed lower odds of correct triage classification compared with nurse triage, with a pooled OR of 0.32 (95% CI: 0.13–0.82; I² = 98%; p = 0.02). This finding suggests that, in studies using real-world ED data, LLMs did not demonstrate comparable overall triage accuracy to nurse triage. In contrast, among studies based on simulated ED cases, the pooled OR was 1.56 (95% CI: 0.90–2.71; I² = 78%; p = 0.11). These subgroup findings suggest that LLMs triage performance may vary across different translational evaluation contexts, with more favorable performance observed in simulated scenarios than in retrospective clinical-data. However, substantial heterogeneity remained within both subgroups, and these results should therefore be interpreted cautiously. Triage performance for patients assigned to the highest-acuity triage category For the diagnostic accuracy analysis, the target-positive group was defined as patients assigned to the highest-acuity triage category according to the reference standard in the included studies. Accurate identification of such patients is an important task in ED triage. Therefore, this study focused on patients classified into the highest-acuity triage category in seven included studies [15–18, 20, 24, 28]. Data on true positives, false positives, true negatives, and false negatives were extracted from each study to populate 2 × 2 contingency tables. These data were then pooled to calculate sensitivity, specificity, and the SROC curve, in order to evaluate the performance of LLMs in identifying these patients. The pooled sensitivity across different LLMs was 61% (95% CI: 48%–73%), and the pooled specificity was 97% (95% CI: 88%–99%). The differences were statistically significant (p < 0.05). The SROC curve yielded an AUC of 0.82 (95% CI: 0.78–0.85), as presented in Figs. 6 and 7. Fig. 6Forest plot for the triage performance of LLMs in patients assigned to the highest-acuity triage category by the reference standard Forest plot for the triage performance of LLMs in patients assigned to the highest-acuity triage category by the reference standard Fig. 7SROC curve for the triage performance of LLMs SROC curve for the triage performance of LLMs Publication bias The Deeks’ funnel plot asymmetry test showed no statistically significant evidence of funnel plot asymmetry (p = 0.13), as shown in Fig. 8, suggesting no clear evidence of publication bias in the included studies. Nevertheless, because the number of included studies was limited, this finding should be interpreted cautiously. Fig. 8Deeks’ funnel plot asymmetry test for publication bias Deeks’ funnel plot asymmetry test for publication bias Sensitivity analysis A leave-one-out sensitivity analysis was conducted for the pooled analysis of overall triage accuracy, as shown in Fig. 9. After omitting each model-specific comparison in turn, the pooled OR showed only moderate fluctuation, indicating that the overall result was not driven by any single comparison. The omission-specific pooled estimates were lowest when the comparison from Paslı, S. et al. [15] was removed, whereas omission of the Tortum, F. et al. [16] comparisons yielded relatively higher pooled estimates. Overall, although several comparisons had a noticeable influence on the magnitude of the pooled effect, the direction and interpretation of the findings did not materially change, supporting the robustness of the main result. Fig. 9Leave-one-out analysis of the pooled OR for overall triage accuracy Leave-one-out analysis of the pooled OR for overall triage accuracy Discussion Main findings This systematic review and meta-analysis synthesized the currently available evidence on the performance of LLMs in ED triage. Overall, LLM-based triage did not demonstrate superior accuracy compared with triage nurses. In the pooled analysis of overall triage accuracy, the summary effect estimate did not show a statistically significant advantage for LLMs, and substantial between-study heterogeneity was observed. These findings suggest that, based on the currently available retrospective and simulation-based evidence, LLMs are not yet ready to replace human triage in ED settings. At the same time, some potentially encouraging signals were observed. In particular, LLMs supported by RAG showed a higher point estimate for triage accuracy, although this result did not reach statistical significance. Taken together, the available evidence suggests that current LLMs may have potential as adjunctive triage support tools, but their performance remains inconsistent and their limitations are clinically important. Safety implications of limited sensitivity for patients assigned to the highest-acuity triage category A central finding of this review is the limited sensitivity of LLMs in identifying patients assigned to the highest-acuity triage category by the reference standard. The pooled sensitivity was 61%, whereas the pooled specificity was 97%, with an AUC of 0.82. Although the high specificity suggests that LLMs may perform relatively well in ruling out non-high-acuity cases, the lower sensitivity raises an important safety concern. The reference standards used in the original studies were study-defined triage classifications, such as emergency physician triage, expert-panel triage, or prespecified correct triage levels in simulated cases. These labels may not fully represent true physiological severity and may themselves be influenced by clinician experience, local triage practice, ED crowding, resource availability, and institutional constraints. This concern is supported by large-scale evidence from Sax et al. [2], who reported an Emergency Severity Index version 4 mistriage rate of 33.7% among 5.1 million US ED visits, with under-triage concentrated among clinically critical patients. Because the reference standards used in the original studies were not consistently anchored to objective patient outcomes, the pooled sensitivity, specificity, and SROC AUC should be interpreted as the ability of LLMs to identify cases labelled as highest acuity by study-defined reference standards. These results do not establish the ability of LLMs to identify true physiological critical illness or patient-centered outcomes. From a clinical perspective, a sensitivity of 61% implies that approximately 39% of patients classified as highest acuity by the reference standard may not be identified as such by the LLM. In emergency triage, this pattern is more concerning than over-triage, because under-triage may delay physician assessment, monitoring, diagnostic testing, and time-sensitive treatment [29]. Such delays may be especially consequential for patients with rapidly evolving conditions or subtle but high-risk presentations. In such cases, delayed recognition of the highest level of urgency may increase the risk of clinical deterioration before definitive evaluation or intervention can be provided. Therefore, even if LLMs show acceptable overall discrimination, their limited sensitivity for highest-acuity triage cases warrants considerable caution. This interpretation is consistent with prior research. Haim et al. [30] reported that, when using the Emergency Severity Index, ChatGPT tended to assign some patients who should have been prioritized at the highest acuity to level 2 rather than level 1. This pattern is aligned with the present findings and suggests that current LLMs may be relatively conservative in assigning the most urgent triage category. Accordingly, any potential role for LLMs in ED triage should be considered supportive rather than autonomous, with triage nurse judgment and clinical assessment remaining central to patient prioritization. Heterogeneity in LLM triage accuracy and possible explanations Another important finding was the substantial heterogeneity in overall triage accuracy across studies. As shown in the subgroup analysis (Fig. 10), LLM performance appeared more favorable in some simulated case settings than in studies based on real ED cases. This finding should not be interpreted only as a difference in case type, but also as a difference in translational evaluation context. Fig. 10Subgroup analysis of the pooled OR for overall triage accuracy according to case type and translational context Subgroup analysis of the pooled OR for overall triage accuracy according to case type and translational context Several explanations are plausible. First, ED triage is inherently complex and dynamic, involving heterogeneous case-mix, variable symptom presentations, and contextual information that may be incomplete or difficult to standardize. Current LLMs may not have been trained on data that adequately represent the breadth and complexity of real-world triage encounters [31]. Second, most LLM-based triage evaluations rely primarily on text input. This may omit clinically relevant non-textual cues that triage nurses routinely use, such as general appearance, work of breathing, facial expression, skin color, behavior, mental status, or other immediately observable warning signs [16, 32, 33]. Third, methodological differences across studies—including the use of real versus simulated cases, different triage systems, different LLM versions, prompt formats, and reference standards—are also likely contributors to heterogeneity. These observations suggest that the performance of LLMs in ED triage is highly dependent on the case context. Moreover, regardless of whether real or simulated cases were used, the included studies were essentially preclinical in nature. As a result, pooled summary estimates should be interpreted cautiously, particularly when considering translation from structured or simulated scenarios to real clinical workflows. Mitigation strategies and future directions for LLM-supported triage The present findings also highlight several possible directions for improving LLM-supported triage. One notable signal in this review was the more favorable point estimate observed in studies using RAG. Although this subgroup result was not statistically significant, it suggests that providing LLMs with structured, task-relevant external information may improve factual grounding and consistency in triage-related outputs [34, 35]. This possibility is clinically relevant because one recognized limitation of LLMs is the generation of plausible but incorrect responses, often referred to as hallucinations [26]. By incorporating retrievable external evidence, RAG may help reduce unsupported outputs and improve traceability [35, 36]. In theory, further gains may also be achieved through integration with structured clinical knowledge bases, protocol-based triage rules, or multimodal inputs capable of capturing visual and other non-textual clinical cues [37, 38]. However, these approaches should be viewed as developmental strategies rather than established solutions. Important challenges remain, including limited interpretability, model version instability, privacy and data governance concerns, and uncertainty regarding generalizability across institutions and triage systems [39–42]. Recent prospective evidence from an AI-based triage decision-support system embedded in ED nurse workflow provides a useful translational example, highlighting the need to evaluate triage AI within real clinical workflows [43]. Future studies should therefore focus on standardized prompting, domain-specific adaptation, multimodal model development, and prospective real-world validation. Particular attention should be paid to safety-critical outcomes, especially under-triage and missed identification of patients requiring the highest level of urgency. Limitations and implications The findings of this review should be interpreted within the boundaries of the available evidence. The included studies were retrospective or in-silico evaluations without integration into live emergency department workflows and did not directly establish prospective clinical utility, workflow impact, or improvement in patient-centered outcomes. In addition, it should be noted that most of the original studies used the triage decisions of multiple emergency physicians as the reference standard. However, this standard is not based on actual patient clinical outcomes and may not fully capture all dimensions of real clinical severity. Therefore, the results of this study should be interpreted as evidence comparing the performance of triage nurses and LLMs in ED triage, rather than as direct evidence of LLMs’ effectiveness in real-world clinical settings. Overall, current LLMs show potential for use as adjunct decision-support tools in ED triage, but they should not be considered replacements for triage nurses. Their limited sensitivity for highest-acuity cases, together with the substantial heterogeneity across studies, underscores the need for cautious interpretation and further rigorous evaluation before broader implementation is considered. Limitations This review has several limitations that should be considered when interpreting the findings. First, substantial between-study heterogeneity was observed, particularly for the analysis of overall triage accuracy, which reduces the certainty and generalizability of the pooled estimates. Second, the included studies were preclinical observational studies rather than prospective assessments conducted in real-time ED workflows. As a result, the present findings reflect comparative triage classification performance under study conditions and should not be interpreted as direct evidence of clinical utility, workflow effectiveness, or improvement in patient-centered outcomes. Third, the reference standards used in the original studies were study-defined triage classifications, largely non-outcome-anchored triage labels, which may not fully represent true clinical severity and may themselves be influenced by under-triage, over-triage, clinician experience, and contextual and institutional factors. As a result, reference-standard label bias is a structural limitation of the current evidence base. Fourth, for the diagnostic accuracy analysis, the target-positive group was operationally defined as patients assigned to the highest-acuity triage category by the reference standard. This definition was necessary because objective clinical endpoints such as intensive care unit admission, deterioration, or mortality were not consistently reported in the included studies. Therefore, this analysis should be interpreted as reflecting identification of the highest-acuity triage category rather than definitive physiological critical illness. In addition, this review included only English- and Chinese-language publications, which may have excluded relevant studies published in other languages and introduced language bias. Future research should consider multilingual searching with appropriate translation support. Finally, although QUADAS-2 provided a structured framework for assessing methodological quality, it does not fully capture AI-specific sources of bias, including possible training-data leakage, prompt instability, model version drift, and limited transportability across settings. Future research should also consider using emerging AI-specific risk-of-bias tools, such as QUADAS-AI [44], once they become fully developed and available for direct application. Such tools may help future reviews better assess AI-specific methodological concerns. For these reasons, the findings of this review should be interpreted cautiously. Nevertheless, this study provides an updated synthesis of the available evidence on LLM performance in ED triage and highlights important directions for future research and evaluation. Conclusion This systematic review and meta-analysis suggests that current LLMs are not yet ready for standalone use in ED triage. Although they demonstrated moderate discriminatory performance and relatively high specificity for excluding non-high-acuity cases, their limited sensitivity for patients assigned to the highest-acuity triage category by the reference standard remains an important safety concern. In addition, current evidence does not show superior triage accuracy over triage nurses. At the same time, the available evidence suggests some potentially encouraging signals. LLMs showed high specificity for ruling out non-high-acuity cases and demonstrated moderate overall discriminatory performance. In addition, RAG-enhanced models showed higher point estimates for triage accuracy, suggesting a possible direction for future improvement. However, the current evidence base is derived primarily from observational studies and does not establish prospective clinical effectiveness. Future research should prioritize prospective real-world evaluation, more standardized study designs, clearer and more objective reference standards, and assessment of patient-centered and safety-critical outcomes. With these safeguards, LLMs may have a future role as adjunct decision-support tools for triage nurses, but further rigorous validation is required before broader implementation can be considered. Supplementary Information Below is the link to the electronic supplementary material. Supplementary Material 1 Supplementary Material 1 Supplementary Material 2 Supplementary Material 2