ChatGPT4’s diagnostic accuracy in inpatient neurology: A retrospective cohort study Authors: Sebastian Cano-Besquet, Tyler Rice-Canetto, Hadi Abou-El-Hassan, Simon Alarcon, Jason Zimmerman, Leo Issagholian, Nasser Salomon, Ivan Rojas, Joseph Dhahbi, Michael M Neeki Published: 2024 ABSTRACT AbstractBackgroundLarge language models (LLMs) such as ChatGPT-4 (CG4) are proving to be valuable tools in the medical field, not only in facilitating administrative tasks, but in augmenting medical decision-making. LLMs have previously been tested for diagnostic accuracy with expert-generated questions and standardized test data. Among those studies, CG4 consistently outperformed alternative LLMs, including ChatGPT-3.5 (now no longer publicly available for use) and Google Bard (now known as “Google Gemini”). The next logical step was to begin to explore CG4’s accuracy within a specific clinical domain. Our study evaluated the diagnostic accuracy of CG4 within an inpatient neurology consultation service.MethodsWe conducted a review of all patients listed on the daily neurology consultation roster at Arrowhead Regional Medical Center in Colton, CA, for all days surveyed until we reached a total of 51 patients, ensuring a complete and representative sample of the patient population. ChatGPT-4, using HIPAA-compliant methodology, received patient data from the Epic EHR as input and was asked to provide an initial differential diagnoses list, investigations and recommended actions, a final diagnosis, and a treatment plan for each patient. A comprehensiveness scale (an ordinal scale between 0 and 3) was then used to rate match accuracy between consultant and CG4 initial diagnoses and the consultants’ final diagnoses. In this proof-of-concept study, we assumed that the neurology consultants’ final diagnoses were accurate. We employed a non-parametric bootstrap resampling method to create 95 % confidence intervals around mean scores, a Fisher’s Exact test, a Wilcoxon Rank Sum test, and ordinal logistic regression models to compare the performance between consultant and CG4 groups.FindingsOur study found that CG4 demonstrated diagnostic accuracy comparable to that of consultant neurologists. The most frequent comprehensiveness score achieved by both groups was "3," with consultant neurologists achieving this score 43 times and CG4 achieving it 31 times. The mean comprehensiveness scores were 2.75 (95 % CI: 2.49–2.90) for the consultant group and 2.57 (95 % CI: 2.31–2.67) for the CG4 group. The success rate for comprehensive diagnoses (a score of "2″ or "3″) was 94.1 % (95 % CI: 84.1%–98.0 %) for consultants and 96.1 % (95 % CI: 86.8%–98.9 %) for CG4, with no statistically significant difference in success rates (p = 1.00). The Wilcoxon Rank Sum Test indicated that the consultant group had a higher likelihood of providing more comprehensive diagnoses (W = 1583, p = 0.02). Ordinal logistic regression models identified significant predictors of diagnostic accuracy, with the consultant diagnosis group showing an odds ratio of 3.68 (CI 95 %: 1.28–10.55) for higher value outcomes. Notably, integrating CG4’s initial diagnoses with those from consultants could achieve comprehensive diagnostics in all cases, indicating a number needed to treat (NNT) of 17 to attain one additional comprehensive diagnosis.InterpretationOur findings suggest that CG4 can serve as a valuable diagnostic tool within the domain of inpatient neurology, providing comprehensive and accurate initial diagnoses comparable to those of consultant neurologists. The use of CG4 might contribute to better patient outcomes by serving as an aid in diagnosis and treatment recommendations, potentially leading to reduced missed diagnoses and quicker diagnostic processes. Continuous strategies and evaluations to improve LLMs' accuracy remain crucial. Further studies with larger sample sizes and independent third-party evaluations are recommended to confirm these findings and assess the impact of LLMs on patient health.Keywords: Large language models (LLMs), ChatGPT-4 (CG4), diagnostic accuracy, Inpatient neurology, Differential diagnoses, Treatment recommendations, Epic EHR, Artificial intelligence (AI), Clinical decision support, Ordinal logistic regression Research in context Evidence before this study A targeted literature survey was conducted for articles on the application of LLMs within the medical domains to date. This was done via a search of the PubMed database with the search terms “Large Language Model AND Diagnostic Accuracy.” Studies were limited to those published between January 1st, 2022 and March 19th, 2024. Surveyed articles were evaluated by three separate study authors to ensure that the content was relevant to either artificial intelligence (AI) or LLMs and healthcare. Articles primarily focused on AI and LLM ability to pass medical licensing examinations, diagnostic accuracy with respect to imaging, and evaluation of LLM diagnostic accuracy as assessed by expert panel review. There was a notable dearth of studies assessing clinical diagnostic accuracy with real patient data within medical domains like inpatient neurology. Additionally, inpatient neurology was selected as a specialty to study because it represents a medical specialty that has geographic disparities with respect to access to clinicians [1]. This presents a critical opportunity for AI technologies, such as large language models, to help bridge the gap by improving diagnostic accessibility and accuracy, particularly in regions with limited access to expert clinicians. Added value of this study Our study provides a comprehensive analysis of the diagnostic accuracy of ChatGPT4 within the domain of inpatient neurology. This study is notable for its use of real patient data at a bustling regional medical center, novel analytic methodology, and thorough statistical analysis. While we acknowledge an ideal of having expert consensus for diagnoses, we have instead implemented a novel method to assess diagnostic accuracy via ICD 10 code agreement. We assume the validity of neurology consultant diagnoses, and then grade CG4’s accuracy based on its ability to match those items. Our study adds value by offering a more direct comparison between artificial intelligence and expert consultants that is lacking in current literature. Additionally, we implement a less biased comparison by relying on ICD 10 code agreement as opposed to third party expert opinion. By assuming accurate consultant diagnoses we bypass the need to have an expert panel review large numbers of cases. Using ICD 10 code matching to help guide our researchers, we then assign to CG4’s diagnoses success or failure as determined by whether they match those of the consultants. Our comprehensive analysis details comparative statistics between consultant and CG4 performance. We used analytical methods combined with bootstrapping methods, a two-tailed Fisher’s Exact Test, Wilcoxon Rank Sum Test, and Ordinal Logistic Regression to compare the diagnostic accuracy of neurology consultant and CG4 groups. Implications of all the available evidence This study reveals that CG4 might serve as a valuable diagnostic tool within the domain of inpatient neurology. A growing body of evidence is revealing that LLMs might contribute to better patient outcomes by serving as an aid in diagnosis and treatment recommendations. The use of this technology has utility particularly in medical domains like neurology where geographic disparities in the presence of expert clinicians exist. The results of this study support the conclusion that LLM use might lead to a reduction in missed diagnoses, especially in geographic regions where expert consultants are unavailable. Introduction Artificial intelligence (AI) refers to the simulation of human intelligence in machines that are programmed to think, learn, and perform tasks autonomously. In medicine, AI has been increasingly applied to areas such as image recognition, diagnostic decision-making, and data analysis [2,3]. One specific type of AI, large language models (LLMs), refers to deep learning models trained on vast amounts of text data to understand and generate human language [4]. These models, such as ChatGPT-4, have demonstrated the ability to process complex medical information and assist in generating diagnoses and treatment recommendations by analyzing patient data from electronic health records (EHRs) [5]. In neurology, LLMs have the potential to assist clinicians by rapidly analyzing large datasets, generating differential diagnoses, and suggesting potential interventions based on symptoms, imaging, and patient history. This study focuses on evaluating the accuracy of LLMs in the context of inpatient neurology consultations, where timely and accurate diagnoses are critical for patient outcomes. The use of computers to aid in medical diagnosis has been postulated as early as the 1950’s with research articles examining how physicians generate medical diagnoses and if this ability could be replicated by computers [6]. Some of the first computer assisted diagnostic tools emerged in the 1970’s. Systems like MYCIN developed by Stanford were able to offer antimicrobial recommendations for patients with severe bacterial infections [7]. However, these tools were often limited in their scope and were often too cumbersome in manual data input to offer translational changes to practice. The advent of tools that take advantage of machine learning like generative artificial intelligence and large language models can help eliminate previous barriers that existed with past computer aided diagnostic models. Generative artificial intelligence and large language models use machine learning to allow computers to produce natural language autonomously by using sets of training data as source material to create new outputs [8]. The recent release and advancements in tools like ChatGPT, Bard (now known as Gemini), and LLaMA have led to renewed interest in the impact that computer aided diagnosis can have on medicine and healthcare. To inform the current study, a targeted literature survey was conducted to investigate the uses and recent progress of AI and LLM use in medicine. A search of the PubMed database for articles published from January 1st, 2022 until March 19th, 2024 was conducted using the search terms “Large Language Model AND Diagnostic Accuracy”. Surveyed articles were evaluated by three separate study authors to ensure that the content was relevant to either AI or LLMs and healthcare. The search returned 136 articles, of which 8 were excluded because they were unrelated to either AI or LLMs and healthcare. The 128 included articles were evaluated for several features, including their primary medical domains and whether their experiments utilized fictitious or real patient data. The articles spanned 33 distinct primary medical domains with the largest number of studies in radiology (17, 13.3 %), medical education (16, 12.5 %), neurology (13, 10.2 %), ophthalmology (12, 9.4 %), psychiatry (10, 7.8 %), and oncology (8, 6.3 %). Only 71 of these 128 studies evaluated AI models that used actual patient data. It may be the case that further interest in LLMs as diagnostic tools has been piqued by recent studies demonstrating LLMs ability to pass physician licensing exams [[9], [10], [11], [12], [13]]. Most articles evaluated during the targeted literature survey focused primarily LLMs’ ability to pass physician licensing board exams, AI’s diagnostic abilities with respect to diagnosis of imaging data, or the evaluation of LLM diagnostic accuracy by expert consensus. In these domains and others, LLM capabilities and performance results differed significantly by model. A study from Toyoma et al. found that CG4 outperformed earlier versions of ChatGPT and Google’s Bard (now known as Gemini) in its ability to answer medical radiologic licensing exam questions [14]. One study found that the LLM performed better than many of the medical professionals who took the same exam [10]. While LLMs appear to have made great strides in their abilities to perform well on licensing exams, a more applicable use of their abilities would be their performance on analyzing real patient data to inform clinical decision-making. Publications on usage of AI in clinical decision-making have been appearing more frequently, particularly with respect to the use of AI in evaluating imaging studies [15,16] LLMs have been used within radiology to generate image findings, impressions, and differential diagnoses for patient CT scans with relatively high readability and have generated reasonable image findings [17]. For example, Cid et al. considered AI’s ability to identify significant findings on chest X-rays and found that it performed similarly to historical reports [15]. Some studies evaluating LLM’s ability to perform chest x-ray interpretations have, however, yielded mixed results. Lee et al. found that LLMs like Kakao Brain Artificial Neural Network for Chest X-ray Reading (KARACXR) and ChatGPT were able to obtain ratings of acceptable accuracy with 70.50 %, 68.00 % for KARACXR and 40.50 %, 47.00 % for ChatGPT when reviewed by independent expert reviewers [18]. LLM utility is also being explored in fields such as Ophthalmology, where it is used not only for image analysis, but for information screening, predictive modeling, patient inquiry responses, and summarization [19]. LLMs have also been explored in the domain of data retrieval from patient records and automation of repetitive tasks. In one study, LLMs have been shown to outperform human data collectors when extracting clinical information from ultrasound notes and operative reports [20]. In the study performed by Abu-Ashour et al., humans made extraction errors at a rate of 28.2 % compared to just 2.9 % for CG4. CG4 was also able to perform these more accurate extractions 40 times faster than the human extractors. With respect to the medical domain of neurology, previous studies have demonstrated LLM ability to extract relevant neurological data like Mini Mental Status Exams and dementia rating scores from clinical notes with an accuracy of 83 % and sensitivity of 89.7 % [21]. Additionally, LLMs have demonstrated competency when tasked with answering Neurology certification examination questions, having received passing scores on examinations [22]. Together, these results suggest that CG4 and other LLMs can perform many of the individual steps involved in coming to a diagnosis in an inpatient neurology service context with accuracy and speed. However, the evidence to support AI’s ability to make accurate diagnoses in a clinical environment is under-represented in current literature. There is a notable lack of studies investigating the performance of LLMs like CG4 in inpatient neurology consultation services, where real-time decision-making and the integration of complex clinical data are crucial. Existing literature has not sufficiently addressed how LLMs can complement human neurologists in such settings, particularly in terms of diagnostic speed, accuracy, and comprehensiveness. Fraser et al. used patient-reported symptoms as an input and compared LLM to physician diagnoses [23]. However, most studies to date assess LLM performance in a non-clinical setting [24]. In these studies, in which LLMs generate diagnoses based on acquired patient data and then expert reviewers evaluate and score the LLM’s performance [13,25,26]. Our study departs from previous investigations in this field via its use of the patient’s clinical record as a source input and its direct comparison of LLM and consultant’s performance within a clinical context. Furthermore, the usage of ICD 10 codes evaluate LLM and consultant performance adds objectivity to LLM performance assessment. Performance evaluation that uses an objective metric like ICD 10 agreement can help eliminate potential bias that can occur with expert reviewers’ subjective assessments of LLM diagnostic performance. This study aims to evaluate the diagnostic accuracy of CG4 in an inpatient neurology consultation service by comparing its performance to that of human neurologists. We investigate whether CG4 can match or exceed the comprehensiveness of consultant diagnoses and explore its potential to reduce diagnostic times. Additionally, we aim to analyze how CG4 handles complex neurological cases involving various disease categories, including structural, vascular, and autoimmune conditions. Methods Ethics Our study was reviewed and approved by the Arrowhead Regional Medical Center Institutional Review Board (IRB) as Protocol #23–49 on October 30, 2023. Our study design is HIPAA compliant. We utilized a private OpenAI instance of CG4 via Microsoft Azure. All data was encrypted in transit and at rest in our own virtual server. Upon conclusion of the investigation, all data in the virtual system was deleted. De-identified data, however, were retained for publication. Data security and HIPAA compliance All patient data were stored and processed using Microsoft Azure, a platform designed to meet HIPAA and other regulatory requirements through robust security features. Azure provides encrypted storage and data protection, both at rest and in transit, using industry-standard encryption protocols (https://learn.microsoft.com/en-us/legal/cognitive-services/openai/data-privacy?tabs=azure-portal). Access to Azure resources was restricted to authorized personnel only, and all data access was logged and monitored for security purposes. Data access was only permitted via secure login protocols, either on-campus or remotely, through the use of Forticlient VPN. Forticlient establishes a secure, encrypted connection, ensuring that all data transmissions between the user and Azure are protected through secure tunneling methods (www.fortinet.com). Study Design Primary outcome The primary outcome of this study was to evaluate the diagnostic accuracy and comprehensiveness of ChatGPT-4 compared to that of human neurologists in an inpatient neurology consultation service. Diagnostic accuracy was defined as the extent to which ChatGPT-4’s diagnoses matched those of the neurologist consultants, and was measured using a binary "match" versus "non-match" criterion for each diagnosis, based on ICD-10 coding as described below. Comprehensiveness was evaluated using a scoring system that assessed the number of relevant diagnoses identified by both ChatGPT-4 and the neurologists. Higher scores indicated a more thorough and comprehensive assessment of the patient’s diagnoses. Our study was conducted between October 31st, 2023, and December 28th, 2023 at Arrowhead Regional Medical Center (ARMC), a 456-bed regional safety net hospital in Colton, San Bernardino County, California. Patients who received a neurology consultation and were signed off by neurology on days on which the consultation list was reviewed by researchers were added to the study population. Days on which the consultation list was reviewed were not consecutive due to researcher availability. ‘Signing off’ in the inpatient context means the completion of the consultant evaluation and the conclusion of their involvement with the case. Fig. 1 illustrates the patient screening and selection process, including the criteria for exclusion based on whether consultant neurologists rendered final diagnoses for each case. Our goal was to provide a complete and representative sample of the patient population under neurological evaluation at ARMC. To achieve sufficient statistical power, our minimum sample size was set at N ≥ 51 consultations. Patient screening flowchart. This flowchart illustrates the process of identifying, screening, and including patient cases in the study. A total of 62 patient cases were identified from the EPIC database. After screening, 11 cases were excluded: 4 cases where the neurology consultation concluded before rendering a final diagnosis, and 7 cases where the patient expired before a final diagnosis could be rendered. A total of 51 cases were included in the final analysis. For each consultation, the date and time of the consultant neurologist’s initial note was documented, as well as the neurologist’s initial differential diagnoses and recommendations. If there was a separate final note in which the neurologist signed off, the date and time of the final note, as well as final diagnoses and final treatment recommendations were also extracted. In cases where there were no subsequent notes and the neurologist signed off on the patient in the initial note, the initial note also served as the final note, along with the diagnoses and recommendations. In this proof-of-concept study, we assumed the accuracy of all consultant final diagnoses. For each consultation included in the study, the patient’s physician notes, imaging results, and laboratory results, which we shall henceforth refer to as the patient’s considered medical data (CMD), were then inputted from EPIC into CG4 in a HIPAA-compliant manner from December 12, 2020 to the date and time of the of the initial neurology consultation note in order to provide as much context regarding the patient’s health status as possible. The initial date of considered patient data (December 12, 2020) was selected because it marks the beginning of the implementation of the EPIC system at ARMC, thus providing as comprehensive a data set as possible within the electronic health record (EHR). Records internal to ARMC but preceding this initial date, and records external to ARMC, such as those available through Care Everywhere, were excluded from the input data. Specific types of laboratory and imaging results included in the CMD were case-specific, depending on the data available for each patient. In addition to patient age and sex, we categorized and documented the neurologic diseases identified in the consultant’s final diagnoses to gain further insight into their predictive value with respect to CG4 and consultant diagnostic comprehensiveness. Patient data for this study were extracted using the EPIC electronic health record (EHR) system, which is one of the most widely implemented EHR platforms in the United States. EPIC is a comprehensive EHR system that integrates clinical, administrative, and financial data to support patient care, documentation, and decision-making processes. It allows healthcare providers to access a patient's medical history, diagnostic tests, imaging, lab results, and clinical notes in a centralized interface. In this study, EPIC was used to retrieve patient data relevant to the inpatient neurology consultation service, including demographic information, clinical notes, and diagnostic data. EPIC is used by over 250 million patients (www.epic.com). CG4 was queried between February 2nd, 2024 to March 13, 2024. We queried CG4’s default pre-trained model without querying the internet or fine-tuning. Fig. 2 illustrates the flow of information and study design. The CMD as described above was inputted into CG4 as part of series of prompts visible in Supplementary Data 1, available in a deidentified form upon request. These prompts elicited from CG4 lists of initial differential diagnoses and initial recommendations. A comprehensiveness scale, or ordinal scale between 0 and 3, was then used to rate match accuracy between consultant and CG4 initial diagnoses and the consultants’ final diagnoses. In this proof-of-concept study, we assumed that the neurology consultants’ final diagnoses are accurate. The ordinal scale values are defined as follows: 0 is Failure, that is 0 % of the final diagnoses were appreciated in the initial diagnoses; 1 is Mostly Incomplete, that is > 0 % but < 50 % of the final diagnoses were appreciated in the initial diagnoses; 2 is Mostly Complete, that is ≥ 50 % but < 100 % of the final diagnoses were appreciated in the initial diagnoses; 3 is Complete, that is 100 % of the final diagnoses were appreciated in the initial diagnoses. We also used one other method of outcome categorization for hypothesis testing that used binary variables. Mostly Complete or Complete initial diagnoses were included in one group, which we shall call comprehensive or success, and Mostly Incomplete or Failure in initial diagnoses were included in the other group, which we shall call not comprehensive or failure. Outlines the study design for comparing the performance of consultant neurologists and CG4 in forming differential diagnoses (DDx) and making treatment recommendations. Patient charts are initially reviewed by both the consultant neurologists and CG4 to generate initial differential diagnoses (Physician Initial DDx and CG4 Initial DDx). Based on these initial diagnoses, both groups generate initial treatment recommendations. CG4 is given an opportunity to auto-correct its initial diagnoses if the accuracy is less than 3. If CG4 does not comment on a parameter of interest regarding recommendations, a query is made to refine the recommendations. Final patient charts are then reviewed, and both groups generate final differential diagnoses and treatment recommendations (Physician Final DDx and CG4 Final DDx). The consultant's processing steps are represented by blue boxes, and CG4's processing steps are indicated by red boxes. This figure illustrates the sequential steps in the study, highlighting the parallel processes and decision points for evaluating the performance of the two groups. Matching criteria (diagnosis validation process) For both the neurologist consultants and CG4, diagnoses were validated and assigned ICD-10 codes to ensure consistent and objective comparison. Consultant diagnoses did not originally contain ICD codes, so study authors recoded both consultant and CG4 diagnoses to the nearest ICD-10 codes using the World Health Organization’s ICD-10 system as a standardized reference. Each diagnosis was assigned the highest-level parent ICD-10 code that still accurately reflected the core components of the diagnosis. For example, if the consultant’s diagnosis were “ST elevation myocardial infarction of the anterior wall” (ICD-10 code I21.0), the core diagnosis of “acute myocardial infarction” (ICD-10 code I21) would be used for comparison. CG4’s diagnoses were considered a match if they aligned with the core diagnosis of the consultant, regardless of the level of detail CG4 provided. In cases where CG4 missed key components of a diagnosis, open-ended queries were used to allow CG4 to elaborate on its initial diagnosis (see Table 1 for a list of cases requiring re-prompting). We defer to a future study the creation of a strategy to address the match rates of secondary components of diagnoses. Table 1Case-by-case analysis of patient demographics, cohort performances, and CG4 reprompting requirements.Table 1Case IDPatient AgePatient SexDisease CategoriesConsultant Initial DDX Match RateConsultant Initial DDX Comprehensiveness ScoreCG4 Initial DDX Match RateCG4 Initial DDX Comprehensiveness ScoreCG4 Final DDX Match RateCG4 Final DDX Comprehensiveness ScoreCG4 Required Reprompting Following Initial DDX?CG4 Required Reprompting Following Final DDX?150MVascular2/2 (100 %)32/2 (100 %)32/2 (100 %)3NoNo250FOther1/1 (100 %)31/1 (100 %)31/1 (100 %)3YesNo361MOther1/1 (100 %)31/1 (100 %)31/1 (100 %)3NoNo431FVascular1/1 ((100 %)31/1 (100 %)31/1 (100 %)3NoNo541MTrauma1/1 (100 %)31/1 (100 %)31/1 (100 %)3NoNo663MEpilepsy1/1 (100 %)31/2 (50 %)21/2 (50 %)2YesNo989FEpilepsy; Structural3/3 (100 %)33/3 (100 %)33/3 (100 %)3NoYes1077FDegenerative; Other1/2 (50 %)21/2 (50 %)21/2 (50 %)2NoNo1156MOther; Epilepsy2/2 (100 %)32/2 (100 %)32/2 (100 %)3NoNo1271FNeuromuscular2/2 (100 %)32/2 (100 %)32/2 (100 %)3NoNo1357MEpilepsy1/3 (33 %)12/3 (67 %)22/2 (100 %)3NoNo1473FOther1/1 (100 %)31/1 (100 %)31/1 (100 %)3NoNo1533FStructural3/3 (100 %)33/3 (100 %)33/3 (100 %)3NoNo1666MMetabolic1/1 (100 %)31/1 (100 %)31/1 (100 %)3NoNo1748MStructural1/1 (100 %)31/1 (100 %)31/1 (100 %)3NoNo1959FOther3/3 (100 %)32/3 (66 %)22/3 (66 %)2NoNo2042MOther; Vascular3/3 (100 %)33/3 (100 %)33/3 (100 %)3NoNo2149MDegenerative1/1 (100 %)31/1 (100 %)31/1 (100 %)3NoNo2268MDegenerative1/1 (100 %)31/1 (100 %)31/1 (100 %)3NoNo2344FInfectious; Autoimmune1/2 (50 %)21/2 (50 %)21/2 (50 %)2NoNo2559FMetabolic; Vascular; Infectious4/4 (100 %)32/4 (50 %)22/4 (50 %)2YesNo2654FAutoimmune; Other4/4 (100 %)33/4 (75 %)23/4 (75 %)2NoNo2783FVascular2/2 (100 %)32/2 (100 %)32/2 (100 %)3NoNo2825MOther2/2 (100 %)31/1 (100 %)31/1 (100 %)3NoNo2950FAutoimmune0/2 (0 %)02/2 (100 %)32/2 (100 %)3NoNo3050FOther; Metabolic; Autoimmune0/2 (0 %)02/4 (50 %)22/4 (50 %)2NoNo3150FVascular1/1 (100 %)31/1 (100 %)31/1 (100 %)3NoNo3277MVascular3/3 (100 %)32/3 (67 %)22/3 (67 %)2NoNo3346MEpilepsy; Vascular3/4 (65 %)23/4(75 %)23/4 (75 %)2NoNo3555MVascular; Other4/4 (100 %)32/4 (50 %)22/4 (50 %)2NoNo3625FEpilepsy; Other2/3 (66 %)23/3 (100 %)33/3 (100 %)3NoNo3761MVascular; Other; Metabolic5/5 (100 %)34/5 (80 %)24/5 (80 %)2NoNo3883MVascular; Autoimmue4/4 (100 %)33/4 (75 %)23/4 (75 %)2NoNo3925FEpilepsy; Other4/4 (100 %)34/4 (100 %)34/4 (100 %)3NoNo4044FAutoimmune1/1 (100 %)31/1 (100 %)31/1 (100 %)3NoNo4173FOther1/1 (100 %)31/1 (100 %)31/1 (0 %)3NoNo4252MOther1/1 (100 %)31/1 (100 %)31/1 (100 %)3NoNo4372MVascular; Epilepsy3/3 (100 %)32/3 (67 %)2NANANoNo4461MVascular2/2 (100 %)31/2 (50 %)21/2 (50 %)2NoNo4560FEpilepsy1/1 (100 %)31/1 (100 %)31/1 (100 %)3NoNo4664FVascular1/1 (100 %)31/1 (100 %)31/1 (100 %)3NoNo4768MVascular1/2 (50 %)21/2 (50 %)22/2 (100 %)3NoNo4825FEpilepsy2/2 (100 %)32/2 (100 %)32/2 (100 %)3NoNo4925FEpilepsy; Other1/1 (100 %)31/1 (100 %)31/1 (100 %)3NoNo5066FEpilepsy; Other3/3 (100 %)32/3 (66 %)22/3 (66 %)2NoNo5260FVascular2/2 (100 %)32/2 (100 %)32/2 (100 %)3NoNo5342MVascular; Infectious3/3 (100 %)33/3 (100 %)33/3 (100 %)3NoNo5436MVascular1/1 (100 %)31/1 (100 %)31/1 (100 %)3NoNo5659MTrauma; Vascular3/3 (100 %)31/3 (33 %)11/3 (33 %)1YesNo Case-by-case analysis of patient demographics, cohort performances, and CG4 reprompting requirements. Exceptions to the ICD-10 recoding process Two exceptions to the ICD-10 recoding process were applied in cases of symptom-based diagnoses and when the researcher exercised discretion in interpreting diagnoses as equivalent based on core components. For transparency, Supplementary Table 1 provides a detailed comparison of all diagnoses and the ICD-10 codes assigned to each, enabling a direct comparison of consultant and CG4 performance based on objective coding criteria. 1.Symptom-Based Diagnoses: The first exception was when the consultant made a symptom-based diagnosis without a clear etiology, as occurred in Case 2: “Stroke-like symptoms of unclear etiology.” A CG4 diagnosis was considered a match if the symptom presentation given as the neurologist’s diagnosis was directly explained by the etiology or specific condition given in CG4’s diagnosis. The opposite would not be considered a match: if the neurologist described a condition or etiology, a match would not occur if CG4 described only a symptom presentation. Symptom-Based Diagnoses: The first exception was when the consultant made a symptom-based diagnosis without a clear etiology, as occurred in Case 2: “Stroke-like symptoms of unclear etiology.” A CG4 diagnosis was considered a match if the symptom presentation given as the neurologist’s diagnosis was directly explained by the etiology or specific condition given in CG4’s diagnosis. The opposite would not be considered a match: if the neurologist described a condition or etiology, a match would not occur if CG4 described only a symptom presentation. 2.Discretion in Equivalent Diagnoses: The second exception arose when ICD-10 codes were different, but the researcher exercised discretion to interpret a match in core diagnostic components. For example, in Case 15, the consultant identified “Altered mental status and delirium secondary to frontotemporal tumor resection,” which was interpreted as “R41.0; Disorientation, unspecified,” by the researcher. CG4 identified “Postoperative changes following resection right sphenoid wing meningioma,” which was interpreted as “Z98.890; Other specified postprocedural states.” Though the ICD-10 codes differ, they were deemed equivalent in core diagnosis by the researcher. Discretion in Equivalent Diagnoses: The second exception arose when ICD-10 codes were different, but the researcher exercised discretion to interpret a match in core diagnostic components. For example, in Case 15, the consultant identified “Altered mental status and delirium secondary to frontotemporal tumor resection,” which was interpreted as “R41.0; Disorientation, unspecified,” by the researcher. CG4 identified “Postoperative changes following resection right sphenoid wing meningioma,” which was interpreted as “Z98.890; Other specified postprocedural states.” Though the ICD-10 codes differ, they were deemed equivalent in core diagnosis by the researcher. Lastly, 30 cases initially included in the study population were excluded due to reasons such as resolution of symptoms or patient expiration prior to a final neurologist diagnosis. Statistical analysis All statistical was performed in R version 4.1.2 “Bird Hippie”. Firstly, we performed descriptive statistics, and created comparative counts of comprehensive scores by consultant and CG4 groups. We then calculated mean comprehensiveness scores for each group. A non-parametric bootstrap resampling method was employed to create 95 % CIs around each mean. This technique is particularly well-suited for small sample sizes or when the underlying population distribution is unknown or not normal. This thereby provides a more flexible and robust estimation of confidence intervals compared to traditional methods based on the assumption of normality. We next created comparative counts of success and failure of initial diagnoses per the binary categorization described in the Study Design section above. We constructed Wilson 95 % confidence intervals around the success rate for each group. We next conducted a two-tailed Fisher’s Exact test and obtained a p-value and confidence interval. We defined a p-value of less than 0.05 as statistically significant. Prior to data collection, a power analysis was conducted to determine the necessary sample size for adequate statistical power. The analysis aimed to estimate the ability of the study to detect a difference in the probability of success between two diagnostic groups, consultants and CG4, in making comprehensive initial diagnoses. The probabilities of success were set at 0.95 for the Consultant group and 0.75 for the CG4 group based on preliminary data and clinical relevance. Using these parameters, a significance level of 0.05, and a two-tailed test, the power analysis estimated that the study would have a power of 0.82 to detect the specified difference in probabilities between the two groups. The power analysis was conducted in R using the “stats” library and the function “power.prop.test(.).” We next performed a Wilcoxon Rank Sum test with continuity correction to compare the central tendencies of both groups. We then then calculated the analytical probabilities and estimated 95 % confidence intervals of drawing higher values from either group at random using a bootstrap resampling technique. We generated 10,000 bootstrap samples by randomly drawing with replacement from the original datasets representing two diagnostic groups, each comprising 51 samples. For each bootstrap sample, we calculated the probability of a higher score being drawn from the Consultant group versus the CG4 group, the reverse scenario, and the probability of drawing scores of equal values. This non-parametric approach allows for the direct incorporation of the empirical distribution of our data, including the handling of tied scores, thereby avoiding assumptions about the underlying distributional form of the data. The 95 % confidence intervals for each estimated probability were derived from the 2.5th and 97.5th percentiles of the bootstrap probability distributions. To investigate the factors influencing diagnostic accuracy, we employed ordinal logistic regression (OLR) models, alongside variation inflation factors and Brant tests for each model, and a collinearity matrix for all predictors. OLR was chosen because our dependent variable, diagnostic comprehensiveness, was ordinal in nature, with higher scores indicating greater comprehensiveness. This approach allowed us to model the relationship between the independent variables (e.g., patient characteristics, disease category, and diagnostic group) and the ordinal outcome of diagnostic accuracy. In the first OLR model (Model 1), we included all available independent variables: patient age, patient sex, disease category (e.g., structural, vascular, autoimmune), and diagnosis group (consultant or CG4). This allowed us to assess how these factors contributed to higher comprehensiveness scores. To assess multicollinearity between the independent variables, we calculated variance inflation factors (VIFs) for each model. A VIF exceeding 10 is generally considered indicative of high multicollinearity, which can undermine the stability of regression models. In Model 1, no VIF exceeded 2.07, indicating that multicollinearity was not a concern in our analysis. Supplementary Table 3 provides VIFs for all models. Additionally, we conducted Brant tests to verify the proportional odds assumption, which is critical for ordinal logistic regression models. The proportional odds assumption posits that the relationship between each independent variable and the outcome is consistent across all thresholds of the outcome variable. In our analysis, the proportional odds assumption held for all independent variables in Model 1, with no statistically significant violations (Supplementary Table 4). The diagnosis group was approaching statistical significance in the Brant test, but it did not reach the threshold of concern. Further models were created to explore the sensitivity of our findings, particularly with respect to disease categories that were perfect predictors in certain models (e.g., "structural" and "trauma"). These models allowed us to assess whether the removal of perfect predictors significantly altered the results, and no significant changes were observed. Role of funders This study was conducted without funding. Results The study included a total of 51 inpatient neurology consultation cases, each evaluated by both a neurologist consultant and CG4. The patients ranged in age from 23 to 89 years old, with 26 females and 25 males. Patient demographics including age and sex are provided in Table 1. The cases represented a broad spectrum of neurological conditions, including but not limited to acute subdural hemorrhage, stroke-like symptoms, seizures, and post-concussion syndrome. Patient diagnoses offered a diverse representation of common inpatient neurology consultations. Diagnoses were coded using the ICD-10 system, with comprehensive diagnosis details provided in Supplementary Table 1. Both the initial differential diagnoses and final diagnoses made by consultants were compared with those generated by CG4. The accuracy and comprehensiveness of both groups were evaluated, with match rates and comprehensiveness scores recorded for each case. Detailed characteristics, including ICD-10 codes, diagnoses, and diagnostic match rates, can be found in Supplementary Table 1. Fig. 3A is a comparative bar chart illustrating the frequency distributions of comprehensiveness scores for initial diagnoses for each group: consultant neurologists and CG4. The most frequent score achieved by both groups is “3,” with a frequency of 43 for consultant neurologists and 31 for CG4. Scores of “2” are less common for both, while CG4 achieved a score of “2” 18 times, and the consultant neurologists did so 5 times. Scores of “0” and “1” were rare for both groups, with respective frequencies of 2 and 1 for consultant neurologists, and respective frequencies of 0 and 2 for CG4. Fig. 3B shows the mean comprehensiveness scores for initial diagnoses achieved by each group. The mean for the consultant group is 2.75 (95 % CI 2.49–2.90), while the mean for the CG4 group is 2.57 (95 % CI 2.31–2.67). Fig. 3A compares the comprehensiveness scores of the initial differential diagnoses (DDx) between two groups: consultant neurologists and CG4. The comprehensiveness scores range from 0 to 3, with 0 being the least comprehensive and 3 being the most comprehensive. The blue bars represent the consultant neurologists, while the red bars represent CG4. Counts for each score category are displayed above the bars. Consultant neurologists had more diagnoses with the highest comprehensiveness score of 3 (43 counts) compared to CG4 (31 counts), indicating the potential stochastic dominance in comprehensiveness scores of initial diagnoses in the consultant group. Fig. 3B presents the mean comprehensiveness scores for the initial differential diagnoses (DDx) of consultant neurologists and CG4, along with their 95 % confidence intervals (CI). The mean score for the consultant neurologists is 2.75 (95 % CI: 2.49–2.90), represented by the blue bar. The mean score for CG4 is 2.57 (95 % CI: 2.31–2.67), represented by the red bar. Error bars (in black) indicate the 95 % confidence intervals of the population mean estimates. Fig. 4A shows the observed number of successful (a score of “2” or “3”) and failed (a score of “0” or “1”) comprehensive initial diagnoses by group. The consultant group achieved success in 48 out of 51 cases, while CG4 did so in 49 out of 51 cases. The success rate for comprehensive diagnoses was 94.1 % (95 % CI 84.1%–98.0 %) in the consultant group and 96.1 % (95 % CI 86.8%–98.9 %) in the CG4 group (Fig. 4B). A Fisher’s Exact test on the count data was performed (Table 2). The data has an odds ratio of 0.66 (95 % CI 0.05–5.99) and a p-value of 1.00. There is no statistically significant difference in the success rates of the consultant and CG4 groups. A power analysis to estimate our ability to detect a difference in success rates of the two groups using a two-sided Fisher’s Exact Test with a significance level of 0.05, 51 samples in each group, and probabilities of success set at 0.95 for the consultant group and 0.75 for the CG4 group results in a power of 0.82 (Table 2). We chose these probabilities in the case of the consultant group to approximate the assumed actual rate of success, and in the case of CG4 as a minimum rate of success for usefulness as an adjunct diagnostic tool. Fig. 4A displays the success and failure counts for the initial differential diagnoses (DDx) made by consultant neurologists and CG4. Success is represented by green bars, while failure is represented by yellow bars. The consultant neurologists achieved 48 successful diagnoses and 3 failures. In comparison, CG4 achieved 49 successful diagnoses and 2 failures. This figure illustrates the performance of each group in terms of the comprehensiveness of their initial diagnoses. Fig. 4B shows the Wilson confidence intervals for the proportions of comprehensiveness scores achieved by consultant neurologists and CG4. The blue line represents the consultant neurologists, with a proportion of 0.941 (95 % CI: 0.841–0.980). The red line represents CG4, with a proportion of 0.961 (95 % CI: 0.868–0.989). The error bars indicate the Wilson confidence intervals, highlighting the range of proportions within which the true value lies with 95 % confidence. This figure illustrates the reliability and precision of the comprehensiveness scores for both groups. Table 2Fisher's exact test and power analysis.Table 2Fisher's Exact Test (two-tailed)ParameterValueOdds Ratio0.66 (95 % CI 0.05–5.99)p-value1Power AnalysisParameterValue⍺0.05alternativetwo-tailedn/group51Consultant Success Rate0.95CG4 Success Rate0.75β0.82 Fisher's exact test and power analysis. A Wilcoxon Rank Sum Test between the sets of comprehensiveness scores produced by the neurology consultant group and CG4 group resulted in a p-value of 0.02 (Table 3). The neurologist consultant group showed stochastic dominance over the CG4 group (W = 1583, p = 0.02) with respect to initial diagnoses comprehensiveness scores. This means that for a pair of randomly selected comprehensiveness scores from the two groups, the probability of the consultant comprehensiveness score being larger than that of the CG4 group is greater than the probability of the CG4 comprehensiveness score being larger than that of the consultant group. Analytically, we calculated the probabilities of a higher comprehensiveness score, or a draw given random selection of a comprehensiveness score from each group via exhaustive pairwise comparison of all possible comprehensiveness score pairings in the data set (Table 4). The probability of a higher comprehensiveness value from the consultant group is 0.33 (95 % CI 0.22, 0.46), the probability of a higher comprehensiveness value from the CG4 group is 0.12 (95 % CI 0.05, 0.21), and the probability of a draw is 0.55 (95 % CI 0.44, 0.66). Table 3Wilcoxon Rank Sum test.Table 3Outcome ComparisonWilcoxon W Statisticp-valueNeurology Consultants15830.02CG410190.02 Wilcoxon Rank Sum test. Table 4Comparison of comprehensiveness score probabilities.Table 4Outcome ComparisonAnalytic Probability95 % CIConsultant's Score is Higher0.33(0.22, 0.46)CG4's Score is Higher0.12(0.05, 0.21)Draw0.55(0.44,0.66) Comparison of comprehensiveness score probabilities. To achieve a more granular understanding of the effects of disease category and diagnostic source on initial diagnosis accuracy we sorted each diagnostic component of each case into one of seven categories: structural, trauma, vascular, epilepsy, infectious, autoimmune, and other. Fig. 5A displays the distribution of comprehensiveness scores by disease category and diagnosis source in a series of boxplots. From Fig. 4A it is evident that ‘structural’ is a perfect predictor and that the category ‘autoimmune’ presented some difficulty to the consultant group. Table 5, which shows the disease category frequencies, suggests one possible explanation for ‘structural’ being a perfect predictor: small sample size in this category. Finally, a series of ordinal logistic regression (OLR) models were created to investigate predictors that have a significant effect on initial diagnostic accuracy (Table 6). Model 1 (Table 6A) utilizes all available independent variables, including patient age, patient sex, disease categories, and diagnosis group (consultant or CG4). As observed, the category ‘structural’ is a perfect predictor. Additional predictors of statistical significance included ‘epilepsy,’ ‘autoimmune,’ and ‘diagnosis group.’ The odds ratio (OR) for being in a higher value outcome given a consultant diagnosis is 3.68 (CI 95 %, 1.28 to 10.55). This is consistent with analytical probabilities previously calculated (Table 4). The odds ratios for a case containing a diagnosis in the category of ‘epilepsy’ or ‘autoimmune’ is 0.16 (CI 95 % 0.04 to 0.66) and 0.07 (95 % CI, 0.01 to 0.34), respectively. Fig. 5B is a collinearity matrix showing the correlations between predictors. No predictors have a correlation value above |0.39|. Supplementary Table 3 shows variance inflation factors (VIF)s for each of the ordinal logistic regression models. In model 1, no VIF exceeds 2.07. Supplementary Table 4 shows the results of Brant Tests for OLR models. The proportional odds assumption holds for all independent variable, that is, the odds ratios do not vary at each threshold of the outcome variable in a statistically significant way. Although, diagnosis source is approaching statistically significance. (Note also that the omnibus calculations for Models 1 and 2 could not be calculated likely due to absence of combinations of dependent and independent variables in our data set.) OLR Model 2 was created to investigate the sensitivity of Model 1 to the disease category ‘structural’ as a perfect predictor. No significant changes to outcomes were observed. OLR Model 3 investigated effects of predictors in the consultant diagnosis group. In addition to the disease category ‘structural’, ‘trauma’ was a perfect predictor. OLR Model 4 removed perfect predictors. ‘Autoimmune’ shifts into the threshold of statistical significance (p-value = 0.03, OR 0.06, CI 95 %, 0.01–0.72) in this model, meaning the presence of an ‘autoimmune’ diagnosis makes a lower comprehensiveness score likelier. OLR Model 5 investigated the significance predictors in the CG4 diagnosis group. Both ‘vascular’ (p-value = 0.01, OR 0.13, CI 95 %, 0.03 to 0.60) and ‘autoimmune’ (p-value = 0.01, OR 0.05, CI 95 % 0.01 to 0.41) are statistically significant predictors of poorer outcomes. VIFs and Brant Test results for each of these models are displayed in Supplementary Tables 3 and 4, respectively (see Table 6). Fig. 5A compares the comprehensiveness scores for initial diagnoses across different disease categories between consultant neurologists and CG4. The blue boxes represent the scores from consultants, while the red boxes represent the scores from CG4. Disease categories include Autoimmune, Epilepsy, Infectious, Other, Structural, Trauma, and Vascular. Each box plot shows the median, interquartile range, and potential outliers for each category. This figure illustrates the variability and distribution of comprehensiveness scores for each group across various disease categories. Fig. 5B is a collinearity matrix of the predictors used in our logistic regression analysis. Predictors include Diagnosis Source, patient sex (Pt Sex), patient age (Pt Age), and disease categories such as Autoimmune, Epilepsy, Infectious, Vascular, Trauma, Structural, and Other. The color scale ranges from −1 to 1, with blue representing a positive correlation, green representing a negative correlation, and lighter shades indicating weaker correlations. Values inside the cells indicate the correlation coefficients between pairs of predictors. This figure helps to identify multicollinearity among predictors, which can impact the reliability and interpretation of regression models. Table 5Disease category variable counts.Table 5StructuralTraumaVascularEpilepsyInfectiousAutoimmuneOther644024101450 Disease category variable counts. Table 6Ordinal logistic regression models.Table 6A. Ordinal Logistic Regression (OLR) Model 1 - All PredictorsVariableValueStd. Errort valuep valueOdds Ratio2.50 %97.50 %Pt Age−0.030.02−1.890.060.970.941.00Pt Sex1−0.350.56−0.620.540.710.242.12Structural∗15.770.006.36E+070.007.05E+067.05E+067.05E+06Trauma−1.621.35−1.200.230.200.012.77Vascular−0.750.70−1.070.280.470.121.87Epilepsy∗−1.800.71−2.540.010.160.040.66Infectious−0.910.75−1.210.230.400.091.77Autoimmune∗−2.700.82−3.280.000.070.010.34Other−0.430.60−0.720.470.650.202.12Diagnosis Group∗1.300.542.430.023.681.2810.55InterceptsThresholdValueStd. Errort valuep value0|1−7.471.55−4.830.001|2−6.491.44−4.490.002|3−4.001.29−3.090.00Model MetricsMetricValueResidual Deviance123.70AIC149.69B. Ordinal Logistic Regression Model 2 - No Perfect PredictorsVariableValueStd. Errort valuep valueOdds Ratio2.50 %97.50 %Pt Age−0.020.02−1.570.120.980.951.01Pt Sex1−0.440.55−0.790.430.650.221.92Trauma−1.651.33−1.240.210.190.012.61Vascular−1.000.68−1.470.140.370.101.39Epilepsy∗−1.700.68−2.490.010.180.050.70Infectious−0.810.74−1.080.280.450.101.92Autoimmune∗−2.830.81−3.500.000.060.010.29Other−0.640.58−1.100.270.530.171.65Diagnosis Group∗1.290.532.430.023.651.2810.34InterceptsThresholdValueStd. Errort valuep value0|1−7.311.48−4.950.001|2−6.321.37−4.620.002|3−3.851.22−3.170.00Model MetricsMetricValueResidual Deviance129.81AIC149.81C. Ordinal Logistic Regression Model 3 - Consultant ModelVariableValueStd. Errort valuep valueOdds Ratio2.50 %97.50 %Pt Age0.000.03−0.110.911.000.941.06Pt Sex10.160.980.160.871.170.178.00Structural∗7.1532.470.220.831274.140.005.58E+30Trauma∗7.250.002446.660.001414.741406.551422.99Vascular0.851.320.640.522.340.1731.33Epilepsy−1.121.20−0.940.350.330.033.41Infectious0.871.620.530.592.380.1057.42Autoimmune−2.321.31−1.770.080.100.011.29Other0.961.060.900.372.600.3220.92InterceptsThresholdValueStd. Errort valuep value0|1−3.422.21−1.550.121|2−2.912.18−1.330.182|3−1.642.15−0.760.45Model MetricsMetricValueResidual Deviance48.93AIC72.93D. Ordinal Logistic Regression Model 4 - Consultant Model No Perfect PredictorsVariableValueStd. Errort valuep valueOdds Ratio2.50 %97.50 %Vascular0.421.100.380.711.520.1713.16Epilepsy−1.371.05−1.300.190.250.032.00Infectious0.821.500.540.592.260.1243.16Autoimmune−2.771.24−2.220.030.060.010.72Other0.510.940.540.591.660.2610.46InterceptsThresholdValueStd. Errort valuep value0|1−3.951.27−3.120.001|2−3.451.20−2.880.002|3−2.221.10−2.020.04Model MetricsMetricValueResidual Deviance50.81AIC66.81E. Ordinal Logistic Regression Model 5 - CG4 ModelVariableValueStd. Errort valuep valueOdds Ratio2.50 %97.50 %Trauma−2.891.64−1.760.080.060.001.38Vascular∗−2.050.79−2.600.010.130.030.60Epilepsy−1.430.88−1.640.100.240.041.33Autoimmune∗−2.981.06−2.800.010.050.010.41InterceptsThresholdValueStd. Errort valuep value1|2−5.631.23−4.580.002|3−2.200.74−2.990.00Model MetricsMetricValueResidual Deviance67.15AIC79.15 Ordinal logistic regression models. Discussion Our study provides a robust analysis of CG4’s diagnostic performance in the context of an inpatient neurology consultation service. Our study is the first, as far as we are aware, to test a LLM in this context. The results are clear that CG4 can provide comprehensive and accurate initial diagnoses and can potentially augment the diagnostic accuracy of consultant neurologists. In our analysis, consultants do provide more comprehensive diagnoses on average as evidenced by the Wilcoxon Rank Sum test CG4 (W = 1583, p = 0.02), analytical probability of a higher score if diagnostic group outcomes are sampled at random (33 % vs 12 %, with 55 % draws), and OLR Model 1, which shows an odds ratio of 3.68 (CI 95 %, 1.28 to 10.55) favoring consultant diagnoses. Our OLR models suggest that CG4, like consultants, have great accuracy in assessing diseases that are structural in nature. This, we speculate, might largely be due to the reliance on imaging reports that form the basis of diagnoses in these cases. While ‘structural’ is a perfect predictor for both diagnostic groups, one advantage that CG4 has is its ability to process information more quickly than a human. In ‘structural’ cases where both the consultants and CG4 make accurate diagnoses, CG4 might thus be able to quicken the diagnostic process. It is also important to note that while consultants did outperform CG4 overall in our methodology, without independent third-party appraisal of each case, consultants may have benefitted from anchoring or other biases. In this proof-of-concept study, consultant final diagnoses are assumed to be correct and served as the comparative standards for accuracy of initial diagnoses. If, however, and consultant final diagnosis were inaccurate, and their diagnosis were unchanged from initial to final diagnoses (potential anchoring bias), they would be rewarded in our model. Nonetheless, viewed through the lens of whether CG4 can be a contributory tool as assessed by reducing outcomes to success (≥ 50 % match) or failure (< 50 % match), CG4 performs comparably to consultants. There is a 94.1 % (95 % CI 84.1%–98.9 %) success rate in the consultant group and 96.1 % (95 % CI 86.8%–98.9 %) success rate in the CG4 group. An important result comes from evaluating the failures of each individual group. The consultant group failed in cases 13, 29, and 30, while CG4 failed in cases 56 and 57. This difference in outcomes yields insights into CG4’s potential as an adjunct diagnostic tool. Let us briefly examine these cases one by one, beginning with the cases not initially comprehensively diagnosed by CG4. Case 56 had final consultant diagnoses of “Concussion; post-traumatic headache; Vertebrobasilar insufficiency given dominant R vertebral artery.” In this case the consultant provided only one patient note, and their initial and final diagnoses were thus the same. The researcher who processed this case awarded CG4 with only one of three matches for its response of “post-traumatic sequelae” within the following: “Principal neurological diagnosis: Acute ischemic stroke; Differential diagnoses: Hemorrhagic stroke; Subdural hematoma; Cervical spine injury with radiculopathy or myelopathy; Post-traumatic sequelae; Transient ischemic attack (TIA).” Case 57 had final consultant diagnoses of “Hyponatremia causing altered mental status; Addison's pituitary dysfunction; Sepsis; Fat.” The final diagnosis of “fat” is verbatim from the consultant’s note. The researcher was unable to parse a determinant meaning of this diagnosis from the consultant’s note. This case received only one note from the consultant, and thus the consultant’s initial and final diagnoses were a 100 % match. CG4 correctly identified hyponatremia in its initial diagnoses but did not identify the other three of the physician’s final diagnoses. We’ll next turn our attention to the cases not initially comprehensively diagnosed by the consultant group. Case 13 had final consultant diagnoses of “Status epilepticus secondary to subtherapeutic anti-epileptic dosing,” and “Status epilepticus secondary to medication non-compliance.” At the researcher’s discretion, this was divided into three principal diagnoses, “G40; Z51.81; Z91.199,” corresponding to “Epilepsy and recurrent seizures; Encounter for therapeutic drug level monitoring; Patient's noncompliance with other medical treatment and regimen due to unspecified reason”. The consultant’s initial diagnosis was “Status epilepticus.” On the other hand, CG4 in its initial diagnoses suggested potential etiologies that included medication non-compliance. In case 29, consultant final diagnoses were “Lupus cerebritis with associated autoimmune neuropathy not responding to steroids or hydroxychloroquine; Multiple sclerosis.” Consultant initial diagnoses were “Chronic inflammatory demyelinating polyradiculopathy; Insidious indolent systemic etiology (ex: infectious, autoimmune; neoplastic); Neurologic disease; Focal injury.” CG4, on the other hand, identified lupus and multiple sclerosis specifically in its initial diagnoses. Finally, in case 30, consultant final diagnoses were “Toxin exposure; Metabolic derangement; Profound nutritional deficiency; Sicca.” Consultant initial diagnoses were “Chronic infectious etiology (ex TB); Multiple sclerosis.” CG4, on the other hand, provided multiple potential etiologies, including “Sjogren’s syndrome with neurological manifestations,” and “vitamin B12 deficiency neuropathy.” In this case, CG4 again succeeded in the specificity and appropriateness of its initial diagnoses. These latter three cases illustrate the potential of CG4 to suggest accurate etiologies in initial diagnoses, potentially hastening the time to definitive diagnosis and reducing patient length of stay. If CG4’s initial diagnoses were included among the consultant’s own initial diagnoses in the previous 3 cases, the result would be comprehensive in all cases. This would mean a number needed to treat (NNT) of 17 for 1 additional comprehensive case diagnosis, which effect would be significant if scaled. The results of this study suggest that CG4 can play a valuable role as a diagnostic aid in the context of an inpatient neurology consultation service. While there were no statistically significant differences in overall diagnostic comprehensiveness between CG4 and human consultants (via Fisher’s Exact Test), CG4 demonstrated a significant advantage in processing EHR-extracted data rapidly, potentially reducing diagnostic and intervention times. This time-saving capability could allow healthcare providers to focus more on complex decision-making or treat additional patients, enhancing overall efficiency. Furthermore, a detailed analysis revealed that CG4 and human physicians made different types of diagnostic errors. Notably, the study identified a NNT of 17 for CG4 to provide an additional diagnosis that would otherwise have been missed without its involvement. These findings highlight the potential complementary role of CG4 in supporting human physicians, particularly in terms of diagnostic speed and accuracy. Future research will explore the impact of integrating CG4 into real-time clinical workflows and its effects on patient outcomes in an inpatient neurology consultation service. Limitations This study has several limitations that may impact the interpretation of our findings. First, we suggest a future study to assess the accuracies of case diagnoses with respect to third-party, expert panel-derived diagnoses, which could help alleviate potential anchoring bias that may have benefited the accuracy score of the consultant group. Additionally, the retrospective nature of the study introduces inherent limitations, as we relied on pre-existing patient data from the EPIC electronic health record (EHR) system. These data were not originally collected for research purposes, and any inconsistencies or missing information could affect the accuracy and comprehensiveness of the findings. We were also limited by using an LLM subject to updates at the discretion of OpenAI. While we documented the time at which interactions with CG4 occurred, any changes to the model between then and the present might be undisclosed and proprietary to OpenAI. This introduces a challenge in replicating results exactly, as the current version of the model may differ from the one used in this study. Furthermore, our sample size, while sufficient for preliminary analysis, was relatively small. We recommend a larger sample size in future studies to improve the robustness of the modeling and ensure stronger statistical power. With respect to statistical treatment, while no independent variables violated the proportional odds assumption in our ordinal logistic regression models, the results of the Brant tests and the near statistical significance of the diagnostic source within these results suggest that multinomial logistic regression modeling might be beneficial for future analyses. Finally, while our study examined multiple diagnoses per case, future studies could benefit from hierarchical or multilevel modeling to account for the influence of multiple diagnoses within a single case on the accuracy of each individual diagnosis. This would allow for a more nuanced understanding of diagnostic performance, especially in cases with complex or multiple diagnoses. CRediT authorship contribution statement Sebastian Cano-Besquet: Writing – review & editing, Writing – original draft, Visualization, Validation, Supervision, Software, Resources, Project administration, Methodology, Investigation, Funding acquisition, Formal analysis, Data curation, Conceptualization. Tyler Rice-Canetto: Data curation. Hadi Abou-El-Hassan: Supervision, Data curation. Simon Alarcon: Writing – original draft, Investigation, Formal analysis, Data curation. Jason Zimmerman: Data curation. Leo Issagholian: Data curation. Nasser Salomon: Resources. Ivan Rojas: Resources. Joseph Dhahbi: Writing – review & editing, Supervision. Michael M. Neeki: Writing – review & editing, Supervision, Resources. Data and code availability The data that has been used is confidential. Deidentified patient data are protected and can only be accessed with approval from the Arrowhead Regional Medical Center Institutional Review Board (IRB). Supplementary Data 1 includes anonymized records of researcher interactions with CG4. All patient data within this supplementary material has been redacted. Access to Supplementary Data 1 can be granted to researchers with methodologically sound proposals that align with ethical research practices. Proposals will be evaluated on a case-by-case basis. To request access, please email Sebastian.cano-besquet@md.cusm.edu, including details of the intended use of the data. All requests must adhere to the IRB's guidelines to ensure the protection of patient privacy and data integrity. Funding This study was conducted without funding. Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. Acknowledgements Thank you to Maria Skillbred and the ARMC Epic Electronic Health Record team for providing the support necessary to complete this experiment. Footnotes Supplementary data to this article can be found online at https://doi.org/10.1016/j.heliyon.2024.e40964. Supplementary data The following is the Supplementary data to this article: Multimedia component 1 References Associated Data Supplementary Materials Multimedia component 1 Data Availability Statement The data that has been used is confidential. Deidentified patient data are protected and can only be accessed with approval from the Arrowhead Regional Medical Center Institutional Review Board (IRB). Supplementary Data 1 includes anonymized records of researcher interactions with CG4. All patient data within this supplementary material has been redacted. Access to Supplementary Data 1 can be granted to researchers with methodologically sound proposals that align with ethical research practices. Proposals will be evaluated on a case-by-case basis. To request access, please email Sebastian.cano-besquet@md.cusm.edu, including details of the intended use of the data. All requests must adhere to the IRB's guidelines to ensure the protection of patient privacy and data integrity.