A comparative study of GPT-4o and human ophthalmologists in glaucoma diagnosis Authors: Junxiu Zhang, Yao Ma, Rong Zhang, Yanhua Chen, Mengyao Xu, Su Rina, Ke Ma Published: 2024 ABSTRACT Artificial intelligence (AI), particularly large language models like GPT-4o, holds promise for enhancing diagnostic accuracy in healthcare. This study evaluates the diagnostic performance of GPT-4o compared to human ophthalmologists in glaucoma cases. A prospective, observational study was conducted at a tertiary care ophthalmology center. Twenty-six glaucoma cases, including both primary and secondary types, were selected from publicly available databases and institutional records. The cases were analyzed by GPT-4o and three ophthalmologists with varying levels of experience. The accuracy and completeness of primary and differential diagnoses were assessed using 10-point and 6-point Likert scales, respectively. Statistical analyses were performed using nonparametric methods, including the Kruskal–Wallis and Mann–Whitney U tests. GPT-4o was significantly less accurate in primary diagnosis compared to human ophthalmologists. Specifically, GPT-4o achieved a mean score of 5.500 (p < 0.001) compared to Doctor C, who had the highest score of 8.038 (p < 0.001). Completeness scores for GPT-4o 3.077 (p < 0.001) were also lower than Doctor B, who had the lowest score of 3.615 (p < 0.001) among human ophthalmologists. However, for differential diagnosis, GPT-4o (7.577) showed comparable accuracy to Doctor A (7.615) and Doctor C (7.673) (p < 0.0001) while achieving the highest completeness score (4.096), outperforming Doctor C (3.846), Doctor A (2.923), and Doctor B (2.808) (p < 0.0001). AI, including GPT-4o, is currently not an acceptable standalone method for diagnosing glaucoma due to its lower accuracy compared to human clinicians. These findings suggest that GPT-4o could serve as a valuable adjunct in clinical practice, particularly in complex cases, but should not replace human expertise, especially for initial diagnoses. Future improvements in AI models could enhance their utility in ophthalmology. Introduction The application of artificial intelligence (AI) in healthcare offers promising advancements, particularly in supporting diagnostic processes1,2. Large language models, such as GPT-4o developed by OpenAI, have demonstrated the capability to analyze complex medical data, potentially serving as valuable tools for clinicians across various specialties3,4. However, the effectiveness of these models in fields that require detailed clinical assessments, like ophthalmology, warrants thorough evaluation5. Glaucoma, a leading cause of irreversible blindness globally, presents diagnostic challenges due to its diverse subtypes and potential secondary causes6. Diagnosing glaucoma involves interpreting multiple clinical tests, including visual field assessments, optical coherence tomography (OCT), and intraocular pressure measurements7. This multifaceted approach underscores the potential for AI to play a role in supporting diagnostic accuracy by integrating and analyzing data across these tests in a consistent manner8. In this study, we compare the diagnostic performance of GPT-4o with that of ophthalmologists of varying experience levels, focusing on both primary and differential diagnoses of glaucoma. By examining cases of primary open-angle glaucoma (POAG) and secondary glaucoma, we aim to assess the potential and limitations of GPT-4o as a complementary diagnostic tool in real-world clinical settings. This research seeks to determine whether AI can enhance diagnostic workflows and provide additional insights, particularly in complex cases. Methods Study design This study employed a prospective, observational design and was conducted at the Beijing TongRen Eye Center, Capital Medical University, Beijing. Ethical approval was obtained from the Institutional Review Board (IRB), and all patient data were de-identified. Case selection A total of 26 glaucoma cases were selected as a convenience sample, without formal sample size calculation. The cases were randomly chosen from the institutional public database of Beijing Tongren Eye Center, Capital Medical University, to ensure representation of both primary and secondary glaucoma types. The selected cases included various subtypes of glaucoma, such as primary open-angle glaucoma (POAG), normal-tension glaucoma (NTG), and neovascular glaucoma (NVG). All patients had a confirmed diagnosis of glaucoma, and some cases also presented with concomitant ocular conditions, which were documented during the study. The selected cases included various subtypes of glaucoma, with the distribution as follows:Primary open-angle glaucoma (POAG): 2 casesJuvenile open-angle glaucoma (JOAG): 1 caseNormal-tension glaucoma (NTG): 1 casePrimary angle-closure glaucoma (PACG):1 casePseudo-exfoliation glaucoma (PXG): 2 casesUncontrolled intraocular pressure after glaucoma surgery: 2 casesSecondary glaucoma due toLens subluxation: 2 casesTrue microphthalmia: 1 caseIridocyclitis: 2 casesAphakia: 2 casesAxenfeld-Rieger syndrome: 2 casesIridocorneal endothelial syndrome (ICE): 3 casesChronic uveitis: 2 casesTraumatic iridodialysis with recession of the anterior chamber angle: 1 caseNeovascular glaucoma (NVG): 2 cases Primary open-angle glaucoma (POAG): 2 cases Primary open-angle glaucoma (POAG): 2 cases Juvenile open-angle glaucoma (JOAG): 1 case Juvenile open-angle glaucoma (JOAG): 1 case Normal-tension glaucoma (NTG): 1 case Normal-tension glaucoma (NTG): 1 case Primary angle-closure glaucoma (PACG):1 case Primary angle-closure glaucoma (PACG):1 case Pseudo-exfoliation glaucoma (PXG): 2 cases Pseudo-exfoliation glaucoma (PXG): 2 cases Uncontrolled intraocular pressure after glaucoma surgery: 2 cases Uncontrolled intraocular pressure after glaucoma surgery: 2 cases Secondary glaucoma due to Secondary glaucoma due to Lens subluxation: 2 cases Lens subluxation: 2 cases True microphthalmia: 1 case True microphthalmia: 1 case Iridocyclitis: 2 cases Iridocyclitis: 2 cases Aphakia: 2 cases Aphakia: 2 cases Axenfeld-Rieger syndrome: 2 cases Axenfeld-Rieger syndrome: 2 cases Iridocorneal endothelial syndrome (ICE): 3 cases Iridocorneal endothelial syndrome (ICE): 3 cases Chronic uveitis: 2 cases Chronic uveitis: 2 cases Traumatic iridodialysis with recession of the anterior chamber angle: 1 case Traumatic iridodialysis with recession of the anterior chamber angle: 1 case Neovascular glaucoma (NVG): 2 cases Neovascular glaucoma (NVG): 2 cases Data collection The clinical information provided to GPT-4 and ophthalmologists was typically derived from the most recent outpatient visit records. All cases were selected from the hospital’s inpatient database, where patients were diagnosed and underwent surgery. Prior to admission, these patients generally had multiple outpatient visits, which allowed for comprehensive, longitudinal clinical evaluations that informed their final diagnosis. For each case, a complete set of clinical information was collected, including patient demographics, medical and ocular history, presenting symptoms, and detailed examination findings. The examination data from the most recent visit included intraocular pressure (IOP) measurements, optic nerve head assessments, visual field test results, and other relevant diagnostic findings such as corneal thickness and anterior chamber depth. When available, imaging data such as fundus photographs and optical coherence tomography (OCT) scans were also included. For OCT, we specifically used optic nerve head (ONH) scans and retinal nerve fiber layer (RNFL) thickness measurements to assess glaucomatous damage and monitor disease progression. GPT-4o model GPT-4o, developed by Open AI, is a state-of-the-art large language model optimized for understanding and generating human-like text9. For this study, the clinical information of each case was input into the GPT-4o model. The input included a structured format comprising the patient’s history, symptoms, and examination findings (see in Supplementary Information). The model was then asked to provide both a primary diagnosis and a differential diagnosis list. Comparison with ophthalmologists Three ophthalmologists (Doctor A, B and C, with working experience of 2 years, 10 years and 15 years respectively) from different provinces of our country were selected to participate in the study. The ophthalmologists were recruited based on their clinical expertise in glaucoma, ensuring a representative group with a range of years of experience. The demographic and professional characteristics of the ophthalmologists were presented in the Supplementary Information (SI) section (Table S1). Each ophthalmologist independently reviewed the same set of 26 cases without access to the GPT-4o model or each other’s responses. They were each tasked with providing a primary diagnosis and a differential diagnosis list. Their responses were recorded for subsequent comparison. Likert scale definitions Diagnostic accuracy was evaluated using a 10-point Likert scale10, conducted by an independent panel of expert ophthalmologists, distinct from the ophthalmologists involved in the study. This expert panel typically consisted of three ophthalmology specialists. Scores ranging from 1 to 2 indicated extremely poor accuracy, characterized by unacceptable errors that could significantly compromise patient safety. Scores of 3–4 reflected inadequate accuracy, with potentially dangerous errors that might result in incorrect diagnosis or treatment. Scores of 5–6 suggested moderate accuracy, with some risk of misinterpretation, but not posing immediate harm. Scores of 7–8 represented good accuracy, with only minor inaccuracies that were non-harmful. Finally, scores of 9–10 denoted excellent accuracy, free from any clinically significant errors. Additionally, diagnostic completeness was assessed on a 6-point Likert scale11. Scores of 1–2 indicated that responses were significantly incomplete, failing to address major aspects of the diagnostic process or management recommendations. Scores of 3–4 were given when responses adequately covered the essential diagnostic information, addressing the core aspects of the case. Scores of 5–6 were assigned when responses were diagnostically thorough, providing a broader context and including additional relevant and detailed information. All definitions and criteria for the Likert scales were standardized and agreed upon by the expert panel prior to the study, ensuring consistency in the evaluation process. Statistical analysis The primary outcome measures were the diagnostic accuracy and completeness of the preliminary diagnoses and differential diagnoses provided by GPT-4o and the ophthalmologists. The Likert scores were assigned by a panel of expert ophthalmologists, serving as the reference standard for accuracy and completeness evaluations. Descriptive statistics, including medians, mean ranks, and interquartile ranges, were calculated to summarize the response data. Given the ordinal nature of the Likert scale and the non-normal distribution of the dataset, nonparametric methods were applied. Specifically, the Mann–Whitney U test was utilized to compare differences in accuracy and completeness between GPT-4o and each of the three ophthalmologists, with a threshold of p < 0.05 for all analyses. Two-tailed tests were conducted throughout. The Kruskal–Wallis test was used to evaluate overall differences across groups, followed by Dunn’s post-hoc tests for pairwise comparisons. For the primary outcome of diagnostic accuracy, nonparametric methods were chosen due to the Likert scale’s ordinal nature, which does not assume a normal distribution of data. Sensitivity was not calculated because the primary focus of the study was on overall diagnostic accuracy and completeness, rather than specific diagnostic test performance characteristics like sensitivity or specificity. Statistical analyses were performed using GraphPad Prism 10. Ethical considerations Institutional Review Board (IRB) approval was obtained for this study. Informed consent was obtained from all participants and/or their legal guardians before participation. As part of the ethical approval process, all patient data from the institution’s records were de-identified to ensure privacy and confidentiality. No identifiable personal information was retained, and publicly accessible data did not contain any patient identifiers. Furthermore, all methods were carried out in accordance with the relevant guidelines and regulations, including the Declaration of Helsinki. Results The accuracy and completeness of primary diagnoses were evaluated across GPT-4o and three ophthalmologists (Doctor A, Doctor B, and Doctor C) using Kruskal–Wallis tests (Table 1, Fig. 1A,B). The overall accuracy for primary diagnosis demonstrated significant differences among the groups (Kruskal–Wallis statistic = 32.78, p < 0.0001). Specifically, Doctor C had the highest accuracy mean rank (8.038), followed by Doctor A (7.577), Doctor B (6.808), and GPT-4o (5.500). Completeness of the primary diagnosis also showed significant differences (Kruskal–Wallis statistic = 22.35, p < 0.0001), with Doctor A leading in completeness (4.115), followed by Doctor C (3.846), Doctor B (3.615), and GPT-4o (3.077).Table 1The overall score for GPT4o and ophthalmologists in both primary diagnosis and differential diagnosis of glaucoma cases.MeasureTotal, No.Kruskal–Wallis statisticP valueMean score (N = 26)GPT 4oDoctor ADoctor BDoctor CPrimary diagnosisAccuracy72632.78<0.00015.5007.5776.8088.038Completeness38122.35<0.00013.0774.1153.6153.846Differential diagnosisAccuracy736.524.62<0.00017.5777.6155.4627.673Completeness355.530.57<0.00014.0962.9232.8083.846Accuracy is presented on a 10-point scale, with scores ranging from 1 (extremely poor) to 10 (excellent). Completeness is evaluated on a 6-point scale, with scores ranging from 1 (significantly incomplete) to 6 (thorough and diagnostically detailed).Fig. 1Comparion of accuracy in primary diagnosis (A), completeness in primary diagnosis (B), accuracy in differential diagnosis (C), completeness in differential diagnosis (D) between GPT4o and all three doctors. Table 1The overall score for GPT4o and ophthalmologists in both primary diagnosis and differential diagnosis of glaucoma cases.MeasureTotal, No.Kruskal–Wallis statisticP valueMean score (N = 26)GPT 4oDoctor ADoctor BDoctor CPrimary diagnosisAccuracy72632.78<0.00015.5007.5776.8088.038Completeness38122.35<0.00013.0774.1153.6153.846Differential diagnosisAccuracy736.524.62<0.00017.5777.6155.4627.673Completeness355.530.57<0.00014.0962.9232.8083.846Accuracy is presented on a 10-point scale, with scores ranging from 1 (extremely poor) to 10 (excellent). Completeness is evaluated on a 6-point scale, with scores ranging from 1 (significantly incomplete) to 6 (thorough and diagnostically detailed). The overall score for GPT4o and ophthalmologists in both primary diagnosis and differential diagnosis of glaucoma cases. Accuracy is presented on a 10-point scale, with scores ranging from 1 (extremely poor) to 10 (excellent). Completeness is evaluated on a 6-point scale, with scores ranging from 1 (significantly incomplete) to 6 (thorough and diagnostically detailed). Comparion of accuracy in primary diagnosis (A), completeness in primary diagnosis (B), accuracy in differential diagnosis (C), completeness in differential diagnosis (D) between GPT4o and all three doctors. For differential diagnosis, significant differences were observed in both accuracy and completeness across the groups (Table 1, Fig. 1C,D). The accuracy mean ranks for differential diagnosis (Kruskal–Wallis statistic = 24.62, p < 0.0001) indicated Doctor C had the highest accuracy (7.673), closely followed by Doctor A (7.615), GPT-4o (7.577), and Doctor B (5.462). Completeness in differential diagnosis showed the greatest significant variation (Kruskal–Wallis statistic = 30.57, p < 0.0001), with GPT-4o having the highest mean rank (4.096), outperforming Doctor C (3.846), Doctor A (2.923), and Doctor B (2.808). Pairwise comparisons using the Mann–Whitney U test further detailed the differences between GPT-4o and each ophthalmologist (Table 2). For primary diagnosis accuracy, GPT-4o was significantly less accurate than all doctors, with the lowest accuracy when compared with Doctor C (Mann–Whitney U = 71, p < 0.0001). Completeness followed a similar trend, where GPT-4o was outperformed by all doctors, particularly by Doctor A (Mann–Whitney U = 117.5, p < 0.0001).Table 2Comparison of GPT4o vs ophthalmologists on accuracy and completeness in glaucoma cases primary diagnosis.ComparisonMean ranksSEMann–Whitney U statisticP valueAccuracy ratingGPT 4o vs Doctor A18.13 vs 34.870.3385 vs 0.2727120.5<0.0001GPT 4o vs Doctor B20.85 vs 32.150.3385 vs 0.24821910.0056GPT 4o vs Doctor C16.23 vs 36.770.3385 vs 0.188071<0.0001Completeness ratingGPT 4o vs Doctor A18.02 vs 34.980.1680 vs 0.1187117.5<0.0001GPT 4o vs Doctor B21.85 vs 31.150.1680 vs0.15012170.0206GPT 4o vs Doctor C19.77 vs 33.230.1680 vs 0.091021630.0006 Table 2Comparison of GPT4o vs ophthalmologists on accuracy and completeness in glaucoma cases primary diagnosis.ComparisonMean ranksSEMann–Whitney U statisticP valueAccuracy ratingGPT 4o vs Doctor A18.13 vs 34.870.3385 vs 0.2727120.5<0.0001GPT 4o vs Doctor B20.85 vs 32.150.3385 vs 0.24821910.0056GPT 4o vs Doctor C16.23 vs 36.770.3385 vs 0.188071<0.0001Completeness ratingGPT 4o vs Doctor A18.02 vs 34.980.1680 vs 0.1187117.5<0.0001GPT 4o vs Doctor B21.85 vs 31.150.1680 vs0.15012170.0206GPT 4o vs Doctor C19.77 vs 33.230.1680 vs 0.091021630.0006 Comparison of GPT4o vs ophthalmologists on accuracy and completeness in glaucoma cases primary diagnosis. In the differential diagnosis (Table 3), GPT-4o was comparable to Doctor A and Doctor C in accuracy (Mann–Whitney U = 330.5, p = 0.8599; Mann–Whitney U = 326.5, p = 0.8169, respectively), but significantly more accurate than Doctor B (Mann–Whitney U = 134.5, p < 0.0001). For completeness, GPT-4o outperformed both Doctor A and Doctor B (Mann–Whitney U = 128, p < 0.0001; Mann–Whitney U = 108, p < 0.0001), while the comparison with Doctor C was not statistically significant (Mann–Whitney U = 247, p = 0.0793).Table 3Comparison of GPT4o vs ophthalmologists on accuracy and completeness in glaucoma cases differential diagnosis.ComparisonMean ranksSEMann–Whitney U statisticP valueAccuracy ratingGPT 4o vs Doctor A26.21 vs 26.790.2429 vs 0.3555330.50.8599GPT 4o vs Doctor B34.33 vs 18.670.2429 vs 0.4196134.5<0.0001GPT 4o vs Doctor C26.94 vs 26.060.2429 vs 0.2269326.50.8169Completeness ratingGPT 4o vs Doctor A34.58 vs 18.420.1144 vs 0.2248128<0.0001GPT 4o vs Doctor B35.25 vs 17.650.1144 vs 0.2114108<0.0001GPT 4o vs Doctor C30.00 vs 23.000.1144 vs 0.91022470.0793 Table 3Comparison of GPT4o vs ophthalmologists on accuracy and completeness in glaucoma cases differential diagnosis.ComparisonMean ranksSEMann–Whitney U statisticP valueAccuracy ratingGPT 4o vs Doctor A26.21 vs 26.790.2429 vs 0.3555330.50.8599GPT 4o vs Doctor B34.33 vs 18.670.2429 vs 0.4196134.5<0.0001GPT 4o vs Doctor C26.94 vs 26.060.2429 vs 0.2269326.50.8169Completeness ratingGPT 4o vs Doctor A34.58 vs 18.420.1144 vs 0.2248128<0.0001GPT 4o vs Doctor B35.25 vs 17.650.1144 vs 0.2114108<0.0001GPT 4o vs Doctor C30.00 vs 23.000.1144 vs 0.91022470.0793 Comparison of GPT4o vs ophthalmologists on accuracy and completeness in glaucoma cases differential diagnosis. Discussion The present study aimed to evaluate the performance of GPT-4o compared to human ophthalmologists in both primary and differential diagnoses of glaucoma. While the AI demonstrated some potential, particularly in generating comprehensive differential diagnoses, it ultimately fell short in the area of primary diagnosis accuracy and completeness. These findings highlight the current limitations of AI, such as GPT-4o, in performing as a standalone diagnostic tool for glaucoma. In primary diagnosis, which requires the prioritization of information and the application of cognitive reasoning, GPT-4o performed significantly worse than the three human ophthalmologists, with Doctor C showing the highest accuracy, followed by Doctors A and B. This is in line with prior study that while ChatGPT’s accuracy improved after updates, particularly in areas like general medicine and cornea12,13, it still struggled in highly specialized fields such as neuro-ophthalmology and ocular pathology. However, our study contrasts with other study14 where large language models (LLMs) demonstrated comparable or even superior performance to trained specialists, especially in terms of diagnostic accuracy and completeness. These discrepancies may be attributed to differences in study design, case complexity, or dataset characteristics. For example, our study included a broad spectrum of glaucoma subtypes, many of which are less common or more complex, potentially challenging the model’s diagnostic capabilities. Moreover, our study demonstrates that AI does not yet possess the refined diagnostic reasoning required for primary diagnosis. Thus, GPT-4o is not acceptable as a solitary method for diagnosing glaucoma (since it could be used as a complementary method). Conversely, GPT-4o demonstrated better performance in differential diagnosis, particularly regarding completeness. This is likely attributable to the AI’s ability to reference a large database and compile a broad list of potential diagnoses based on patterns recognized in the input data15. However, it is important to note that differential diagnosis is primarily a cataloging process, not one that requires significant prioritization of diagnostic possibilities. As such, GPT-4o’s exhaustive list may be less useful in practice since a long list of possibilities could overwhelm clinicians and increase the risk of cognitive bias, potentially leading to misdiagnosis. Therefore, while GPT-4o might assist in generating a broad range of differential diagnoses, it cannot replace the critical thinking and judgment required for selecting and prioritizing the most likely diagnoses. A critical observation from this study, which must be underscored, is that the human ophthalmologists also exhibited less-than-ideal diagnostic accuracy, even with their specialized training and clinical experience. This calls attention to a more pressing issue: the overall accuracy of glaucoma diagnoses is concerningly low, even among trained ophthalmologists. This finding should prompt clinicians to consistently re-evaluate their diagnoses and openly communicate the inherent uncertainties to their patients. The acknowledgment that misdiagnosis is a possibility should factor into patient management to mitigate potential harm. Our study’s limitations should be acknowledged, especially regarding the generalizability of findings. The sample size of n = 26 may limit the robustness of the conclusions and expanding the sample could yield more comprehensive insights. Moreover, the sample selection in our study was focused on covering as many glaucoma types as possible, rather than reflecting the actual prevalence of each type as seen in clinical practice. This approach did not fully account for the epidemiological characteristics of glaucoma subtypes, such as the higher prevalence of POAG and NTG. Recognizing this limitation, future research with a larger, more representative sample that aligns with real-world prevalence rates would improve the applicability of these findings across diverse clinical settings. Future research should focus on improving the primary diagnosis capabilities of AI systems like GPT-4o through enhanced training datasets, incorporating clinician feedback, and exploring more sophisticated reasoning algorithms16. For instance, a future study could include a control group and explore GPT-4o’s efficacy in primary care settings compared to tertiary care, where the AI could be compared with optometrists or general practitioners rather than ophthalmologists. Additionally, assessing its real-time utility in clinical workflows would be valuable to understand the clinical feasibility and impact on diagnostic processes, especially in non-specialist settings. In conclusion, GPT-4o, like other AI systems, shows promise in differential diagnosis by broadening the scope of potential diagnoses, but it remains inadequate for primary glaucoma diagnosis. This study further emphasizes that current diagnostic methods, both human and AI-driven, are imperfect, underscoring the need for continuous reevaluation and patient communication in clinical practice. Future advancements in AI may eventually enhance diagnostic accuracy, but until then, it should be viewed as a complementary tool, not a replacement for human expertise. Supplementary Information Supplementary Information. Supplementary Information.