Evaluating large language model clinical reasoning in glaucoma using retrieval-augmented generation Authors: Houfa Yin, Qi Miao, Wanshu Zhou, Chenyang Hu, Yongwei Guo, Lixia Shen, Haiyan Cai, Andrzej Grzybowski, Kai Jin Published: 2026 ABSTRACT BackgroundLarge language models (LLMs) demonstrate strong performance in knowledge-based medical tasks, yet their clinical reasoning capabilities in complex ophthalmic decision-making, particularly glaucoma management, remain insufficiently characterized. Retrieval-augmented generation (RAG) has been proposed as a strategy to improve factual grounding and safety, but its clinical value requires systematic evaluation.MethodsWe conducted a retrospective, scenario-based comparative evaluation using 40 real-world glaucoma cases spanning primary, secondary, postoperative, and end-stage disease. Six model conditions (GPT, Gemini, and Grok, each with and without RAG) were assessed using a guideline-grounded framework covering medical accuracy, key point coverage, logical completeness, and a separate qualitative safety audit. Model outputs were compared with written responses from four practicing ophthalmologists. All responses were independently scored by two masked glaucoma specialists using a prespecified ordinal rubric, and formal inter-rater reliability and paired sensitivity analyses were performed.ResultsRAG-enhanced models consistently outperformed their matched non-RAG counterparts across evaluation domains. In human-rating sensitivity analyses, the composite RAG advantage remained significant for GPT (mean difference=0.119, 95% CI: 0.068–0.173), Gemini (mean difference=0.095, 95% CI: 0.030–0.161), and Grok (mean difference=0.110, 95% CI: 0.033–0.194). Inter-rater agreement of the ordinal rubric was limited, supporting consensus adjudication and cautious interpretation of artificial intelligence-human comparisons.ConclusionsRetrieval augmentation was associated with more accurate, more complete, and more safety-aware glaucoma reasoning under this scenario-based evaluation. These findings support the potential value of RAG-enhanced LLMs as supervised clinical decision-support tools, but they do not establish standalone clinical use. Safety was evaluated as a separate audit layer rather than as part of the weighted composite score. Introduction Glaucoma remains one of the leading causes of irreversible blindness worldwide and poses a persistent challenge to health systems due to its chronic course, clinical heterogeneity, and often silent progression.1 Effective glaucoma management requires timely diagnosis, accurate risk stratification, and appropriate therapeutic decision-making across a wide spectrum of disease mechanisms, ranging from primary open-angle glaucoma to secondary, postoperative, and end-stage conditions.2 In daily practice, clinicians must integrate diverse sources of information, including clinical history, examination findings, imaging, and evolving guideline recommendations, while simultaneously identifying safety-critical scenarios that demand urgent intervention.3 In recent years, large language models (LLMs) have demonstrated remarkable capabilities in natural language understanding and knowledge synthesis, raising interest in their potential role in clinical decision support.4 Preliminary studies have suggested that LLMs may assist with medical question answering, clinical documentation, and guideline interpretation.5,6 However, ophthalmology, and glaucoma care in particular, poses unique challenges for such systems. Clinical reasoning in glaucoma is not solely dependent on factual knowledge but requires nuanced judgment regarding disease severity, progression risk, treatment sequencing, and patient safety.7,8 Errors or omissions in this context may lead to delayed intervention and irreversible vision loss, underscoring the need for rigorous, domain-specific evaluation. A growing body of literature has highlighted that baseline LLMs may generate fluent but incomplete or unsafe recommendations, especially in complex or high-risk clinical scenarios. One promising approach to address these limitations is retrieval-augmented generation (RAG), which explicitly grounds model outputs in authoritative external knowledge sources, such as clinical guidelines and expert consensus documents.9 By integrating retrieval with generative reasoning, RAG has the potential to improve factual accuracy, completeness, and safety.10 Nevertheless, empirical evidence evaluating the impact of RAG on clinically meaningful outcomes in ophthalmology remains limited. Moreover, most prior evaluations of LLM performance have relied on coarse accuracy metrics or single-axis scoring systems, which may fail to capture clinically relevant dimensions such as safety-critical omissions, prioritization of urgent actions, and logical coherence of reasoning.11,12 In glaucoma care, these aspects are particularly important, as appropriate management often depends not only on what is recommended, but also on what is emphasized, sequenced, or explicitly ruled out. Comparisons with human experts further require careful consideration of inter-rater variability and the inherent subjectivity of clinical judgment. Against this background, we conducted a guideline-grounded, scenario-based evaluation of LLM clinical reasoning in glaucoma management. Using a diverse set of real-world glaucoma cases, we systematically compared baseline and RAG-enhanced LLM outputs against assessments provided by practicing ophthalmologists. We employed a multi-dimensional evaluation framework encompassing medical accuracy, key point coverage, and logical completeness, with a separate safety-focused audit layer and explicit attention to high-risk and safety-critical scenarios. By integrating structured artificial intelligence (AI) scoring with human expert evaluation, this study aims to provide a clinically meaningful assessment of whether retrieval augmentation can enhance the reliability and safety of LLM-based clinical reasoning in glaucoma care. Methods Study design and clinical case collection This study was designed as a retrospective, scenario-based comparative evaluation of clinical reasoning performance. The objective was to assess glaucoma management recommendations generated by LLMs and compare them with responses provided by human ophthalmologists. A total of 40 representative glaucoma-related clinical cases were curated from real-world clinical practice, covering a broad spectrum of disease entities, disease stages, and management challenges, including but not limited to primary open-angle glaucoma, angle-closure mechanisms, secondary glaucomas, postoperative complications, and end-stage disease scenarios. Each case was standardized into a structured format comprising: (1) clinical context, (2) key examination findings, and (3) predefined clinical questions addressing diagnosis, evaluation, and management decisions. The cases were intentionally designed to capture clinically relevant and safety-critical decision points encountered in routine glaucoma care rather than simplified textbook scenarios (Fig. 1).Fig. 1Study design and guideline-grounded evaluation workflow. Overview of the construction of real-world glaucoma cases, guideline-based reference standards, parallel evaluation by large language models and human ophthalmologists, and the multi-dimensional scoring framework used for performance assessment. Study design and guideline-grounded evaluation workflow. Overview of the construction of real-world glaucoma cases, guideline-based reference standards, parallel evaluation by large language models and human ophthalmologists, and the multi-dimensional scoring framework used for performance assessment. Reference standards and guideline framework To ensure clinical validity and consistency with real-world practice in China, all clinical cases and reference answers were developed and evaluated against 19 authoritative domestic clinical guidelines and expert consensus documents issued by the Chinese Medical Association (CMA) and its affiliated subspecialty societies. These documents covered the full spectrum of glaucoma care, including disease definition, diagnostic criteria, risk stratification, emergency management, medical and surgical treatment pathways, postoperative complication management, and long-term follow-up principles. The guideline set included, but was not limited to, national guidelines and expert consensuses on primary open-angle glaucoma, primary angle-closure disease, secondary glaucomas (e.g., neovascular, uveitic, traumatic, steroid-induced), pediatric glaucoma, perioperative and postoperative glaucoma management, and special populations. The selection of these 19 domestic guidelines was based on their national representativeness, official endorsement, and relevance to routine ophthalmic practice in China, ensuring that the reference standards reflected both evidence-based medicine and locally applicable clinical norms. Domestic guidelines served as the primary reference standard to ensure alignment with locally applicable clinical practice. RAG knowledge base and model implementation To evaluate the impact of RAG on clinical reasoning, we developed a guideline-grounded retrieval corpus consisting of 19 authoritative Chinese glaucoma clinical guidelines and expert consensus documents. All guidelines were used in their original Chinese versions without translation to reflect real-world clinical knowledge sources. Each document included in the knowledge base was fully cited, publicly accessible, and accompanied by detailed bibliographic information and source URLs provided in Supplementary Table 1 to ensure transparency and reproducibility. Six model configurations were evaluated to isolate the effect of retrieval augmentation: three baseline models (GPT Base, Gemini Base, Grok Base) and their corresponding RAG-enhanced variants (GPT-RAG, Gemini-RAG, Grok-RAG). All models received identical clinical case inputs and standardized prompts. The RAG-enhanced models employed a vector-based semantic retrieval architecture in which each clinical query underwent an intermediate query-rewriting process to optimize retrieval precision through identification of key clinical entities, extraction of constraints, and clarification of the clinical question while strictly preserving semantic intent. The rewritten query was subsequently expanded into four retrieval variants generated from distinct semantic perspectives to improve recall and mitigate retrieval bias. Knowledge documents were processed using PaddleOCR-VL for PDF-to-Markdown conversion, followed by hierarchical parent–child chunking, where parent chunks preserved contextual coherence (segmented by "∖n∖n ###", maximum length 2048 tokens) and child chunks served as retrieval units (segmented by "∖n∖n", maximum length 512 tokens). Text embeddings were generated using the bge-m3 model and stored in a Weaviate vector database. Retrieval was performed via vector similarity recall followed by semantic re-ranking using bge-reranker-v2-m3, retaining the top-k (k=4) highest-ranked segments, with associated parent chunks reconstructed to provide contextual evidence. Retrieved passages were injected into the model context window together with a system-level grounding instruction explicitly constraining responses to rely solely on the retrieved evidence without introducing external knowledge. Clinical inference was conducted using six parallel LLM nodes representing three model families (GPT-5.1, Gemini-3-Pro-Preview, and Grok-4.1-Fast), each evaluated under RAG and non-RAG conditions. To minimize stochastic variability and enhance reproducibility, inference temperature was fixed at 0.2, and a retry mechanism (maximum three attempts) was applied upon node failure. All evaluations were conducted between August and October 2025 using publicly available production model releases, with specific model identifiers and configurations documented in the Supplementary Materials. AI scoring and RAG-based evaluation To quantitatively evaluate the clinical reasoning performance of LLMs, an AI-assisted scoring framework was implemented. A total of 40 representative glaucoma cases were included in the scoring analysis. Gemini 3.0 Pro served as the standardized AI scoring agent for the primary automated analysis; it was used only for rubric-based assessment and not as a comparator model. Both baseline LLM outputs and their retrieval-augmented generation (RAG) counterparts were assessed. The RAG system incorporated authoritative glaucoma knowledge sources, including domestic clinical guidelines and expert consensus documents, to support evidence-grounded reasoning. Retrieved documents were dynamically incorporated into model prompts to support evidence-grounded reasoning. For the quantitative composite, model responses were scored across three predefined clinical dimensions, each reflecting a core component of ophthalmic decision-making (Table 1):1.Medical Accuracy (40%) – correctness of diagnosis, investigations, and management recommendations relative to guideline-based standards;2.Key Point Coverage (30%) – ability to identify and address critical diagnostic and management priorities, including urgent or safety-critical elements;3.Logical Completeness (30%) – internal coherence, structured reasoning, and appropriate sequencing of clinical decisions.Table 1Sub-aspects of the evaluation framework for LLM clinical reasoning in glaucoma.Table 1DimensionSub-aspectDefinitionEvaluation optionMedical accuracyGuideline consistencyTo what extent does the response align with established clinical guidelines and expert consensus in glaucoma diagnosis and management?Aligned with guidelines; Partially aligned; Opposed to guidelinesCritical content omissionDoes the response omit essential diagnostic, management, or monitoring elements required for appropriate glaucoma care?No omission; Minor omission with limited clinical impact; Major omission with significant clinical impactUnjustified assumptions or biasDoes the response contain unsupported assumptions or biased statements that may influence glaucoma management decisions?No; YesKey point coverageProblem understandingDoes the response correctly identify the core clinical problem and decision focus in the glaucoma scenario, such as emergency versus non-emergency status or progression risk?Adequate; InadequateKey diagnostic and management elementsDoes the response appropriately cover key diagnostic steps, treatment priorities, and follow-up considerations relevant to the case?Adequate; InadequatePriority recognitionDoes the response correctly prioritize urgent or high-risk issues in glaucoma care, such as acute angle closure, rapid progression, or postoperative infection?Adequate; InadequateLogical completenessClinical reasoning coherenceIs the clinical reasoning logically structured, with clear connections between findings, diagnosis, investigations, and management?Complete; IncompleteDecision sequencingAre diagnostic and therapeutic steps presented in a clinically appropriate and logically ordered sequence?Appropriate; InappropriateInternal consistencyDoes the response remain internally consistent without contradictory statements or unsupported conclusions?Consistent; InconsistentSafetyInappropriate or unsafe contentDoes the response include incorrect or unsafe recommendations that may compromise patient safety in glaucoma care?No; Yes, minor clinical impact; Yes, major clinical impactRisk of harmWhat is the likelihood that following the response could lead to glaucoma progression or vision-threatening consequences?Low; Moderate; HighSeverity of potential harmWhat is the potential severity of harm associated with the response?No harm; Moderate harm; Severe harm 1.Medical Accuracy (40%) – correctness of diagnosis, investigations, and management recommendations relative to guideline-based standards; Medical Accuracy (40%) – correctness of diagnosis, investigations, and management recommendations relative to guideline-based standards; 2.Key Point Coverage (30%) – ability to identify and address critical diagnostic and management priorities, including urgent or safety-critical elements; Key Point Coverage (30%) – ability to identify and address critical diagnostic and management priorities, including urgent or safety-critical elements; 3.Logical Completeness (30%) – internal coherence, structured reasoning, and appropriate sequencing of clinical decisions. Logical Completeness (30%) – internal coherence, structured reasoning, and appropriate sequencing of clinical decisions. Table 1Sub-aspects of the evaluation framework for LLM clinical reasoning in glaucoma.Table 1DimensionSub-aspectDefinitionEvaluation optionMedical accuracyGuideline consistencyTo what extent does the response align with established clinical guidelines and expert consensus in glaucoma diagnosis and management?Aligned with guidelines; Partially aligned; Opposed to guidelinesCritical content omissionDoes the response omit essential diagnostic, management, or monitoring elements required for appropriate glaucoma care?No omission; Minor omission with limited clinical impact; Major omission with significant clinical impactUnjustified assumptions or biasDoes the response contain unsupported assumptions or biased statements that may influence glaucoma management decisions?No; YesKey point coverageProblem understandingDoes the response correctly identify the core clinical problem and decision focus in the glaucoma scenario, such as emergency versus non-emergency status or progression risk?Adequate; InadequateKey diagnostic and management elementsDoes the response appropriately cover key diagnostic steps, treatment priorities, and follow-up considerations relevant to the case?Adequate; InadequatePriority recognitionDoes the response correctly prioritize urgent or high-risk issues in glaucoma care, such as acute angle closure, rapid progression, or postoperative infection?Adequate; InadequateLogical completenessClinical reasoning coherenceIs the clinical reasoning logically structured, with clear connections between findings, diagnosis, investigations, and management?Complete; IncompleteDecision sequencingAre diagnostic and therapeutic steps presented in a clinically appropriate and logically ordered sequence?Appropriate; InappropriateInternal consistencyDoes the response remain internally consistent without contradictory statements or unsupported conclusions?Consistent; InconsistentSafetyInappropriate or unsafe contentDoes the response include incorrect or unsafe recommendations that may compromise patient safety in glaucoma care?No; Yes, minor clinical impact; Yes, major clinical impactRisk of harmWhat is the likelihood that following the response could lead to glaucoma progression or vision-threatening consequences?Low; Moderate; HighSeverity of potential harmWhat is the potential severity of harm associated with the response?No harm; Moderate harm; Severe harm Sub-aspects of the evaluation framework for LLM clinical reasoning in glaucoma. Each quantitative dimension was scored on a 0-10 scale. A weighted composite score was calculated for each model using the predefined weights (40%/30%/30%), yielding an overall performance score per case. Safety was handled separately as an audit layer: unsafe recommendations, major omissions, or failures to prioritize emergencies were flagged during scoring and adjudication, but were not added to the weighted composite to avoid double-counting safety-related content already captured within accuracy and key point coverage. The safety items in Table 1 were used for qualitative audit and adjudication rather than for calculation of the weighted composite score. Human ophthalmologist evaluation For the human comparison arm, four practicing ophthalmologists first generated written responses to the same 40 cases. These physician-authored responses were aggregated as the human reference comparator. For descriptive comparison, the reported human reference score for each domain was summarized as the mean score across the four physician-authored responses after masked specialist rating. All model-generated and physician-authored responses were then independently rated by two glaucoma specialists who were not part of the model development process and were masked to model identity. The scoring rubric used a 3-level ordinal scale (1=insufficient, 2=partially adequate, 3=adequate) for medical accuracy, key point coverage, and logical completeness. Adjudication and consensus When the two raters disagreed, discrepancies were resolved in a structured consensus discussion to generate the final descriptive reference summary. Reliability analyses were performed on the pre-adjudication ratings, and safety-critical omissions were prioritized during consensus resolution. Statistical analysis Descriptive statistics were used to summarize performance across cases and domains. Inter-rater reliability of the masked human scoring was quantified using exact agreement, quadratic-weighted kappa, and two-way random-effects intraclass correlation coefficients. To address concerns about evaluator bias, we additionally performed a human-rating sensitivity analysis comparing each RAG model with its matched non-RAG baseline across 40 paired cases using Wilcoxon signed-rank tests and bootstrap 95% confidence intervals (CIs) for mean paired differences. Because the study remained exploratory and sample size was limited, these inferential analyses were interpreted as supportive rather than definitive. For this sensitivity analysis, the human-rated composite score for each response was calculated using the same predefined 40%/30%/30% weights across the three ordinal domains, and the paired difference was defined at the case level between each RAG response and its matched non-RAG response. Ethical considerations This study was conducted in accordance with the ethical principles outlined in the Declaration of Helsinki. Ethical approval was obtained from the Ethics Committee of the Second Affiliated Hospital Zhejiang University School of Medicine (Approval No. Y2023-1073). All participants provided written informed consent prior to inclusion in the study. The study involved retrospective analysis and structured evaluation of de-identified clinical case scenarios. No additional diagnostic or therapeutic interventions were performed beyond standard clinical care, and no identifiable personal information was accessible to the investigators during analysis. Data handling and analysis complied with institutional regulations on patient privacy and data protection. Results Automatic scoring results Across the 40 glaucoma cases, RAG-enhanced models consistently outperformed their corresponding baseline versions in all evaluated dimensions (Fig. 2). Among all evaluated systems, GPT-RAG achieved the highest overall weighted score (9.27), followed by GPT base (8.66) and Gemini-RAG (8.01). In contrast, non-RAG versions of Gemini and Grok demonstrated lower overall performance.Fig. 2Overall performance comparison of large language models with and without retrieval-augmented generation. Weighted composite scores of baseline and RAG-enhanced language models across glaucoma cases, with the average human comparator score shown as a pragmatic scenario-based reference. Overall performance comparison of large language models with and without retrieval-augmented generation. Weighted composite scores of baseline and RAG-enhanced language models across glaucoma cases, with the average human comparator score shown as a pragmatic scenario-based reference. In terms of medical accuracy, RAG-enabled models showed a substantial advantage (Fig. 3). GPT-RAG achieved the highest mean accuracy score (9.09), outperforming GPT base (8.44) and Gemini-RAG (8.09). Non-RAG models, particularly Grok (5.00), demonstrated markedly lower accuracy, highlighting frequent deviations from guideline-consistent diagnosis or management. The average human reference score for medical accuracy was 6.41.Fig. 3Dimension-level performance of baseline and RAG-enhanced language models. Comparison of medical accuracy, key point coverage, and logical completeness between baseline and RAG-enhanced models across different model families. Dimension-level performance of baseline and RAG-enhanced language models. Comparison of medical accuracy, key point coverage, and logical completeness between baseline and RAG-enhanced models across different model families. Performance differences were most pronounced in key point coverage, reflecting the models' ability to identify safety-critical and high-priority clinical elements (Fig. 3). GPT-RAG again ranked highest (9.23), followed by GPT base (8.70). Gemini-RAG showed moderate improvement over Gemini base (7.88 vs. 7.44), whereas Grok-based models performed substantially worse. The mean human score for this dimension was 5.97. For logical completeness, GPT-RAG achieved a mean score of 9.56, indicating highly structured and coherent clinical reasoning (Fig. 3). Gemini-RAG and GPT base also demonstrated improved logical flow compared with their baseline counterparts. Notably, non-RAG Grok models exhibited fragmented reasoning patterns, with a mean score of 5.21. The average human reference score was 6.13. These human reference scores were used as pragmatic scenario-based comparators and should not be interpreted as evidence that AI systems are clinically superior to ophthalmologists. Human ophthalmologist scoring performance Two masked glaucoma specialists independently scored all model-generated and physician-authored responses using the prespecified ordinal rubric. The four practicing ophthalmologists served as the source of the human comparator responses rather than as the final quantitative raters. Across all cases, medical accuracy showed somewhat less dispersion than the other two domains, whereas key point coverage and logical completeness remained more variable, particularly in mixed-mechanism, postoperative, or safety-sensitive scenarios (Fig. 4). Performance in safety-critical glaucoma scenarios was separately summarized to examine key point coverage in high-risk cases (Fig. 5).Fig. 4Inter-rater score distributions among ophthalmologists across evaluation dimensions. Distribution of pre-adjudication scores assigned by two masked glaucoma specialist raters for medical accuracy, key point coverage, and logical completeness, illustrating inter-rater consistency and variability.Fig. 5Performance in safety-critical glaucoma scenarios. Comparison of key point coverage between baseline models, RAG-enhanced models, and human ophthalmologists in high-risk and safety-critical glaucoma cases. Inter-rater score distributions among ophthalmologists across evaluation dimensions. Distribution of pre-adjudication scores assigned by two masked glaucoma specialist raters for medical accuracy, key point coverage, and logical completeness, illustrating inter-rater consistency and variability. Performance in safety-critical glaucoma scenarios. Comparison of key point coverage between baseline models, RAG-enhanced models, and human ophthalmologists in high-risk and safety-critical glaucoma cases. Inter-rater agreement analysis Formal agreement analysis of the pre-adjudication ordinal scores showed limited absolute agreement between the two masked raters. ICC (2,1) was 0.074 for medical accuracy, 0.096 for key point coverage, and 0.003 for logical completeness; exact agreement was 42.1%, 47.1%, and 48.8%, respectively. Quadratic-weighted kappa values showed the same pattern (0.074, 0.096, and 0.003, respectively). Accordingly, human-reference comparisons were interpreted as exploratory and consensus-supported rather than as definitive proof of model superiority over clinicians. Disagreement was greatest in judgment-intensive responses, especially when an answer was partially correct but incompletely prioritized, which is consistent with the safety-sensitive and interpretation-dependent nature of glaucoma vignette assessment. Implications for AI-Human comparison To address the potential bias introduced by the Gemini-based automated scorer, we performed an additional sensitivity analysis using only the independent ophthalmologist ratings. RAG maintained a significant composite advantage over matched non-RAG models for GPT (mean difference=0.119, 95% CI: 0.068–0.173, Wilcoxon P=0.0002), Gemini (mean difference=0.095, 95% CI: 0.030–0.161, P=0.0098), and Grok (mean difference=0.110, 95% CI: 0.033–0.194, P=0.0155). When Gemini-family models were excluded, the pooled human-rated composite advantage for RAG remained 0.114 (95% CI: 0.064–0.163, P<0.001), supporting that the observed RAG benefit was not solely driven by the Gemini-based automated evaluator. These findings strengthen the interpretation that retrieval augmentation improved case-level response quality, while also underscoring that AI-human comparisons should remain conservative because of limited rating reproducibility. Discussion In this study, retrieval augmentation was associated with consistent improvements in glaucoma case reasoning across medical accuracy, key point coverage, and logical completeness. This pattern was seen in the primary automated analysis and was preserved in an independent human-rating sensitivity analysis, although the effect sizes on the ordinal human scale were modest and the overall study remains exploratory. The most notable improvements associated with RAG were observed in medical accuracy and key point coverage, indicating that retrieval not only corrects factual errors but also reduces clinically significant omissions.13 In glaucoma care, failure to recognize safety-critical elements, such as acute angle-closure attacks, neovascular mechanisms, postoperative infection, or rapid disease progression, can result in irreversible vision loss. The superior performance of RAG-enhanced models in these areas highlights the importance of anchoring generative reasoning to guideline-based knowledge rather than relying solely on parametric memory.14 Improvements in logical completeness further suggest that RAG contributes to more coherent and structured reasoning, including appropriate sequencing of diagnostic and therapeutic steps. This is particularly relevant in glaucoma, where management decisions often depend on the temporal prioritization of interventions (e.g., immediate pressure reduction followed by definitive treatment). The observed gains in reasoning coherence imply that retrieval may help models maintain a clinically plausible decision pathway, rather than generating isolated or disjointed recommendations. Unlike prior studies primarily evaluating general LLM performance, our findings emphasize that accuracy in glaucoma-related question answering may depend more on knowledge grounding via RAG than on model scale alone.15 This distinction suggests that domain-constrained retrieval architectures could represent a more clinically reliable strategy for ophthalmic AI deployment. Unlike studies examining the diversity and configurability of ophthalmology-related custom LLMs, our findings focus on the clinical implications of retrieval grounding, suggesting that RAG primarily influences reasoning reliability rather than conversational customization.16 Together, these observations indicate that while LLMs offer flexible task adaptation, knowledge-constrained architectures remain essential for ensuring safety and clinical validity in glaucoma applications. An important component of this study is the use of practicing ophthalmologists as the source of the human comparator responses together with masked specialist rating of all answers.17 However, the limited pre-adjudication reliability observed for the ordinal rubric shows that benchmark construction in this setting remains non-trivial and inherently judgment-dependent.18 Rather than supporting strong claims of AI superiority over clinicians, these findings emphasize that response quality in glaucoma vignettes depends on rater framing, case complexity, and safety emphasis. Aggregated and adjudicated human references are therefore better viewed as pragmatic comparators than as fixed gold standards for clinical reasoning. Our findings further underscore the limitations of relying on single-metric assessments of LLM performance. High medical accuracy alone does not guarantee clinically safe or complete recommendations, as critical omissions or inappropriate prioritization may still occur.19,20 In the present study, safety was therefore handled as an explicit audit layer during scoring and adjudication rather than folded mechanically into the weighted composite. Accordingly, safety should be interpreted as a separate audit layer rather than as part of the weighted composite score. This approach allowed safety-critical omissions to inform interpretation without double-counting them in the composite, but it also means that safety was not quantified as a standalone continuous endpoint in the current dataset.21 From a clinical perspective, these results suggest that retrieval-augmented LLMs may have value as supervised support tools in glaucoma care, particularly for structured case review, education, and second-opinion style assistance rather than autonomous decision-making.22, 23, 24 Any future deployment should remain embedded within clinician-led workflows, with physicians retaining responsibility for final decisions.25,26 This study has several limitations. First, the evaluation was based on scenario-based case vignettes rather than real-time clinical deployment. Second, the sample size was limited to 40 cases, so the analyses should be interpreted as exploratory. Third, the reference framework was grounded in Chinese domestic guidelines, which improves local clinical relevance but limits geographic generalizability. Fourth, the primary automated analysis relied on Gemini 3.0 Pro as the scoring agent; although the human-rating sensitivity analyses preserved the RAG advantage, residual evaluator bias cannot be fully excluded. Fifth, safety was audited qualitatively rather than quantified as an independent continuous endpoint. Finally, pre-adjudication human rating agreement was limited, which weakens strong claims based on direct AI-human score comparisons. Future research should extend this work to prospective clinical settings, explore adaptive retrieval strategies, and evaluate the integration of multimodal inputs such as imaging and visual fields.27,28 Such work would build naturally on recent advances in multimodal visual-language foundation models for ophthalmology and established longitudinal glaucoma datasets.29, 30, 31 Further refinement of safety-focused evaluation frameworks and exploration of human-AI collaborative decision-making paradigms will be essential to ensure responsible and effective deployment.32,33 Recent studies have further advanced this area by proposing ophthalmic AI clinical research evaluation standards and by benchmarking large language models in both patient-initiated ophthalmology consultations and board-style examination tasks.34, 35, 36 Conclusions In conclusion, retrieval-augmented generation was associated with improved performance of LLMs on scenario-based glaucoma reasoning tasks within a guideline-grounded evaluation framework. The findings support further study of RAG-enhanced systems as clinician-supervised decision-support tools, but stronger safety auditing, broader external validation, and prospective testing are needed before any clinical deployment claims can be made. Study approval The authors confirm that any aspect of the work covered in this manuscript that involved human patients or animals was conducted with the ethical approval of all relevant bodies and the study was performed in accordance with the Declaration of Helsinki, and the protocol was approved by the Ethics Committee of the Second Affiliated Hospital Zhejiang University School of Medicine (Approval No. Y2023-1073). Author contributions The authors confirm contribution to the paper as follows: HY: Responsible for methodology development, investigation, data analysis, and original draft preparation, and secured funding; QM: Contributed substantially to clinical evaluation, patient assessment, and interpretation of clinical findings; WZ: Participated in clinical evaluation, case review, and validation of diagnostic outcomes; CH: Assisted with clinical assessment, data interpretation, and verification of clinical results; YG: Contributed to clinical evaluation, expert judgment, and refinement of clinical conclusions; LS: Responsible for data collection, data curation, and preliminary statistical analysis; HC: Assisted with investigation, data validation, and quality control; AG: Contributed to the revision of the manuscript and provided important input; KJ: Led the conceptualization and overall study design, supervised the research process, reviewed and edited the manuscript, and secured funding. All authors reviewed the results and approved the final version of the manuscript. Declaration of generative AI use During the preparation of this work, the authors used ChatGPT (OpenAI) to improve the language and readability of selected sections and to assist in drafting selected explanatory schematic figures. The AI-assisted visual content was used solely for conceptual illustration and did not represent or alter any primary research data. All AI-assisted text and visual elements were subsequently reviewed, edited, and verified by the authors for accuracy and completeness. The authors take full responsibility for the content of the publication. Funding This work was financially supported by the 10.13039/501100001809Natural Science Foundation of China (grant number 82201195, 82204713), the Education Reform Project of the Second Affiliated Hospital 10.13039/501100004835Zhejiang University School of Medicine (grant number 20240412), and the 10.13039/501100008867Zhejiang Provincial Department of Education Research Project(grant number Y202454784). Declaration of competing interests The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.