EyeRAG: graph retrieval-augmented generation for safe and accurate clinical dialogue in ophthalmology Authors: Kaikai Zhao, Daohuan Kang, Tao Yu, Rui Yao, Wenzheng Han, Ke Liang, Rupesh Agrawal, Andrzej Grzybowski, Kai Jin Published: 2026 ABSTRACT AbstractLarge language models promise to transform ophthalmic clinical communication but face challenges from factual inaccuracies (hallucinations) and limited domain knowledge. This study introduces EyeRAG, a guideline-grounded GraphRAG system for ophthalmic dialogue that integrates OphthaKG, a domain-specific knowledge graph, constructed exclusively from clinical guidelines. We evaluated EyeRAG across 120 clinical scenarios (40 each for glaucoma, diabetic retinopathy, and cataract), six LLMs (GPT-4o, Gemini 2.5 Flash, Grok 4, Llama 3.3 70B, Claude Sonnet 4, DeepSeek-V2.5), and four RAG configurations, assessing clinical accuracy and patient-centered relevance. Validated by LLM-as-a-judge and board-certified ophthalmologists on internal/external datasets, EyeRAG outperformed Vanilla LLMs and standard RAG. LLM evaluations ranked EyeRAG highest (mean 1.61 ± 1.04 internal/1.72 ± 1.18 external). Experts assigned a mean rank of 1.0, with hallucinations falling to 3.3% (vs. 30% baseline). EyeRAG demonstrates the potential for safe, explainable, and clinically grounded language models to be developed as clinician-supervised aids designed specifically to bridge the communication gap between clinicians and patients. It serves as a tool for patient interpretation rather than clinical decision-making, assisting in patient counselling and tele-ophthalmology by translating complex findings into accessible dialog under professional oversight.Subject terms: Computational biology and bioinformatics, Diseases, Health care, Medical research Introduction Effective communication between clinicians and patients is a cornerstone of high-quality healthcare, yet it is often hindered by the complexity of medical concepts and the time constraints of clinical appointments1. In ophthalmology, where intricate anatomical structures and chronic disease processes must be explained within a typical 15-minute consultation2, this communication gap can compromise patient understanding, adherence to treatment, and ultimately, clinical outcomes3,4. Large language models (LLMs5) have emerged as a transformative technology in digital ophthalmology, offering advanced natural language processing capabilities that can support research summarization, clinical decision-making, and patient education6–8. However, despite their potential, the deployment of LLMs in real-world clinical practice remains constrained by two major challenges: the risk of generating factually inaccurate content, so-called “hallucinations”, and the lack of domain-specific contextual understanding9. Such inaccuracies can lead to misinformation about disease prognosis or treatment recommendations, posing a safety risk in patient-facing communication. Moreover, issues of data privacy and the absence of specialized medical reasoning further limit their clinical utility10–12. From a clinical standpoint, such AI-based dialog systems could substantially enhance ophthalmologist–patient interactions by simplifying technical information, reinforcing key health messages, and supporting patient-centered decision-making. Yet, safe deployment in ophthalmology requires stringent factual accuracy, contextual reasoning, and compliance with established care frameworks such as the Clinical Practice Guidelines issued by the Chinese Ophthalmological Society (COS)13. Bridging the gap between algorithmic performance and clinical reliability remains a central challenge in realizing the full potential of LLMs in ophthalmic care. Retrieval-augmented generation (RAG) has recently been proposed as an effective framework to address these shortcomings14,15. By grounding model outputs in trusted external knowledge sources, RAG can significantly reduce hallucinations16 and enhance factual reliability17–19. Importantly, RAG enables dynamic updates to the knowledge base without the need for retraining, supporting continuous alignment with evolving clinical evidence. Its effectiveness has been demonstrated in several healthcare domains—such as radiology and general medicine—where it improved diagnostic dialog quality, mitigated factual drift, and increased clinician trust16,20. Despite its advantages, standard RAG approaches, which typically rely on semantic similarity searches over unstructured text, have their own limitations. They can struggle with complex, multi-hop reasoning21 and may retrieve redundant or tangentially relevant information22, introducing noise into the generation process23. To address these challenges, a more structured approach is needed. Graph-based retrieval-augmented generation (GraphRAG) offers a promising evolution of this concept by leveraging a domain-specific Knowledge Graph (KG) instead of flat text retrieval. By explicitly modeling relationships between clinical entities, GraphRAG enables structured reasoning across interconnected concepts—diseases, anatomical regions, diagnostic tests, and treatments—thereby improving precision, transparency, and reproducibility24,25. Its utility has already been demonstrated in rare disease diagnosis and genetics, where KGs enhance contextual inference and consistency26. In this study, we introduce EyeRAG, a domain-specific GraphRAG system for ophthalmology that integrates the OphthaKG—a structured knowledge graph built from authoritative, evidence-based sources such as the COS Clinical Practice Guidelines (Fig. 1). We utilized LightRAG23, which is a variant of GraphRAG, to automatically extract 12,393 nodes and 13,731 relationships from 94 PDF documents. The ontology of OphthaKG was designed to capture the essential knowledge required for clinical communication, covering key entity types such as diseases, anatomical structures, diagnostic and therapeutic procedures, medications, findings, outcomes, risk factors, and medical devices. The EyeRAG system is a retrieval-augmented generation pipeline that takes a patient’s natural-language query as input. The system first retrieves the most relevant contextual information from OphthaKG. This structured context is then used to augment the prompt that is fed to a large language model, which in turn generates a patient-facing explanation that is accurate, easy to understand, and grounded in the knowledge graph. Results Through comprehensive evaluation across multiple LLMs and RAG variants, EyeRAG demonstrated superior clinical accuracy, completeness, patient-centeredness, and factual grounding while substantially reducing hallucinations, with a residual rate of approximately 3.3% in expert assessment. These results underscore the potential of domain-specific GraphRAG systems to deliver safe, explainable, and clinically supervised conversational AI. Illustrative examples To demonstrate EyeRAG’s capacity for synthesizing complex, patient-centered clinical advice, we analyzed its performance against Vanilla LLMs and standard RAG methods across three distinct patient scenarios drawn from the 120-scenario EyeRAG evaluation dataset. Case 1: Glaucoma and comorbidity management The first case involved an elderly patient with glaucoma, senile cataract, and type 2 diabetes, for whom a question was generated regarding lifestyle changes to manage all conditions together, as shown in Fig. 2. The details of the case and the complete responses of different methods are in Supplementary Table 1 and Supplementary Table 2, respectively. Comparative response examples demonstrating EyeRAG’s superiority in clinical dialog. The figure presents raw, side-by-side responses from the two illustrative scenarios: (Top) Glaucoma and Diabetes Comorbidity Case, where standard RAGs fail to provide integrated lifestyle advice (e.g., blood pressure and glycemic link to eye health), defaulting to PCP referral. EyeRAG, through multi-hop reasoning, provides cross-domain, actionable recommendations. (Bottom) Cataract and Surgical Risk Assessment Case, where standard RAGs offer narrow, vision-centric advice. EyeRAG contextualizes the elective surgery with holistic patient safety and quality-of-life factors (e.g., risk of falls due to poor vision), demonstrating a deeper, clinically-integrated understanding of the patient’s systemic history. Weakness of Vanilla LLM and Standard RAG Methods: All baseline methods (Vanilla LLM, Naive RAG, Hypothetical RAG, and Hierarchical Index RAG) failed this complex, cross-domain query. They consistently defaulted to a lack of concrete advice, often stating that “no specific lifestyle changes are detailed” in the literature, or the focus is only on medical treatment, deferring the patient to their Primary Care Physician (PCP). This demonstrates a fundamental failure of cross-domain synthesis when relying on flat, unstructured text chunks. EyeRAG’s Superiority: In contrast, EyeRAG provided integrated and actionable recommendations. It explicitly linked the conditions by retrieving evidence on optimal blood pressure management (which reduces diabetic retinopathy progression and supports glaucoma control) and maintaining healthy glycemic control (which significantly reduces retinopathy risk). It further provided common-sense, yet clinically relevant, advice on healthy diet, exercise, and weight control that aids intraocular pressure management. This outcome is a direct result of the GraphRAG architecture, as visualized in Fig. 3. Unlike traditional RAG, which struggles to connect disparate text segments, EyeRAG utilizes the structured OphthaKG to perform multi-hop reasoning. It explicitly connects entities like “Glaucoma” and “Diabetes Management” via shared relationship nodes (e.g., ‘Healthy Diet & Lifestyle’), enabling a synthesized, clinically relevant response that flat-text RAG models cannot generate. Illustrative comparison of Traditional RAG and EyeRAG pipelines. A Traditional RAG (e.g., Naive/Hypothetical/Hierarchical variants): Relies on semantic similarity searches over unstructured text corpora, often retrieving redundant or irrelevant chunks (e.g., noisy multi-hop reasoning in ophthalmic queries like glaucoma progression). B EyeRAG (GraphRAG variant): Employs the structured OphthaKG for explicit entity-relationship traversal (e.g., linking “glaucoma” to “diabetes” via nodes/edges), enabling precise, hallucination-resistant retrieval and generation. This contrast highlights GraphRAG’s advantages in reducing noise and enhancing clinical accuracy for ophthalmic dialog. Case 2: Cataract and Complex Surgical Risk Assessment The second case involved an elderly male with bilateral senile cataract and a complex history including uncontrolled hypertension, liver cirrhosis, and pneumoconiosis, asking: “Does my cataract require immediate surgery?” Weaknesses of Vanilla LLM and Standard RAG Methods: All baseline methods correctly identified the surgery as elective, but their focus remained narrow and ophthalmology-centric. They mentioned the comorbidities only as a pre-requisite for a “comprehensive preoperative evaluation”, failing to synthesize the information into a patient-safety context. EyeRAG’s Superiority: EyeRAG went beyond the visual decline and framed the decision around the patient’s holistic safety and quality of life. It explicitly retrieved and integrated high-level clinical guidelines, highlighting critical, non-ophthalmic factors such as safety considerations (the increased risk of falls and fractures in the elderly with vision loss) and the quality-of-life impact, given the patient’s extensive systemic history. This comprehensive framing demonstrates a deeper, clinically-integrated understanding of surgical timing and risk assessment than standard retrieval methods limited to core ophthalmic text. Case 3: Diabetic retinopathy progression and management The third case concerned a patient with a 15-year history of bilateral type 2 diabetes, moderate non-proliferative diabetic retinopathy progressing to tractional retinal detachment in the right eye and scattered hemorrhages in the left, inquiring how diabetes control affects retinopathy progression and treatment. Weaknesses of Vanilla LLM and Standard RAG Methods: As depicted in Supplementary Fig. 1, baseline methods yielded fragmented or generic responses lacking integration with the patient’s advanced bilateral findings. Vanilla LLM offered broad advice on glycemic control for slowing progression but omitted quantification or study references, emphasizing routine monitoring without personalization. Naive and Hierarchical Index RAG cited HbA1c benefits (39% risk reduction per 10% decrease) and rapid control risks (10% temporary worsening), yet structured outputs didactically, prioritizing literature over tailored coordination for tractional detachment. Hypothetical RAG highlighted monitoring and treatment influences but vaguely addressed progression dynamics, diluting focus on glycemic specifics. Overall, these approaches under-synthesized the 15-year history and oral medication context, defaulting to consultative deferrals that diminished empowerment. EyeRAG’s Superiority: EyeRAG delivered a cohesive, evidence-based reply linking management to mechanisms, citing the DCCT study for HbA1c ≤ 7% targets in delaying progression and curbing neovascularization amid detachment. It holistically addressed bilateral complications, noting improved surgical outcomes and microvascular mitigation via glucose, blood pressure, and lipid control, while advocating frequent imaging and multidisciplinary care for vision protection—surpassing baselines in relevance and factual depth. In all examples, EyeRAG’s structured knowledge retrieval enabled the generation of highly comprehensive, personalized, and patient-ready responses, which is critical for safe and effective clinical dialog. Overall performance and rankings on internal data We generated responses using six LLMs: GPT-4o27, Gemini 2.5 Flash28, Grok 429, Llama 3.3 70B30, Claude 4 Sonnet31, and DeepSeek-V2.532, employing the prompts detailed in Supplementary Note 1. The 120 clinical scenarios evaluated in this study are derived from anonymized real-world patient data from two institutions, containing no protected health information (PHI) or personally identifiable data. Institutional review board (IRB) approvals were obtained from Zhejiang University School of Medicine and Wannan Medical College, ensuring compliance with relevant data protection regulations, including GDPR and HIPAA-equivalent standards in China. All queries to remotely hosted LLMs (e.g., GPT-4o, Gemini 2.5 Flash, Grok 4, Claude Sonnet 4) adhered to the respective providers’ terms of service and data processing policies, which permit anonymized, non-sensitive inputs for research purposes. The evaluation was conducted using LLM-as-a-judge with GPT-4o, Gemini 2.0 Flash, Llama 3.3 70B. We performed pairwise comparisons for the responses from five methods (Vanilla LLM and four RAG methods). For each LLM judge, the ranking of the five methods for a specific LLM model (e.g., GPT-4o) was determined by accumulating the scores from these pairwise matches. The average ranking, obtained by averaging the ranks of the three judges on the six LLM models, of the five methods (Fig. 4A) demonstrates the superiority of the RAG methods over the Vanilla LLM, with EyeRAG achieving an average rank of 1.61 ± 1.04, significantly better than the other RAG methods. This dominance is further illustrated by the frequency of being ranked first (Fig. 4B), where EyeRAG was the top choice in 40.29% of evaluations, substantially outperforming the next best RAG method, Naive RAG15 (21.05%), and the Vanilla LLM (13.29%). Overall performance comparison of EyeRAG, standard RAG methods, and a Vanilla LLM. The results are averaged across evaluations from three distinct LLM-as-a-judge models. A Average rank for each method, where lower scores indicate superior performance. EyeRAG achieved the best average rank (1.61 ± 1.04), demonstrating its overall effectiveness. Error bars represent standard deviation (n = 270 questions). B The percentage of questions where each method was ranked first. EyeRAG was the top-rated method in over 40% of cases, nearly twice as often as the next-best approach. To assess ranking stability, we examined how the average rank evolves with the cumulative number of questions (three individual queries per each of the 90 internal clinical scenarios) evaluated (from 1 to 270; see Supplementary Fig. 2A for details). EyeRAG’s average rank converges rapidly to a low value (near 1.6) after approximately 30 questions and remains stable, indicating robust superiority even with limited data. In contrast, other methods show higher variability, while the Vanilla LLM stabilizes at a poorer value (around 4.5). Supplementary Fig. 2B illustrates the convergence of Friedman test33 P-values across judges (reaching < 0.05 after ∼30 questions), averaged ranks (n = 3; <0.05 after ∼30 questions), and independent ranks (n = 18; <0.05 after ∼25 questions and < 1e−6 thereafter), underscoring the rapid statistical validation of EyeRAG’s performance. The individual ranking results across the three LLM judges, as shown in Fig. 5, reveal a consistent rank order (Friedman test P < 0.05). This consistency in rankings across different LLMs underscores the robustness of the RAG approach and its ability to enhance the performance of various models, as further confirmed in external validation (see External Data Validation). Consistency of method rankings across three different LLM-as-a-judge evaluators. LLMs. The results reveal a consistent and statistically significant performance hierarchy (Friedman test, P < 0.05 for all judges), with EyeRAG reliably outperforming other RAG methods and the Vanilla LLM, regardless of the model used for generation or evaluation. Multidimensional performance evaluation To gain a more granular understanding of each method’s strengths and weaknesses, we conducted a multidimensional performance evaluation across five key criteria: clinical accuracy and safety, patient-readiness, completeness and practical applicability, professional communication and clarity, and patient-centered response. As illustrated in the radar chart in Fig. 6, EyeRAG demonstrated a well-rounded and superior performance profile, excelling in all dimensions. It achieved the highest average rank in Clinical Accuracy and Safety, a critical aspect for any medical application, and scored highest in Completeness and Practical Applicability, indicating its ability to provide comprehensive and actionable information. While all RAG methods showed an improvement over the Vanilla LLM, EyeRAG’s performance was particularly notable in its ability to generate responses that were not only clinically sound but also patient-friendly and ready for real-world application. The Vanilla LLM fell significantly short in Clinical Accuracy and Safety, highlighting the risks associated with its use in a clinical context without the safeguards provided by retrieval augmentation. Multidimensional performance evaluation of all five methods across key qualitative criteria. The radar chart plots the average rank for each method, with scores closer to the center indicating better performance. EyeRAG (red line) demonstrates a superior and well-balanced profile by achieving the best scores across all five dimensions, particularly in the clinically critical areas of “Clinical Accuracy and Safety” and “Completeness and Practical Applicability”. Effects of generation temperature To investigate the robustness of EyeRAG across varying levels of response stochasticity, we evaluated its performance using generation temperatures of 0, 0.2, 0.4, 0.6, 0.8, and 1.0 on a subset of 30 questions from the primary evaluation dataset using three LLMs: GPT-4o, GPT-4o mini, and GPT-3.5 Turbo (see Supplementary Fig. 3 for details). EyeRAG maintained superior performance across all settings, with average ranks ranging from 1.45 ± 0.89 at temperature 0 to 1.78 ± 1.12 at temperature 1.0, consistently outperforming other RAG variants and the Vanilla LLM (which ranged from 3.67 ± 0.78 to 4.23 ± 0.95). Low temperatures (0–0.4) yielded the tightest rankings, emphasizing EyeRAG’s deterministic advantages in factual grounding, while higher temperatures showed increased variability but preserved overall superiority. Evaluation by ophthalmologists To assess real-world clinical applicability, we conducted a blinded expert evaluation involving four board-certified ophthalmologists (total n = 60 evaluations across 30 responses per method). Specifically, we selected the first of the three generated questions for the first ten patients in the internal dataset for each disease (glaucoma, diabetic retinopathy, cataract), yielding 30 questions total. Responses were generated by ‘Claude Sonnet 4’, which was found to be the top-performing LLM for all five methods. The 30 responses per method were randomly divided into two non-overlapping groups of 15 based on question ID and assigned to the first two ophthalmologists; a second distinct random split into two groups of 15 was then assigned to the remaining two ophthalmologists. All English responses were translated into Chinese using the Gemini 2.5 Flash API prior to expert review. As shown in Fig. 7, EyeRAG achieved the highest mean total score of 2.91 ± 0.11, significantly outperforming Vanilla LLM (1.32 ± 0.22; p < 0.001, Dunn’s test), Naive RAG (1.81 ± 0.26; p < 0.001), Hypothetical RAG (2.56 ± 0.26; p < 0.01), and Hierarchical Index RAG (2.68 ± 0.26; p < 0.05). EyeRAG excelled in clinical accuracy and safety (2.98 ± 0.12), patient readiness (2.96 ± 0.18), and completeness (2.95 ± 0.22), reflecting its ability to deliver precise, guideline-aligned advice tailored to patient needs. Overall differences were highly significant (Kruskal-Wallis H test: P < 0.001 across all metrics). These results confirm EyeRAG’s superiority in fostering safe, empathetic, and actionable clinical dialog, as evidenced by its mean expert rank of 1.0 in Fig. 8, confirming the findings from the LLM-as-a-judge. Multi-dimensional expert evaluation scores (mean ± SD, 1-3 scale) across five AI models for ophthalmic clinical Q&A, based on four board-certified ophthalmologists (n = 60 evaluations; 30 scenarios per model). EyeRAG outperforms baselines with the highest total score (2.910 ± 0.119) and lowest hallucination rate (3.3%). Overall group differences were assessed using the Kruskal-Wallis test, followed by Dunn’s post-hoc tests; significance levels are indicated by asterisks in each panel. Expert vs. LLM-as-a-judge ranking. This chart compares the average ranks of the top-performing LLMs in each of the five groups, as evaluated by the expert ophthalmologists and the LLM-as-a-judge. Both evaluation methods show a similar trend, with EyeRAG being the top-performing method. Hallucination mitigation A critical barrier to LLM deployment in clinical settings is factual inaccuracy, or “hallucination”, which can erode trust and compromise safety. In our expert evaluations (Fig. 7), each of the 30 scenarios per method was independently assessed twice—once within each of two distinct random splits—by a total of four board-certified ophthalmologists, yielding 60 evaluations per model. Hallucination was scored as a binary outcome (presence/absence) per evaluation, with the reported rate representing the average proportion of scenarios flagged as containing hallucinations across the four experts (e.g., 30.0% for Vanilla LLM, equivalent to 9/30 scenarios on average; standard deviations reflect inter-expert consistency). Rates decreased progressively across RAG variants to 16.6% (Naive RAG; 5/30), 10.0% (Hypothetical RAG; 3/30), and 6.6% (Hierarchical Index RAG; 2/30). EyeRAG achieved the lowest observed hallucination rate, decreasing to 3.3% compared with 30.0% for Vanilla LLM. Given the small number of hallucination events, pairwise comparisons among RAG variants should be interpreted cautiously. This 9-fold reduction stems from OphthaKG’s structured retrieval, which grounds outputs in verifiable entity relationships from COS guidelines, preventing ungrounded inferences. These findings underscore EyeRAG’s robustness for patient-facing applications like tele-ophthalmology counseling. By integrating graph-based evidence, EyeRAG not only mitigates risks but enhances explainability, enabling clinicians to trace responses to specific knowledge nodes. Evaluator agreement To assess the consistency of the rankings, we employed the Friedman test, supplemented by Kendall’s coefficient of concordance (W) to quantify inter-rater agreement. We applied this analysis to the rankings provided by the four expert evaluators, the three LLM-as-a-judge evaluators, and the combined set of expert and LLM-as-a-judge rankings. The statistical analysis results are presented in Table 1. The inter-rater agreement among the expert evaluators was perfect, with a Kendall’s W of 1.000 (Friedman test, P = 0.003). The agreement among the LLM-as-a-judge evaluators was also high, with a Kendall’s W of 0.956 (Friedman test, P = 0.022). There was strong agreement across the combined expert and LLM-as-a-judge rankings, with a Kendall’s W of 0.829 (Friedman test, P < 0.001). Table 1Statistical evaluation results, summary of ranking correlations and performance metrics across different evaluation methodsEvaluation ComparisonFriedman test statisticsFriedman test P-valueKendall’s WInter-rater reliabilityLLM-as-a-judge (n = 3)11.4670.0220.956Expert Evaluators (n = 4)16.0000.0031.000Cross-method correlationExpert vs. LLM-as-a-judge (n = 7)23.2000.0000.829 Statistical evaluation results, summary of ranking correlations and performance metrics across different evaluation methods Ablation studies on context distillation To validate the optimality of EyeRAG’s context distillation and the fairness of RAG comparisons, we conducted ablations on 24 randomly selected clinical scenarios from the internal dataset (8 per disease, 1 question each). Responses were generated using DeepSeek-V3.2, Gemini 2.5 Flash, and GPT-4o, with evaluations via pairwise comparisons judged by GPT-4o and Gemini 2.0 Flash (average ranks; lower is better). As shown in Supplementary Table 3, EyeRAG variants with 10, 20, and 30 distilled points outperformed Naive RAG and Vanilla LLM baselines, with the 30-point variant achieving the lowest ranks in this ablation. We selected 20 points as the default to balance performance, efficiency, and context length. Supplementary Table 4 applies 20-point summarization to both Naive RAG and EyeRAG (vs. raw formats), revealing that summarized Naive RAG performs similarly to raw but still underperforms EyeRAG, underscoring graph retrieval’s superiority over vector-based methods independent of summarization. External data validation To assess the generalizability of EyeRAG beyond the primary evaluation dataset, we collected an independent dataset from an external hospital. Responses were generated using the top-performing LLM (Claude Sonnet 4) across the five methods (Vanilla LLM, Naive RAG, Hypothetical RAG34, Hierarchical Index RAG35, and EyeRAG) and evaluated via LLM-as-a-judge (using GPT-4o, Gemini 2.0 Flash, and Llama 3.3 70B) with pairwise comparisons, following the same protocol as the internal evaluation. The results, aggregated across the three judges, are shown in Fig. 9. EyeRAG maintained its superior performance, achieving an average rank of 1.72 ± 1.18—comparable to its 1.61 ± 1.04 in the primary ‘Overall Performance and Rankings’ evaluation (Fig. 4A)—and was ranked first in 40.37% of cases, on par with its 40.29% in the larger primary set. This indicates robust stability, with EyeRAG outperforming other RAG methods (e.g., Hierarchical Index RAG at 2.23 ± 1.12) and the Vanilla LLM (4.67 ± 0.78). These findings confirm EyeRAG’s effectiveness on unseen, real-world-derived data, with minimal degradation in rank stability or top-performance frequency compared to the primary 270-question benchmark, underscoring its potential for scalable and reliable deployment in real-world clinical communication systems. External data validation results for EyeRAG and comparison methods. Average ranks (lower is better) across the five methods on the 30-question external hospital dataset, evaluated by three LLM-as-a-judge models (GPT-4o, Gemini 2.0 Flash, Llama 3.3 70B). EyeRAG achieves the best rank, demonstrating consistent superiority over other RAG variants and the Vanilla LLM. Error bars represent standard deviation. These results highlight EyeRAG’s generalizability on independent, hospital-sourced scenarios. Discussion Our study demonstrates that integrating a domain-specific knowledge graph into LLM workflows can substantially enhance clinical safety, accuracy, and contextual relevance in ophthalmology. By combining structured retrieval from the OphthaKG with the generative reasoning capabilities of LLMs, EyeRAG establishes a new paradigm for safe, explainable, and patient-centered communication in medicine. This framework addresses key limitations of conventional language models, notably hallucination risk, context fragmentation, and insufficient domain grounding. Across both internal and external datasets, EyeRAG consistently outperformed baseline LLMs and standard RAG methods in expert-assessed accuracy, completeness, and clinical usability, highlighting its robustness and generalizability. From a clinician’s perspective, EyeRAG represents a practical and immediately applicable augmentation tool intended solely to facilitate patient understanding of clinical interactions, not to serve as a primary decision-making system for the physician. It strengthens the quality of ophthalmologist–patient communication by simplifying complex medical terminology, contextualizing systemic comorbidities, and producing responses that align with established clinical guidelines. In typical eye clinics, physicians are often constrained by time and communication fatigue, which can limit the depth of counselling. EyeRAG mitigates this challenge by serving as an interactive assistant that generates evidence-based, patient-friendly explanations. It can be embedded into outpatient kiosks, teleophthalmology platforms, and digital patient portals, providing consistent, guideline-concordant information before, during, and after consultations. These integration pathways across pre-visit, in-consultation, and post-visit stages are illustrated in Fig. 10. When integrated with electronic health record (EHR) systems, EyeRAG could automatically access relevant patient diagnoses, imaging data, and treatment history, enabling personalized education and informed consent discussions. This integration may ultimately reduce clinician workload while enhancing patient comprehension and satisfaction. Clinical integration pathways of EyeRAG in ophthalmic practice. Schematic showing potential deployment points across the patient journey—pre-visit triage and education, in-consultation decision support, and post-visit follow-up. EyeRAG assists clinicians and patients through accurate, contextual dialog, supporting hospital, community, and tele-ophthalmology applications. (Graphic created with assets from iconscout.com.). The near-complete elimination of hallucinations—reducing from 30.0% in baseline LLMs to just 3.3% in EyeRAG—has profound clinical implications. In clinical practice, even minor factual inaccuracies can lead to patient anxiety, misinterpretation, or suboptimal adherence. By grounding responses in authoritative ophthalmic guidelines such as the COS Clinical Practice Guidelines, EyeRAG ensures factual integrity and interpretive consistency. This is particularly critical for ophthalmology, where small nuances, such as the distinction between ocular hypertension and early glaucoma, carry major therapeutic implications. The model’s interpretability and traceability further strengthen clinician trust, as each response can be linked to its knowledge-graph source, facilitating auditability and regulatory compliance. This capacity to deliver explainable and verifiable reasoning aligns with the growing emphasis on transparent AI systems in healthcare. The strong correlation observed between expert ophthalmologist rankings and LLM-as-a-judge evaluations further validates EyeRAG’s reliability. This alignment suggests that structured LLM evaluation frameworks could serve as scalable proxies for expert benchmarking, reducing the reliance on resource-intensive human review in the development of medical AI systems. It also implies that automated quality assessment, when coupled with domain-specific reasoning constraints, may accelerate the safe translation of AI models into clinical use. Beyond technical performance, EyeRAG’s design philosophy emphasizes patient empowerment and health literacy. Ophthalmic diseases such as diabetic retinopathy, glaucoma, and cataract often involve chronic management, where long-term adherence depends on patient understanding and trust. EyeRAG’s ability to generate tailored, accessible explanations could bridge knowledge gaps, particularly for elderly patients or those with limited health literacy. Furthermore, by embedding this system within teleophthalmology frameworks, EyeRAG can extend high-quality clinical communication to remote or resource-limited settings, promoting equitable access to eye care. In multilingual or multicultural contexts, integrating multilingual modules could further enhance inclusivity and reduce disparities in patient education. While EyeRAG was developed specifically for ophthalmology, its modular architecture and reliance on graph-based retrieval make it broadly adaptable. Other specialties such as cardiology, oncology, and endocrinology could develop analogous domain-specific knowledge graphs, allowing the creation of specialty-tailored RAG systems that preserve clinical accuracy while supporting natural communication. The GraphRAG framework can also integrate multimodal data, including imaging, laboratory results, and genomics, paving the way toward multimodal conversational agents capable of explaining not only textual but also visual clinical findings. Future research will focus on three translational extensions. First, a prospective clinical trial will evaluate EyeRAG’s impact on objective patient outcomes, including comprehension scores, satisfaction indices, and adherence to treatment recommendations. Second, integrating EyeRAG with hospital information systems and AI-driven diagnostic pipelines will enable context-aware, interactive communication that adapts dynamically to evolving patient data. Third, multimodal expansion will allow EyeRAG to interpret and contextualize ophthalmic imaging modalities (fundus, OCT, UWF) alongside textual information, offering a truly comprehensive digital assistant for ophthalmologists. Additionally, a continuous updating pipeline for OphthaKG will be implemented to ensure alignment with evolving guidelines and real-world evidence. Future versions will extend the current MinerU-based Markdown pipeline with multimodal table and flowchart parsing and explore automated contradiction detection and resolution mechanisms to better handle cases where new evidence supersedes prior recommendations. Clinical deployment of dialog-based AI systems must adhere to stringent ethical and regulatory standards. EyeRAG’s structured reasoning and transparent data provenance directly support compliance with international frameworks for trustworthy AI in medicine. The explicit traceability of information sources addresses a key regulatory requirement for model explainability. Nonetheless, broader implementation will require institutional oversight, data governance mechanisms, and patient privacy safeguards, especially when integrating with EHR systems. Ethical deployment also mandates human-in-the-loop supervision, where clinicians remain accountable for the final interpretation and communication of information to patients. While our evaluation demonstrates EyeRAG’s advantages in controlled settings, several limitations should be noted. Notably, while the 120 clinical scenarios are derived from real-world, anonymized patient data, the associated patient questions were generated by large language models (DeepSeek-V2.5) rather than authored by real patients. This may limit the linguistic diversity, idiomatic expressions, and cognitive realism reflective of actual clinical interactions. This synthetic approach enhances scalability and standardization but could introduce biases, such as overly structured phrasing or omission of patient-specific cultural contexts. Future work should validate EyeRAG on real-world patient queries to further assess its robustness. Our binary assessment of hallucinations (present or absent) oversimplified clinically meaningful distinctions, treating minor factual imprecisions (e.g., subtle exaggerations of risk without altering advice), speculative extrapolations (e.g., ungrounded generalizations beyond evidence), and clinically unsafe misinformation (e.g., contraindicated treatments posing harm) as equivalent, despite their differing risk levels. This approach, while simplifying expert review for reproducibility, limits nuanced safety interpretation, potentially underestimating the clinical impact of minor vs. major errors in patient-facing applications; future studies should adopt such systems (e.g., minor/moderate/severe based on potential harm) to more precisely evaluate risks and enhance trustworthiness in ophthalmic AI deployment. Additionally, the OphthaKG knowledge graph is constructed exclusively from the COS Clinical Practice Guidelines, which are tailored to region-specific practices in China, including local epidemiological patterns, healthcare resource availability, and cultural considerations in patient management. This may introduce guideline bias and limit the generalizability of EyeRAG to international contexts where clinical standards differ. For example, recommendations on screening protocols, treatment thresholds, or preferred interventions (e.g., for glaucoma or diabetic retinopathy) could vary significantly from those issued by organizations such as the American Academy of Ophthalmology (AAO). In such cases, EyeRAG might generate responses that are factually accurate within the COS framework but potentially misaligned with local practices, leading to suboptimal or contextually inappropriate advice. Several technical limitations also warrant mention. Although MinerU substantially improved recovery of tabular content, highly visual decision trees and complex flowcharts remain challenging for text-based pipelines and may not be fully represented in OphthaKG. In addition, the back-translation of English responses for expert review was performed using Gemini 2.5 Flash; while this model provides high-quality medical translation, minor translation artifacts cannot be entirely ruled out. Finally, although DeepSeek-V2.5 was used for auxiliary tasks (question generation, query rewriting, and context distillation), the consistency of EyeRAG’s superiority across multiple LLMs in response generation, independent LLM judges, and human expert evaluation indicates that the current evidence suggests EyeRAG’s performance gains are unlikely to be fully explained by model-centric stylistic bias. However, some residual influence from the preprocessing pipeline cannot be completely excluded. Additionally, although the incremental processing pipeline supports efficient updates through deduplication, the current architecture does not yet include automated detection or resolution of contradictory recommendations that may arise when guidelines are revised; such conflicts are currently resolved manually by experts during updates. To address this, future developments could involve expanding OphthaKG to incorporate multiple international guidelines through modular updates or federated knowledge integration, adopting advanced layout-aware and multimodal PDF parsing techniques, and implementing explicit conflict-resolution workflows, thereby allowing for customizable, region-specific adaptations while maintaining the system’s core GraphRAG architecture. Methods Dataset This study was conducted in accordance with the principles of the Declaration of Helsinki. The study protocol was reviewed and approved by the Ethics Committees of the participating institutions: the Second Affiliated Hospital of Zhejiang University School of Medicine (No. Y2023-1073), and the First Affiliated Hospital of Wannan Medical College (No. Y2024-1015). The internal dataset used for model development and primary evaluation was derived from anonymized retrospective clinical records collected at the Second Affiliated Hospital of Zhejiang University School of Medicine. It comprises 90 realistic clinical scenarios derived from anonymized patient data, with three questions (generated by DeepSeek-V2.5) asked for each, totaling 270 questions. The scenarios covered three common ophthalmic conditions: glaucoma (30), diabetic retinopathy (30), and cataract (30). The demographic and clinical breakdown of these scenarios is detailed in Table 2. To ensure high fidelity to real-world clinical complexity, the prompts used in this study were generated by an LLM (DeepSeek-V2.5) based on the actual, anonymized medical history and clinical findings of the patients. This method was used to objectively simulate realistic inquiries that patients would naturally pose regarding their specific conditions, covering a range of common concerns such as comorbidity management, prognosis, and treatment. Representative examples of these questions, categorized by theme, are provided in Table 3. Table 2Statistics of the EyeRAG evaluation dataset, summarizing the total number of clinical scenarios by condition (with totals), age distribution, and sex distribution for internal (n = 90 scenarios) and external (n = 30 scenarios) dataClinical conditionsInternal dataExternal dataGlaucoma3010Diabetic retinopathy3010Cataract3010Total9030Age distribution<60 years35.0%43.3%60–70 years26.7%20.0%70–80 years30.0%26.7%> 80 years8.3%10.0%Sex distributionFemale63.3%53.3%Male36.7%46.7% Statistics of the EyeRAG evaluation dataset, summarizing the total number of clinical scenarios by condition (with totals), age distribution, and sex distribution for internal (n = 90 scenarios) and external (n = 30 scenarios) data Table 3Representative questions automatically generated by an LLM (DeepSeek-V2.5) based on the patients’ anonymized actual clinical conditions to simulate realistic inquiries for evaluating patient-centered interactionComorbidity management1. How will my hypertension affect the treatment and management of my eye conditions?2. How does my diabetes affect the management and progression of my eye conditions?Prognosis and progression3. What is the long-term prognosis for my right eye’s chronic angle-closure glaucoma?4. Will the vision in my left eye continue to deteriorate due to the glaucoma?5. Can my diabetic retinopathy lead to complete blindness?Treatment and surgery6. Will the cataract in my right eye require surgery, and if so, when?7. What are the risks and potential complications of cataract surgery?8. To what extent can my vision be restored after surgery?Lifestyle and self-management9. Are there any lifestyle changes I should make to help manage my eye conditions?10. How can I better manage my diabetes to prevent further eye complications? Representative questions automatically generated by an LLM (DeepSeek-V2.5) based on the patients’ anonymized actual clinical conditions to simulate realistic inquiries for evaluating patient-centered interaction 1. How will my hypertension affect the treatment and management of my eye conditions? 2. How does my diabetes affect the management and progression of my eye conditions? 3. What is the long-term prognosis for my right eye’s chronic angle-closure glaucoma? 4. Will the vision in my left eye continue to deteriorate due to the glaucoma? 5. Can my diabetic retinopathy lead to complete blindness? 6. Will the cataract in my right eye require surgery, and if so, when? 7. What are the risks and potential complications of cataract surgery? 8. To what extent can my vision be restored after surgery? 9. Are there any lifestyle changes I should make to help manage my eye conditions? 10. How can I better manage my diabetes to prevent further eye complications? Each scenario was structured around the following key anonymized patient data fields: Case Context: Admission/Discharge Dates and Diagnoses, Chief Complaint, and History of Present Illness. Patient History: Age, Sex, Past medical history, and Surgical/Trauma history. Patient History: Age, Sex, Past medical history, and Surgical/Trauma history. Clinical Findings: Specialty examination results and Differential diagnosis. Clinical Findings: Specialty examination results and Differential diagnosis. The external validation dataset was obtained from the First Affiliated Hospital of Wannan Medical College and approved for research use by the corresponding institutional ethics committee. It comprises 30 realistic clinical scenarios (10 each for glaucoma, diabetic retinopathy, and cataract). For each scenario, a single representative question was generated to simulate patient inquiries, mirroring the thematic coverage in the internal dataset. All retrospective data were fully anonymized prior to analysis, and patient consent was waived by the ethics committees. Informed consent was obtained from patients contributing to the two prospective validation datasets. OphthaKG: a domain-specific knowledge graph for ophthalmology To construct the Ophthalmic Knowledge Graph, we adapted the LightRAG23 framework, a graph-empowered retrieval-augmented generation system, to process 94 PDF documents from the COS Clinical Practice Guidelines, which encompass guidelines on eye diseases, diagnostic procedures, and treatment protocols. A complete list of these source PDF documents is available in the GitHub repository. The PDFs were first parsed using MinerU36, an advanced open-source document parsing toolkit, to generate structured Markdown files (with tables preserved as HTML and logical reading order maintained). The knowledge graph construction process begins with segmenting this Markdown into manageable chunks of approximately 1000 tokens each, enabling efficient handling of the extensive corpus spanning topics like glaucoma, cataract management, and retinal disorders. Using GPT-4o mini (version: gpt-4o-mini-2024-07-18, with default temperature 1.0), we then extracted entities (e.g., anatomical structures like “retina” or conditions like “diabetic retinopathy”) and relationships (e.g., “glaucoma causes optic nerve damage”) from each chunk, forming an initial graph skeleton that captures interdependencies across documents. This extraction leverages LLM prompts tailored to ophthalmic terminology, ensuring precise identification of clinical entities, procedures, and causal links. Subsequently, the LLM profiles each entity and relation into key-value pairs: entities use their names as keys with descriptive values summarizing relevant excerpts (e.g., key: “macular degeneration”, value: a paragraph on age-related symptoms and risk factors), while relations incorporate multiple global keywords (e.g., “treatment efficacy” or “diagnostic imaging”) derived from connected nodes for enhanced thematic retrieval. Finally, a deduplication step merges redundant nodes and edges across chunks, yielding a compact graph Dˆ=(Vˆ,Eˆ) that optimizes storage and traversal. This incremental processing approach supports future updates without full re-indexing. When new guideline versions are added, overlapping nodes and edges are merged via deduplication, while any potential contradictions are currently resolved through manual expert review prior to integration. The experiments reported in this study were conducted on a fixed snapshot of the 94 COS guideline PDFs; no dynamic post hoc graph editing or real-time manual intervention occurred during model evaluation. The approach thereby facilitates rapid integration of evolving COS guidelines while enabling dual-level retrieval for both specific queries (e.g., entity attributes) and abstract ones (e.g., overarching disease impacts), ultimately powering coherent, context-rich responses in ophthalmic applications. The detailed statistics of built OphthaKG are presented in Table 4, and Fig. 11 illustrates three example subgraphs showcasing connections for distinct entities: ‘Dry Eye’ (a medical condition, e.g., linking to clinical findings and risk factors), ‘Initial Exam History’ (a diagnostic procedure, e.g., linking to anatomical structures and patient populations), and ‘Atropine’ (a medication, e.g., linking to treatment methods and clinical outcomes). These examples highlight the graph’s structured representation of ophthalmic knowledge for precise retrieval in clinical dialog generation. Table 4OphthaKG knowledge graph statisticsComponentCountDescriptionData SourcesCOS Clinical Practice Guidelines94PDF documentsGraph structureTotal nodes12,393Extracted entitiesTotal relationships13,731Entity connectionsEntitytypesMedical conditions~2500Diseases and disordersAnatomical structures~1800Eye anatomy componentsDiagnostic procedures~1200Testing and examination methodsTreatment methods~1500Therapeutic interventionsMedications~800Pharmaceutical treatmentsClinical findings~2000Observable symptoms and signsClinical outcomes~900Treatment results and prognosisRisk factors~600Predisposing conditionsPatient populations~400Demographic groupsMedical devices693Instruments and equipmentExamples of top entities (by connections)Ophthalmologist156Most connected entityVisual acuity142Key clinical measureGlaucoma128Major disease categoryPatient115Central entityQuality of life98Important outcome measureDetailed breakdown of the constructed Ophthalmic Knowledge Graph. OphthaKG knowledge graph statistics Detailed breakdown of the constructed Ophthalmic Knowledge Graph. Example subgraphs from the OphthaKG knowledge graph, illustrating structured relationships for distinct ophthalmic entities. A Global Graph Overview: Displays the first 500 extracted entities and their relationships. This subset is presented to avoid visual clutter, as plotting all 12,393 nodes would make the complex multi-hop connectivity difficult to discern. B Subgraph centered on ‘Dry Eye’ (120 edges), highlighting connections to clinical findings, risk factors, and treatment methods. C Subgraph centered on ‘Initial Exam History’ (57 edges), showcasing links to diagnostic procedures, anatomical structures, and patient populations. D Subgraph centered on ‘Atropine’ (16 edges), demonstrating associations with medications, treatment methods, and clinical outcomes. These visualizations, obtained using GraphXR38, demonstrate the graph’s ability to capture explicit entity relationships extracted automatically from COS Clinical Practice Guidelines. To assess the accuracy of the LLM-based extraction, we conducted a manual audit on a random sample of 200 entities (~1.6% of nodes) and 200 relationships (~1.5% of edges) from OphthaKG. A board-certified ophthalmologist reviewed each item on two criteria: (1) presence in the source COS Clinical Practice Guidelines, and (2) medical accuracy of the descriptions. The audit yielded 100% precision for guideline presence and 100% accuracy for descriptions, with no errors detected. Notably, among the sampled entities, approximately 3% consisted of items with limited generalizability, such as specific dates (e.g., ‘18 weeks’), study references (e.g., ‘2012 Meta-Analysis’), or person names, which may reflect guideline-specific details but could reduce broader applicability in certain contexts. Multilingual processing pipeline All COS Clinical Practice Guidelines were processed in their original Chinese language to construct OphthaKG, ensuring no loss of domain-specific medical nuance. Retrieved context from OphthaKG was therefore initially in Chinese. For the EyeRAG system, this context was summarized by DeepSeek-V2.5 into up to 20 concise English key points (average 1724 tokens) before being provided to the downstream LLMs. Patient clinical scenarios, originally in Chinese, were translated into English once at the study outset to enable consistent question generation and evaluation. All prompts, illustrative examples, and model-generated responses presented in the main text and supplementary materials are in English. For the blinded expert evaluation by four board-certified Chinese ophthalmologists, the English responses from all five methods (Vanilla LLM and four RAG variants) were translated back into Chinese using the Gemini 2.5 Flash API in a batch programmatic process. This ensured that native-speaking clinicians could perform accurate clinical assessments in their preferred language. This hybrid design improves model reasoning compatibility and reviewer accessibility, while potentially introducing limited translation-related noise during back-translation for expert review. Innovations in EyeRAG While EyeRAG leverages established frameworks such as GraphRAG and LightRAG for knowledge graph construction and retrieval, it extends beyond mere domain adaptation to ophthalmic guidelines. EyeRAG is not simply an application of GraphRAG but introduces targeted advancements tailored to safe, patient-facing clinical dialogs in ophthalmology. First, EyeRAG incorporates KG-driven constraints for patient-facing reasoning, enabling clinically grounded multi-hop inference. This involves a custom, task-specific ontology derived from COS guidelines, which structures entities and relationships to prioritize patient education, comorbidity management, and shared decision-making, features not emphasized in general-purpose systems like MedGraphRAG25 or Almanac17. Second, it implements dialog-level safety grounding to mitigate hallucinations, integrating distilled context (e.g., 20 key points from OphthaKG) directly into LLM prompts. This ensures outputs are evidence-based and contextually relevant, reducing factual inaccuracies from 30% in baselines to 3.3%, as validated in our evaluations. Third, EyeRAG features a comprehensive evaluation framework for hallucination suppression, assessed across six LLMs, four RAG configurations, and internal/external datasets using LLM-as-a-judge and expert annotations. This framework provides robust metrics for clinical accuracy and relevance, setting it apart from prior medical GraphRAG systems that often lack domain-specific safety benchmarks. These innovations, supported by OphthaKG’s guideline-exclusive sourcing, enable safer and more effective patient interactions under professional oversight. The EyeRAG system interaction pipeline The EyeRAG system is designed to seamlessly integrate into the clinical workflow as a communication bridge for the patient. The pipeline begins with a simulated patient query generated by a Large Language Model (DeepSeek-V2.5) that is contextually grounded in the patient’s actual clinical data. EyeRAG then processes this prompt, retrieving relevant context from OphthaKG to generate a patient-ready explanation that clarifies the clinician’s findings and interaction. This structured information is then used to augment the input to a large language model, which generates a patient-facing explanation that is clear, accessible, and grounded in the expert knowledge from the knowledge graph, as shown in Fig. 1. Evaluation protocol We evaluated the performance of six LLMs (GPT-4o, Gemini 2.5 Flash, Grok 4, Llama 3.3 70B, Claude Sonnet 4, and DeepSeek-V2.5) across five configurations: a baseline Vanilla LLM, three standard RAG implementations (Naive RAG15, Hypothetical RAG34, and Hierarchical Index RAG35), and the proposed EyeRAG system. For all experiments, the generation temperature for LLM responses was set to 0 unless otherwise stated, to reduce randomness, produce more deterministic outputs, and enhance overall consistency. For the RAG methods (Naive RAG, Hypothetical RAG, Hierarchical Index RAG, and EyeRAG), we retrieved 10 chunks and reranked to the top 3 for context augmentation using consistent parameters across all variants to ensure fair comparison: initial retrieval via semantic similarity (GPT-4o mini for query rewriting), followed by cross-encoder reranking. This resulted in average context lengths of ~2350–2400 tokens for standard RAG methods (Supplementary Table 5), while EyeRAG’s graph retrieval was distilled to 20 key points (avg 1,724 tokens) to handle its denser output without exceeding token limits. Fairness was validated through ablations: varying distilled points in EyeRAG (10/20/30; Supplementary Table 3) and applying 20-point summarization to both Naive RAG and EyeRAG (Supplementary Table 4), confirming that graph structure, not context volume or summarization, drives performance differences. Prior to retrieval in all five methods, following common practice14, we rewrote the original patient questions using DeepSeek-V2.5 to improve the quality of retrieval, incorporating both the original question and the patient’s clinical data to enhance rewriting quality. An example of this rewriting process is provided in Supplementary Table 2, with the full prompt detailed in Supplementary Note 1 (S1.2). Although DeepSeek-V2.5 was also used for simulated question generation and context distillation, query rewriting was applied uniformly across all methods, final responses were generated by six different LLMs, and both LLM-as-a-judge and human expert evaluations were performed independently of DeepSeek-V2.5. In EyeRAG, the retrieved context, consisting of entities, relations, their descriptions, and relevant chunks from OphthaKG, was summarized by DeepSeek-V2.5 to summarize this context into up to 20 key points (approximately two sentences each) to reduce the number of tokens fed to the downstream LLM. Although the 30-point variant achieved the lowest average rank in the small ablation, we selected 20 points as the default to balance performance, efficiency, and context length. An illustrative example of this summarization is included in Supplementary Table 6, expanded to include raw retrieved graph elements alongside distilled versions for two representative cases (glaucoma with diabetes; diabetic retinopathy with hypertension), highlighting multi-hop reasoning (e.g., linking “Blood Pressure Control” to “Retinal Protection” via metabolic relationships), and the associated prompt is in S1.5. DeepSeek-V2.5 was chosen for both tasks due to its cost-effectiveness and efficacy for these relatively simple NLP operations. LLM-as-a-judge evaluation We employed an LLM-as-a-judge approach for comparisons between EyeRAG responses and baseline methods. Three LLMs were used as judges (GPT-4o, Gemini 2.0 Flash, and Llama 3.3 70B) to reduce model-specific bias and ensure robust rankings. Judgments were based on five distinct scoring dimensions: Clinical Accuracy and Safety, Patient-Centered Response, Professional Communication and Clarity, Completeness and Practical Applicability, and Patient-Readiness. Initially, we tested an absolute scoring approach based on predefined criteria across five dimensions. However, we observed that LLM-as-a-judge consistently assigned high scores near the maximum of 3 (within the [1, 3] range), reducing discriminative power, prompting a shift to pairwise scoring. In the pairwise scoring method, the judge LLM received a tailored prompt containing the patient’s question, the two anonymized responses (A and B), the scoring dimension, and the specific evaluation criterion for that dimension. The LLM was constrained to output A (response A is better), B (response B is better), or E (responses A and B are equally good), focusing solely on the given dimension. Detailed prompt templates, criteria definitions, and example inputs are provided in Supplementary Table 7. LLMs tend to favor the first-presented response, introducing a positional bias37. To mitigate this bias, we systematically reversed response order and re-ranked outcomes, then averaged the results to obtain unbiased composite rankings, and then fused the results: identical best responses were retained, A and B outcomes were treated as equal, and an E result deferred to the other rank. Each match awarded 3 points to the superior response and 1 point to both in case of a tie, with scores accumulated across all five dimensions to determine match outcomes for each question using the same response-generating LLM (e.g., GPT-4o). Match results for a given LLM were aggregated across all questions. With five methods, we conducted 10 pairwise matches (5 × 4/2), ranking methods 1 to 5 based solely on total scores. This approach ensured robust evaluation across all clinical scenarios. Expert evaluation Responses were scored across five key dimensions—Clinical Accuracy and Safety, Patient-Centeredness, Professional Communication and Clarity, Completeness and Practical Usability, and Patient Readiness. Scores for indicators 1–5 (dimension) were assigned on an ordered 1–3 scale (higher indicating better performance), while hallucination was assessed as a binary outcome (presence/absence). Hallucinations were operationally defined for expert reviewers as any generated content that was factually inaccurate, unsupported by the retrieved knowledge graph or clinical guidelines, or extrapolated beyond evidence-based sources. Statistical analysis employed the Kruskal-Wallis H test for overall group differences (P < 0.05 threshold), followed by Dunn’s post-hoc pairwise comparisons, and Fisher’s Exact Test with Bonferroni-corrected pairwise tests for hallucination rates. Supplementary information Supplementary information Acknowledgements This study was supported by the National Natural Science Foundation of China (Nos. 82201195 and 62172417). Author contributions Kai Jin conceived the study, designed the overall framework, and secured funding. Kaikai Zhao was responsible for the algorithm, experiments, statistical analysis, and interpretation of the results. Daohuan Kang conducted the clinical validation. Tao Yu collected the clinical guidelines. Daohuan Kang, Tao Yu, and Wenzheng Han collected and organized the data. Kaikai Zhao wrote and revised the initial manuscript. Rui Yao, Ke Liang, Rupesh Agrawal, and Andrzej Grzybowski provided feedback and suggestions for manuscript revisions. All authors have read and approved the manuscript. Data availability The datasets analyzed in this study are not publicly available due to privacy and ethical restrictions protecting sensitive patient information, but are available from the corresponding author upon reasonable request. Code availability The code is available at https://github.com/OpenMedAILab/EyeRAG. Competing interests Kai Jin and Rupesh Agrawal are Associate Editors of npj Digital Medicine and were not involved in the selection of peer reviewers for this manuscript nor in any subsequent editorial decisions. The other authors do not have a competing interest. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. These authors contributed equally: Kaikai Zhao, Daohuan Kang. Supplementary information The online version contains supplementary material available at https://doi.org/10.1038/s41746-026-02860-7. References Associated Data Supplementary Materials Supplementary information Data Availability Statement The datasets analyzed in this study are not publicly available due to privacy and ethical restrictions protecting sensitive patient information, but are available from the corresponding author upon reasonable request. The code is available at https://github.com/OpenMedAILab/EyeRAG.