Comparative evaluation of information quality, readability, and guideline consistency of large language model–generated educational content on hemodialysis complications: a cross-sectional assessment of generative AI-driven chatbots Authors: Liuyingxi Su, Lijun Xu, Jun Tang, Manhua Zuo Published: 2026 ABSTRACT BackgroundLarge language models are increasingly becoming a key resource for hemodialysis patients to access information on disease management. However, the information reliability, readability, and guideline concordance of LLM-generated hemodialysis-related educational texts remain insufficiently evaluated.MethodsThis study identified 42 dialysis-related questions from an initial pool of 200 candidate questions extracted from Google Trends, relevant clinical guidelines, and online forums. Using a standardized single-turn, zero-shot prompting strategy with default web-interface settings, these questions were independently input into five models (ChatGPT-4o, DeepSeek-V2.5, Gemini 2.5 Pro, Perplexity Pro, and Copilot). Two trained raters independently evaluated the outputs using the DISCERN, EQIP, JAMA, and GQS scales in a blinded review, with disagreements adjudicated by a third senior nephrologist. Readability was quantified using the FKGL, FRES, GFI, CLI, and SMOG metrics. Additionally, using internationally authoritative guidelines such as KDIGO as a benchmark, guideline concordance and potential text-level safety concerns in the generated outputs were reviewed against authoritative hemodialysis-related guidelines, and qualitative methods were employed to describe the issue of hallucinations in the model outputs.ResultsSignificant differences were observed across the five LLMs for all four information-quality metrics (P < 0.001 for DISCERN and EQIP; P = 0.002 for GQS and JAMA). RAG-based models, particularly Perplexity and Copilot, showed relatively higher information reliability. None of the outputs met the recommended sixth-grade readability benchmark, and greater guideline concordance was often accompanied by higher linguistic complexity. RAG-based models also showed relatively better alignment with reference guideline statements, whereas non-retrieval-based models more often omitted guideline-recommended elements or provided less specific responses. Qualitative review identified several examples of model-generated “medical hallucinations,” including contraindicated self-management suggestions, potentially inappropriate dietary advice, and out-of-scope clinical instructions presented as self-care, indicating potential text-level safety concerns if used without professional review.ConclusionRAG-based models showed relatively better evidence support, information reliability, and guideline concordance in hemodialysis-related educational text generation. However, all evaluated LLMs produced outputs with readability barriers and occasional potentially unsafe or out-of-scope recommendations at the text level. These findings do not establish the actual clinical safety or effectiveness of LLM use among hemodialysis patients, but they indicate that unsupervised patient-facing use should be approached cautiously and that expert review and plain-language adaptation are necessary before such outputs are used as educational materials.Supplementary InformationThe online version contains supplementary material available at 10.1186/s12882-026-05048-z. Introduction End-stage kidney disease (ESKD) is a rapidly growing global health challenge that presents substantial public health and socioeconomic burdens. Based on data from the United States Renal Data System (USRDS), over 3 million patients worldwide currently rely on renal replacement therapy for survival—a population projected to reach approximately 5.4 million by 2030 [1, 2]. While maintenance hemodialysis is life-sustaining, it exposes patients to numerous acute and chronic complications. These include intradialytic hypotension, vascular access abnormalities, mineral and bone disorders, anemia, uremic pruritus, muscle cramps, and cardiovascular events. Rather than occurring in isolation, these issues stem from interconnected pathophysiological mechanisms, such as uremic toxin accumulation, fluid volume shifts, chronic inflammation, and hemodynamic instability [3–6]. The clinical impact of these individual complications is severe. Intradialytic hypotension, for instance, occurs in 20%–30% of sessions; it severely compromises dialysis adequacy and correlates strongly with cardiovascular events and all-cause mortality [7–8]. Similarly, muscle cramps affect 12% to 45% of patients, acting as a major driver of discomfort and poor treatment adherence [9–10], while vascular access issues remain a primary cause of hospital readmissions [11]. The cumulative burden of these conditions drives up medical costs and profoundly diminishes patients’ quality of life [12–13]. Exacerbating this vulnerability is the fact that approximately 20%–30% of hemodialysis patients experience low health literacy. This deficit impairs their ability to comprehend medical interventions and self-manage, further elevating their risk of adverse clinical outcomes [14]. Relying exclusively on facility-based interventions to manage maintenance hemodialysis complications presents notable limitations. Consequently, current clinical guidelines strongly emphasize active patient engagement and effective self-care as essential components for mitigating risks and improving long-term outcomes [15–16]. Successful self-management, however, is inherently tied to a patient’s health literacy, defined as their capacity to effectively process and apply medical information. Driven by the complex, daily demands of disease management and the time constraints typical of standard clinical encounters, a growing number of patients are turning to the internet and artificial intelligence (AI) chatbots as primary resources for medical guidance [17–18]. In recent years, large language models (LLMs) such as ChatGPT, Gemini, and DeepSeek—have garnered widespread attention for their ability to instantly extract and generate personalized health information from vast amounts of data. They have been proposed as tools that may support patient education and self-management, although their actual effects on patient understanding and behavior remain uncertain [19–21]. Despite their potential, LLMs frequently generate medical content fraught with critical limitations: inconsistent accuracy, informational redundancy, prohibitive language complexity, and poor alignment with established guidelines. Consequently, their reliability and suitability for unsupervised patient education remain uncertain [22–24]. For instance, the readability of ChatGPT-generated medical explanations consistently registers at the college or graduate level—well above the 6th- to 8th-grade standard recommended for patient materials [25]. More alarmingly, these models can output unsubstantiated advice, omit crucial treatment details, or directly contradict clinical protocols, thereby raising potential safety concerns if such information is acted upon without professional interpretation [22, 26–27]. Despite ongoing evaluations of LLMs in other medical specialties [26, 28–29], systematic assessments focusing on hemodialysis complications are notably absent. Managing these hemodialysis-related complications requires patients to understand and apply complex self-care information. Inaccurate, incomplete, or unclear AI-generated responses may pose text-level risks for patients with complex self-management needs [30–31]. This study evaluated the information quality, readability, guideline concordance, and potentially unsafe content of LLM-generated responses to predefined questions on hemodialysis complications. The findings aim to inform the cautious development and expert review of AI-generated educational materials, rather than assess the clinical safety or effectiveness of LLM use in real patients. Methods Study design and data sources Candidate questions were extracted from Google Trends, relevant clinical guidelines, and online forums, generating an initial pool of 200 questions. Following deduplication and screening by two nephrology specialists, 42 questions were retained for final evaluation, as detailed in Supplementary Fig. 1 and Supplementary Table 2. Seven hemodialysis patients were then invited to participate in the PPI process to refine the clarity, relevance, and patient-facing wording of the retained questions. Their education level, dialysis vintage, self-reported health-literacy-related difficulty, and internet use experience are presented in Supplementary Table 3. Health -literacy-related difficulty was based on participants’ self-reported understanding of hemodialysis complication–related knowledge and was not measured using a validated health literacy scale. Model selection and data collection This evaluation included five large language models (LLMs): ChatGPT-4o (OpenAI, version 4.0), Gemini 2.5 Pro (Google, version 2.5), Perplexity Pro (Perplexity AI, version 1.0), Microsoft Copilot (Microsoft, version 1.1), and DeepSeek-V2.5 (DeepSeek, version 2.5), a publicly available open-weight model. Outputs were collected over two weeks, from September 1 to 15, 2025, while the overall study spanned from June 1 to December 1, 2025. Exact query dates for each model are listed in Supplementary Table 4. To comply with regional internet and medical data regulations, queries were made from IP addresses in Zhuhai, Guangdong, China. For consistency with typical user interactions, default web-based interfaces were used instead of API queries, and no inference hyperparameters (e.g., temperature, top-p) were adjusted. Questions were manually entered into each model’s public web interface, with no preloaded guidelines or system instructions. Web-search features, where available, were set to default to minimize user-driven variations. To standardize responses and prevent context carryover, each question was entered in a new session, and browser data (caches, cookies, login sessions) were cleared before every query. This design focused on evaluating single-turn outputs under standardized conditions rather than simulating multi-turn patient-chatbot interactions. Prompting strategy A standardized single-turn, zero-shot prompting approach was used to evaluate each model’s response to 42 hemodialysis-related questions. No prompt engineering or additional clinical context was provided. Each question was entered exactly as phrased in a new conversation, ensuring cross-model comparability of initial responses. Multi-turn interactions, personalized exchanges, and responses incorporating patient-specific information were not assessed. Reliability assessment We employed four internationally recognized scales to assess the generated health texts across multiple domains, including information reliability, structural completeness, linguistic clarity, and source transparency. Both data extraction and scoring were performed blindly by two independent researchers. In cases of disagreement, a third senior nephrologist was consulted to mediate and finalize the scores. The specific tools utilized were: DISCERN Instrument: Comprising 16 items scored on a 5-point Likert scale, this tool evaluates the reliability of health information, specifically focusing on the objective discussion of treatment alternatives and overall content quality [32]. Following established precedents [33], cumulative scores were stratified into four qualitative tiers (Poor, Fair, Good, and Excellent) to enable inter-group comparisons. EQIP Tool: The Ensuring Quality Information for Patients (EQIP) checklist assesses the structural integrity, clarity, and internal logic of the educational materials. This 20-item tool uses a 3-point scoring system: 1 (Yes), 0.5 (Partially), and 0 (No). To prevent scoring bias, “Not Applicable” items were omitted from the denominator when calculating the final percentage scores, which were subsequently mapped to standardized quality thresholds [34]. JAMA Benchmarks: This framework evaluates source transparency and ethical compliance based on four binary (0 or 1) criteria: authorship, attribution, conflict of interest disclosures, and publication date [35–36]. Cumulative scores range from 0 to 4, with higher values indicating superior transparency and accountability. GQS Scale: The Global Quality Scale (GQS) is a 5-point metric that measures the general flow, coherence, and practical utility of the generated texts, providing a supplementary assessment of their educational value for patients [37]. Readability assessment The readability of the generated texts was assessed using the Readability Formulas platform (readabilityformulas.com). We selected six standardized indices for this analysis, namely: the Automated Readability Index (ARI), Flesch Reading Ease Score (FRES), Gunning Fog Index (GFI), Flesch-Kincaid Grade Level (FKGL), Coleman-Liau Index (CLI), and SMOG Index. The equations used to compute these metrics are as follows: Automated Readability Index(ARI) \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eqalign{ {\rm{ARI}} & = 4.71\left( {{{{\rm{characters}}} \over {{\rm{words}}}}} \right) \cr & + 0.5\left( {{{{\rm{words}}} \over {{\rm{sentences}}}}} \right) - 21.43 \cr}$$\end{document} Flesch Reading Ease Score(FRES) \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eqalign{ {\rm{FRES}} = & 206.835 \cr & - 1.015\left( {{{{\rm{words}}} \over {{\rm{sentences}}}}} \right) \cr& - 84.6\left( {{{{\rm{syllables}}} \over {{\rm{words}}}}} \right) \cr}$$\end{document} Gunning Fog Index(GFI) \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\mathrm{G}\mathrm{F}\mathrm{I}=0.4\left[\right(\frac{\mathrm{words}}{\mathrm{sentences}})+100(\frac{\mathrm{complex\:words}}{\mathrm{words}}\left)\right]$$\end{document} Flesch–Kincaid Grade Level(FKGL) \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eqalign{ {\rm{FKGL}} = & 0.39\left( {{{{\rm{words}}} \over {{\rm{sentences}}}}} \right) \cr& + 11.8\left( {{{{\rm{syllables}}} \over {{\rm{words}}}}} \right) - 15.59 \cr}$$\end{document} Coleman–Liau Index(CL) \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eqalign{ {\rm{CL}} = & 5.89\left( {{{{\rm{characters}}} \over {{\rm{words}}}}} \right) \cr & - 0.3\left( {{{{\rm{sentences}}} \over {{\rm{words}}}}} \right) - 15.8 \cr}$$\end{document} Simple Measure of Gobbledygook(SMOG) \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eqalign{ {\rm{SMOG}} = & 14.430 \cr & \times \sqrt {{{{\rm{polysyllables}} \times 30} \over {{\rm{sentences}}}}} + 3.1291 \cr}$$\end{document} By evaluating syntactic density, sentence length, and lexical complexity, these metrics objectively estimate the minimum educational level required for lay comprehension. Both the American Medical Association (AMA) and the National Institutes of Health (NIH) recommend that patient-oriented educational materials not exceed a sixth-grade reading level [38]. Adopting this standard as our primary benchmark, we established an acceptable lower limit of 80.0 for the FRES, while the threshold for the remaining five indices was capped at the sixth-grade level or below [39]. We graphically compared the average readability scores across all models with the recommended clinical thresholds [40]. By assessing linguistic complexity in this manner, the study establishes a baseline standard to assess whether the generated texts met commonly recommended readability standards for patient-facing educational materials [41]. Guideline consistency assessment Guideline concordance was evaluated using a framework adapted from prior studies [42], without creating new clinical criteria. Reference answers were extracted directly from authoritative guidelines and consensus documents, including KDIGO [15], KDOQI [43], CDC infection control guidelines [44], the 2021 Standard Operating Procedures for Hemodialysis [45], the 2021 Chinese CKD Nutritional Therapy Guidelines [46], the 2022 consensus on intradialytic hypotension [47], the Clinical Practice Guideline for Hemodialysis [48], the BC Renal hemodialysis complications nursing guideline [49], and the IRLSSG consensus statements on restless legs syndrome [50]. For each predefined question, guideline statements explicitly addressing the clinical scenario were identified, and irrelevant or duplicative content was excluded. The extracted reference answers were independently reviewed by two board-certified nephrologists to ensure accuracy and content validity, with cross-verification against multiple guidelines to confirm coverage of all essential elements. Supplementary Table 10 provides a complete mapping of all 42 patient-oriented questions to the corresponding guideline sources and predefined reference answer elements used for the guideline-concordance evaluation. This approach ensured that the assessment relied on established evidence-based standards rather than author-derived recommendations, maintaining consistency and reproducibility. Responses were evaluated across four domains: applicability, accuracy, supplementary information, and incompleteness. Each domain was coded as binary without weighting. The operational definitions and scoring criteria for these four domains are presented in Supplementary Table 1. Two blinded nephrology specialists independently scored all responses, and disagreements were adjudicated by a third senior nephrologist. Inter-rater agreement was almost perfect (Cohen’s κ = 0.832, 95% CI: 0.78–0.93). This framework assessed only text-level guideline concordance, not clinical safety or effectiveness. Qualitative review of medical hallucinations and potential risks Building upon the guideline concordance evaluation, we conducted a qualitative analysis of model-generated “medical hallucinations”—defined as pseudo-medical assertions that contradict pathophysiology, violate established guidelines, or misapply general knowledge [51]. Two independent hemodialysis specialists performed a line-by-line review of the potentially unsafe text segments. Diagnostic discrepancies were resolved through blinded adjudication by a third senior nephrologist. Ultimately, potentially unsafe text segments were categorized into three categories of potential text-level safety concern if used without professional supervision: (1) contraindicated self-management suggestions, (2) dietary recommendations that may be inappropriate for patients with kidney failure, and (3) out-of-scope clinical instructions presented as patient self-care. Statistical analysis All statistical analyses were performed using R version 4.3.2. Descriptive statistics were used to summarize DISCERN, EQIP, JAMA, GQS, and readability scores. Because the same 42 questions were answered by each of the five models, model comparisons were treated as paired repeated-measures data, with question serving as the blocking factor. Continuous and ordinal scores were summarized as mean ± standard deviation and median [Q1, Q3], as appropriate. For non-binary outcomes, overall differences among the five models were assessed using Friedman tests. Kendall’s W was reported as the effect size for Friedman tests. When the overall test was significant, post-hoc pairwise comparisons were performed using paired Wilcoxon signed-rank tests. Pairwise effect estimates were reported as Hodges–Lehmann median differences with 95% confidence intervals. For each outcome metric, the ten pairwise model comparisons were treated as one comparison family, and P values were adjusted using the Benjamini–Hochberg procedure. Adjusted P values greater than or equal to 0.001 were reported as exact values, whereas values less than 0.001 were reported as P < 0.001. In the guideline-concordance assessment, applicability, accuracy, supplementary information, and incompleteness were coded as categorical variables. These data were summarized using frequencies and percentages. For paired binary comparisons, McNemar tests were used for post-hoc model comparisons, with Benjamini–Hochberg correction applied within each outcome. For accuracy, supplementary information, and incompleteness, only guideline-applicable questions were included in the denominator. Inter-rater reliability for DISCERN, EQIP, JAMA, and GQS total scores was assessed using the intraclass correlation coefficient for average measures, whereas inter-rater agreement for categorical guideline-concordance ratings was assessed using Cohen’s kappa coefficient. All tests were two-sided, with statistical significance defined as adjusted P < 0.05 where applicable. Results Characteristics of the included prompts This study included 42 questions on hemodialysis complications, constructed to reflect common educational topics relevant to chronic hemodialysis patients (Supplementary Table 5). These prompts spanned key clinical areas, including arteriovenous access abnormalities, intradialytic blood pressure volatility (hypotension and hypertension), anemia management, mineral and bone disorders, uremic pruritus, muscle cramps, dialysis-associated headaches and gastrointestinal distress, malnutrition, chronic microinflammation, and restless legs syndrome. Structurally, the queries were designed to elicit model responses across four core domains of patient care: risk factor identification, symptom recognition, treatment strategies, and self-management practices. Reliability analysis Before group comparisons, inter-rater reliability was examined for the four quality assessment instruments. The total scores of the four quality assessment instruments showed high inter-rater reliability between the two primary raters. The average-measures ICC estimates were 0.871 for DISCERN (95% CI: 0.842–0.895), 0.863 for EQIP (95% CI: 0.801–0.910), 0.894 for GQS (95% CI: 0.812–0.964), and 0.892 for JAMA (95% CI: 0.821–0.952), all P < 0.001 (Supplementary Table 8). As shown in Table 1, Friedman tests indicated significant differences across the five LLMs for all four quality metrics: DISCERN (P < 0.001, Kendall’s W = 0.937), EQIP (P < 0.001, Kendall’s W = 0.270), GQS (P = 0.002, Kendall’s W = 0.103), and JAMA (P = 0.002, Kendall’s W = 0.098). Perplexity and Copilot generally showed higher DISCERN and EQIP scores than the non-retrieval-based models, whereas ChatGPT and DeepSeek showed lower scores, mainly because of weaker source attribution and less detailed discussion of treatment-related information. The mean reliability scores across the five models are further illustrated in Fig. 1. Post-hoc paired Wilcoxon signed-rank tests with Benjamini–Hochberg correction showed that Perplexity scored significantly higher than ChatGPT, DeepSeek, and Gemini on DISCERN, and also outperformed ChatGPT, DeepSeek, and Gemini on EQIP. Copilot also showed higher DISCERN and EQIP scores than several non-retrieval-based models. Significant post-hoc pairwise comparisons for reliability and quality metrics are summarized in Supplementary Table 6. The table reports paired mean differences, median differences, Hodges–Lehmann median differences with 95% confidence intervals, and Benjamini–Hochberg-adjusted P values. Table 1Comparative analysis of reliability, structural quality, and transparency scores for educational content on complications in patients undergoing maintenance hemodialysis across five LLMsMetricChatGPTCopilotDeepSeekGeminiPerplexityOverall PKendall’s WDISCERN36.00 [34.25, 37.00]43.00 [41.25, 46.00]33.00 [32.00, 35.00]39.50 [38.25, 41.00]45.00 [43.00, 47.75]< 0.0010.937EQIP50.00 [46.25, 55.00]55.00 [50.00, 55.00]50.00 [45.00, 55.00]50.00 [50.00, 55.00]55.00 [50.00, 60.00]< 0.0010.270GQS3.00 [3.00, 4.00]3.00 [3.00, 4.00]3.00 [3.00, 3.00]3.00 [3.00, 4.00]3.50 [3.00, 4.00]0.0020.103JAMA1.00 [0.00, 2.00]1.00 [1.00, 2.00]1.00 [0.00, 1.00]1.00 [0.00, 2.00]1.50 [1.00, 2.00]0.0020.098Note: Values are presented as median [Q1, Q3]. Overall P values were obtained using Friedman tests. Kendall’s W represents the effect size for the Friedman test Table 1Comparative analysis of reliability, structural quality, and transparency scores for educational content on complications in patients undergoing maintenance hemodialysis across five LLMsMetricChatGPTCopilotDeepSeekGeminiPerplexityOverall PKendall’s WDISCERN36.00 [34.25, 37.00]43.00 [41.25, 46.00]33.00 [32.00, 35.00]39.50 [38.25, 41.00]45.00 [43.00, 47.75]< 0.0010.937EQIP50.00 [46.25, 55.00]55.00 [50.00, 55.00]50.00 [45.00, 55.00]50.00 [50.00, 55.00]55.00 [50.00, 60.00]< 0.0010.270GQS3.00 [3.00, 4.00]3.00 [3.00, 4.00]3.00 [3.00, 3.00]3.00 [3.00, 4.00]3.50 [3.00, 4.00]0.0020.103JAMA1.00 [0.00, 2.00]1.00 [1.00, 2.00]1.00 [0.00, 1.00]1.00 [0.00, 2.00]1.50 [1.00, 2.00]0.0020.098Note: Values are presented as median [Q1, Q3]. Overall P values were obtained using Friedman tests. Kendall’s W represents the effect size for the Friedman test Comparative analysis of reliability, structural quality, and transparency scores for educational content on complications in patients undergoing maintenance hemodialysis across five LLMs Note: Values are presented as median [Q1, Q3]. Overall P values were obtained using Friedman tests. Kendall’s W represents the effect size for the Friedman test Fig. 1Mean reliability scores of AI chatbots in answering queries about complications in patients undergoing maintenance hemodialysis Mean reliability scores of AI chatbots in answering queries about complications in patients undergoing maintenance hemodialysis Readability analysis Readability differed significantly across models for all six indices using Friedman tests: ARI (P < 0.001, Kendall’s W = 0.272), CL (P < 0.001, Kendall’s W = 0.348), FKGL (P < 0.001, Kendall’s W = 0.347), GFI (P < 0.001, Kendall’s W = 0.403), SMOG (P < 0.001, Kendall’s W = 0.220), and FRES (P < 0.001, Kendall’s W = 0.435) (Table 2). None of the generated outputs met the recommended sixth-grade readability benchmark. For FRES, all models remained below the recommended 80–90 range, whereas the remaining readability indices generally corresponded to high-school or college-level reading difficulty. The comparison between model readability scores and the recommended sixth-grade benchmark is shown in Fig. 2. DeepSeek showed relatively lower reading difficulty on several indices, including FKGL and GFI, while Perplexity and ChatGPT showed higher linguistic complexity on several measures. However, these differences did not alter the overall finding that all model outputs exceeded recommended readability levels for patient-facing educational materials. Significant post-hoc pairwise comparisons for readability metrics are summarized in Supplementary Table 6, with Hodges–Lehmann median differences and 95% confidence intervals reported as paired effect estimates. Complete pairwise comparisons for all non-binary outcomes are provided in Supplementary Table 7. To further illustrate question-level heterogeneity, Fig. 3 presents heatmap visualizations of reliability and readability scores for each individual hemodialysis complication–related question across the five models. Table 2Quantitative readability metrics of AI-generated content on complications in patients undergoing maintenance hemodialysis based on standardized formulaeMetricChatGPTCopilotDeepSeekGeminiPerplexityOverall PKendall’s WARI14.81 ± 2.62; 14.37 [12.94, 15.79]14.37 ± 2.53; 14.46 [12.58, 15.95]12.99 ± 2.01; 13.02 [11.94, 14.22]15.52 ± 3.40; 15.12 [13.20, 16.10]16.26 ± 2.43; 16.58 [14.28, 17.73]< 0.0010.272CL17.83 ± 3.32; 17.21 [15.77, 19.23]16.76 ± 2.85; 16.71 [14.56, 19.36]14.84 ± 2.36; 14.86 [13.55, 16.17]17.41 ± 2.64; 17.16 [15.61, 18.99]18.74 ± 2.19; 18.47 [16.91, 20.47]< 0.0010.348FKGL13.59 ± 2.87; 13.11 [12.04, 14.77]12.61 ± 2.68; 12.90 [10.50, 14.40]11.38 ± 2.16; 11.62 [9.90, 12.79]13.58 ± 3.12; 13.12 [11.63, 14.79]14.98 ± 2.29; 15.26 [13.17, 16.39]< 0.0010.347GFI16.46 ± 3.38; 16.25 [14.30, 18.40]14.64 ± 2.39; 14.35 [13.12, 16.30]12.95 ± 2.26; 12.75 [11.90, 14.30]14.85 ± 2.38; 14.30 [13.15, 16.27]16.15 ± 2.53; 16.30 [14.43, 17.98]< 0.0010.403SMOG10.28 ± 1.37; 10.53 [9.16, 11.01]10.01 ± 1.49; 10.04 [8.99, 10.69]9.76 ± 1.27; 9.68 [8.99, 10.55]10.89 ± 1.98; 10.50 [9.57, 11.34]11.65 ± 1.69; 11.37 [10.42, 13.07]< 0.0010.220FRES18.74 ± 15.36; 19.00 [4.25, 27.00]24.19 ± 17.74; 22.00 [11.25, 36.00]37.21 ± 12.09; 36.00 [26.25, 46.50]22.43 ± 15.65; 23.00 [8.25, 34.50]12.71 ± 12.40; 11.00 [0.00, 24.75]< 0.0010.435Note: Values are mean ± SD; median [Q1, Q3]. For FRES, higher scores indicate easier readability; for ARI, CL, FKGL, GFI and SMOG, higher scores indicate greater reading difficulty Table 2Quantitative readability metrics of AI-generated content on complications in patients undergoing maintenance hemodialysis based on standardized formulaeMetricChatGPTCopilotDeepSeekGeminiPerplexityOverall PKendall’s WARI14.81 ± 2.62; 14.37 [12.94, 15.79]14.37 ± 2.53; 14.46 [12.58, 15.95]12.99 ± 2.01; 13.02 [11.94, 14.22]15.52 ± 3.40; 15.12 [13.20, 16.10]16.26 ± 2.43; 16.58 [14.28, 17.73]< 0.0010.272CL17.83 ± 3.32; 17.21 [15.77, 19.23]16.76 ± 2.85; 16.71 [14.56, 19.36]14.84 ± 2.36; 14.86 [13.55, 16.17]17.41 ± 2.64; 17.16 [15.61, 18.99]18.74 ± 2.19; 18.47 [16.91, 20.47]< 0.0010.348FKGL13.59 ± 2.87; 13.11 [12.04, 14.77]12.61 ± 2.68; 12.90 [10.50, 14.40]11.38 ± 2.16; 11.62 [9.90, 12.79]13.58 ± 3.12; 13.12 [11.63, 14.79]14.98 ± 2.29; 15.26 [13.17, 16.39]< 0.0010.347GFI16.46 ± 3.38; 16.25 [14.30, 18.40]14.64 ± 2.39; 14.35 [13.12, 16.30]12.95 ± 2.26; 12.75 [11.90, 14.30]14.85 ± 2.38; 14.30 [13.15, 16.27]16.15 ± 2.53; 16.30 [14.43, 17.98]< 0.0010.403SMOG10.28 ± 1.37; 10.53 [9.16, 11.01]10.01 ± 1.49; 10.04 [8.99, 10.69]9.76 ± 1.27; 9.68 [8.99, 10.55]10.89 ± 1.98; 10.50 [9.57, 11.34]11.65 ± 1.69; 11.37 [10.42, 13.07]< 0.0010.220FRES18.74 ± 15.36; 19.00 [4.25, 27.00]24.19 ± 17.74; 22.00 [11.25, 36.00]37.21 ± 12.09; 36.00 [26.25, 46.50]22.43 ± 15.65; 23.00 [8.25, 34.50]12.71 ± 12.40; 11.00 [0.00, 24.75]< 0.0010.435Note: Values are mean ± SD; median [Q1, Q3]. For FRES, higher scores indicate easier readability; for ARI, CL, FKGL, GFI and SMOG, higher scores indicate greater reading difficulty Quantitative readability metrics of AI-generated content on complications in patients undergoing maintenance hemodialysis based on standardized formulae Note: Values are mean ± SD; median [Q1, Q3]. For FRES, higher scores indicate easier readability; for ARI, CL, FKGL, GFI and SMOG, higher scores indicate greater reading difficulty Fig. 2Comparison of readability indices for content on complications in patients undergoing maintenance hemodialysis against the recommended sixth-grade benchmark Comparison of readability indices for content on complications in patients undergoing maintenance hemodialysis against the recommended sixth-grade benchmark Fig. 3Heatmap visualization of reliability and readability scores for each individual question regarding complications in patients undergoing maintenance hemodialysis: (a) Reliability; (b) Readability Heatmap visualization of reliability and readability scores for each individual question regarding complications in patients undergoing maintenance hemodialysis: (a) Reliability; (b) Readability Guideline consistency analysis Using the predefined guideline-concordance framework, the assessment revealed clear differences among the five models (Fig. 4). Among guideline-applicable questions, Perplexity and Gemini showed higher factual concordance with the reference guideline statements. Perplexity provided the highest proportion of supplementary information, often adding relevant explanations of pathophysiology and self-management without contradicting guideline recommendations. Copilot showed fewer omission-related issues in the evaluated outputs, suggesting relatively better coverage of guideline-relevant information at the text level despite slightly lower overall accuracy. However, no model achieved complete concordance across all evaluated domains. For hemodialysis-specific topics, such as intradialytic hypotension management and vascular access infection control, ChatGPT and DeepSeek sometimes provided vague descriptions or omitted key guideline-recommended elements. These findings suggest that non-retrieval-based models may have difficulty capturing specialty-specific guideline details, whereas retrieval-augmented models may be better able to retrieve and integrate guideline-relevant information. Overall, the results reflect text-level differences in guideline concordance rather than evidence of clinical safety or effectiveness. LLM-generated hemodialysis-related educational texts therefore require expert review before being used in patient-facing educational contexts. Fig. 4Evaluation of guideline consistency across five large language models for responses to hemodialysis complication–related questions. LLM: large language model Evaluation of guideline consistency across five large language models for responses to hemodialysis complication–related questions. LLM: large language model Medical hallucinations and potential text-level safety concerns in model outputs Qualitative review identified several model-generated “medical hallucinations” in responses to complex, hemodialysis-specific questions. In these cases, the models appeared to apply general health advice to the dialysis context without fully accounting for dialysis-related restrictions or the need for professional supervision. These outputs contained potentially unsafe recommendations that could be clinically problematic if followed without professional guidance. Representative examples are shown in Supplementary Tables 9 and were grouped into three categories of potential text-level safety concern. Some outputs included contraindicated self-management suggestions, potentially inappropriate dietary recommendations, or out-of-scope clinical instructions presented as patient self-care. For example, both Perplexity and GPT-4o advised patients to “stand and stretch” to relieve intradialytic leg cramps, a recommendation that could be unsafe during dialysis because standing while connected to extracorporeal circulation may increase the risk of orthostatic symptoms or vascular access complications. GPT-4o also suggested sipping an electrolyte-containing drink for mild dehydration, which may be inappropriate for patients with kidney failure because such beverages can contain potassium or phosphorus and should not be recommended without individualized dietary assessment. In another example, Copilot referred to rapid intravenous saline infusion as an emergency step for hypotension during dialysis; although this intervention may be used in clinical practice, it requires professional assessment and supervision and should not be framed as independent self-care. Together, these examples suggest that some LLM-generated responses may not adequately distinguish general health advice from dialysis-specific restrictions and professional boundaries, reinforcing the need for expert review before such texts are used in patient-facing educational materials. Discussion This study evaluated single-turn educational responses on hemodialysis complications generated by five LLMs. After accounting for the paired question-level structure of the data, clear differences were observed in information quality, readability, and guideline concordance. The strongest model effect was found for DISCERN (Kendall’s W = 0.937), indicating substantial variation in information reliability. In contrast, the effects for EQIP, GQS, and JAMA were smaller, suggesting more modest differences in structural quality, overall usefulness, and source transparency. Overall, RAG-based models, particularly Perplexity and Copilot, showed stronger information reliability, whereas non-retrieval-based models more often produced less specific or less fully referenced responses. The reliability results suggest a potential advantage of retrieval-augmented generation in producing hemodialysis-related educational texts. Perplexity achieved the highest mean and median scores across DISCERN, EQIP, JAMA, and GQS, while Copilot also performed well, especially on DISCERN and EQIP. This pattern may reflect the ability of retrieval-supported systems to incorporate external sources and generate more structured, guideline-relevant responses. Similar advantages of retrieval-supported chatbots have been reported in other medical fields, including gynecology and infectious diseases [42, 52]. However, the relatively small effects for GQS and JAMA indicate that retrieval does not automatically ensure superior transparency, flow, or patient-facing usefulness across all domains. Readability remained a major limitation. Although the models differed significantly across all six readability indices, none of the outputs met the sixth-grade reading benchmark recommended for patient-facing health materials [53]. DeepSeek showed relatively lower reading difficulty on several indices, whereas Perplexity and ChatGPT tended to produce more linguistically complex responses. Even so, all models generated texts above the recommended readability threshold. These findings indicate that model selection alone is insufficient; LLM-generated materials require plain-language adaptation before patient-facing use [54]. Prior studies have linked limited health literacy with poorer self-management and adverse health-service outcomes in dialysis populations [55–57]. The present study, however, did not assess patient understanding, adherence, readmissions, or other clinical outcomes. Therefore, the readability results should be interpreted as evidence of limited textual accessibility, not as evidence of actual patient comprehension or behavior. Our findings suggest that text-level guideline concordance may vary by model architecture. RAG models demonstrate a clear advantage in following authoritative KDIGO guidelines [15] by dynamically integrating evidence-based information. They accurately specify complex clinical protocols—such as intradialytic hypotension management and vascular access surveillance—and are superior in clarifying core clinical thresholds [58]. However, non-retrieval-based models, including ChatGPT and DeepSeek, sometimes omitted key details or provided vague descriptions for hemodialysis-specific scenarios [51]. These informational gaps suggest that current LLM-generated texts should not be used as autonomous patient-facing educational or advisory materials without expert review. The qualitative review of medical hallucinations provided further insight into the types of text-level problems that may occur in specialty-specific contexts [51]. Some models appeared to overgeneralize basic health advice to hemodialysis patients without adequately accounting for dialysis-related restrictions, such as impaired potassium excretion, hemodynamic instability, extracorporeal circulation, or the need for professional supervision during intradialytic interventions. For example, recommendations to consume electrolyte-containing drinks or to stand during dialysis may be inappropriate in this population and could raise safety concerns if followed without individualized clinical or dietary guidance [59]. In addition, some outputs presented clinician-performed procedures, such as intravenous saline infusion, as if they were patient self-care measures, which may blur the boundary between patient education and professionally supervised treatment. These findings suggest that current LLM-generated texts may not reliably distinguish general health advice from hemodialysis-specific restrictions. Therefore, the present text-output evaluation does not support the independent use of these responses for patient education without professional review and contextual adaptation [60]. Despite these limitations, LLMs may still have a role in nephrology education. The findings do not support unsupervised patient-facing use, but they suggest that LLMs could assist clinicians in drafting educational outlines, summarizing complex pathophysiological concepts, or revising technical content into more accessible language. In such a supervised workflow, LLMs would support communication rather than replace clinical judgment, personalization, or professional responsibility. Future research should address both technical optimization and empirical evaluation. Technically, expert-supervised fine-tuning with nephrology-specific, high-quality datasets and RAG integration may improve alignment with authoritative guidelines [61–62]. Readability-aware generation is also needed to help LLMs adapt technical medical content into patient-centered educational language [61–62]. Empirically, future studies should move beyond text-output assessment and evaluate whether AI-assisted education affects patient comprehension, self-management behavior, treatment adherence, health-service utilization, and clinical outcomes [63–64]. Such evidence is needed before drawing conclusions about the real-world safety or effectiveness of LLM use in nephrology [63–64]. This study has several limitations. First, although the 42 prompts covered common hemodialysis-related complications, the sample size may have limited the detection of smaller performance differences among models. Second, because public web interfaces were used, inference parameters such as temperature could not be fixed, which may have introduced output variability [65–66]. Although default chatbot interfaces helped approximate how lay users access publicly available AI systems [67], the design did not capture the full complexity of real-world patient–chatbot interactions. To minimize context carryover and improve comparability, caches, cookies, login sessions, and prior conversation history were cleared before each query, and each prompt was tested as an independent single-turn, zero-shot interaction. This standardization improved comparability across models but reduced ecological realism because real patients may ask follow-up questions, provide personal medical information, and engage in multi-turn conversations. Thus, the findings should be interpreted as evidence of text-level quality, readability, and guideline-concordance limitations rather than evidence of clinical safety or effectiveness. Conclusion In this cross-sectional evaluation of single-turn text outputs, RAG-based models showed relatively higher information reliability and guideline concordance in generating educational responses on hemodialysis complications. However, all evaluated LLMs produced outputs with readability barriers, incomplete coverage of some guideline-recommended elements, and occasional potentially unsafe or out-of-scope recommendations at the text level. These findings indicate that LLM-generated hemodialysis educational texts require expert review and plain-language adaptation before patient-facing use. Because this study did not assess multi-turn patient–chatbot interactions, patient understanding, behavior change, treatment adherence, adverse events, or clinical outcomes, the results should be interpreted as text-level evidence of quality and safety-related concerns rather than evidence of clinical safety or effectiveness. Supplementary Information Below is the link to the electronic supplementary material. Supplementary Material 1 Supplementary Material 1