Multidimensional assessment of large language model responses to patient questions on gestational diabetes mellitus Authors: Betul Yigit Yalcın, Ummu Mutlu, Ayse Merve Ok, Ozge Telci Caklili, Hulya Hacisahinogullari, Gulsah Yenidunya Yalin, Ozlem Soyluk Selcukbiricik, Ayse Kubat Uzum, Kubilay Karsidag Published: 2025 ABSTRACT Gestational diabetes mellitus (GDM) is a prevalent condition requiring accurate patient education, yet the reliability and readability of large language models (LLMs) in this context remain uncertain. This study evaluated the performance of four LLMs—ChatGPT-4o, Gemini 2.5 Pro, Grok 3.0, and DeepSeek R-1—using 25 patient-oriented questions derived from clinical scenarios. Seven endocrinologists independently rated the responses with the modified DISCERN (mDISCERN) instrument and the Global Quality Score (GQS). Readability was analyzed using the Flesch Reading Ease (FRES), Flesch–Kincaid Grade Level (FKGL), Gunning Fog Index (GFI), Coleman–Liau Index (CLI), and Simple Measure of Gobbledygook (SMOG), while lexical diversity was assessed through type–token ratio (TTR). Grok and Gemini obtained the highest mDISCERN and GQS scores, whereas ChatGPT performed significantly lower (p < 0.05). DeepSeek generated the most readable outputs, while Grok provided the longest and most complex responses. All models scored below the FRES threshold of 60 recommended for lay audiences. Response length showed strong positive correlations with mDISCERN and GQS, while TTR was inversely related to quality but positively associated with readability. These findings highlight variability among LLMs in GDM education and emphasize the need for model-specific improvements to ensure reliable patient-facing health information.Supplementary InformationThe online version contains supplementary material available at 10.1038/s41598-025-27235-y. Introduction Gestational diabetes mellitus (GDM) is defined as hyperglycemia that develops for the first time during pregnancy. Although its prevalence is uncertain due to variations in screening protocols and diagnostic criteria worldwide, it is estimated to affect approximately 14% of pregnancies1– 2. Uncontrolled GDM is associated with significant maternal and neonatal complications, including preeclampsia, cesarean delivery, macrosomia, preterm birth, and neonatal hospitalization. Additionally, it substantially increases the mother’s long-term risk of developing type 2 diabetes3–5. Effective GDM management requires accurate patient education, appropriate initiation of pharmacotherapy when needed, and close monitoring of adherence6. However, time constraints in clinical practice and health literacy barriers often limit the effectiveness of traditional patient education strategies. These challenges have led to growing interest in innovative digital health solutions that can improve patient engagement and fill the knowledge gaps in GDM care. Among these solutions, artificial intelligence-based approaches are also being increasingly studied. In recent years, large language models (LLMs), as part of artificial intelligence technologies, have been evaluated as a potential tool in patient education, as in many other fields7. These models generate contextualized, human-like responses and may serve as a scalable, cost-effective means of delivering personalized, evidence-based health information, thereby improving health literacy and access to care in resource-limited settings. However, the accuracy, guideline-concordance, and reliability of LLM-generated content for pregnancy-specific patient questions have not been systematically evaluated. Inaccurate or misleading information generated by artificial intelligence may undermine the patient–physician trust relationship, leading to adverse clinical outcomes8. Given the critical need for clear, evidence-based advice on lifestyle modification, blood glucose monitoring, and therapy in GDM, evaluating LLMs’ pregnancy-specific accuracy and safety is essential in this context9. This study aims to systematically and quantitatively evaluate the accuracy, reliability, and clinical appropriateness of responses provided by state-of-the-art LLM systems to frequently asked questions from GDM patients and healthcare professionals. Materials and methods Study design This study was designed as a cross-sectional, observational analysis, conducted in accordance with the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) guidelines10,11. Although formatted according to STROBE for general transparency, the design of this study conceptually aligns with the Chatbot Assessment Reporting Tool (CHART)12, which provides a more suitable approach for the evaluation of AI systems. The responses provided by the ChatGPT-4o, DeepSeek R-1, Gemini 2.5 Pro and Grok-3.0 models to patient scenario–based questions regarding GDM were systematically evaluated in terms of readability, text-level structural criteria, scientific reliability, and educational quality. Development of GDM questions Questions related to the management of GDM were developed by five endocrinology specialists with over 10 years of clinical experience. Current guidelines and relevant original studies were identified through a structured literature review performed in the PubMed, Scopus, and Google Scholar databases, using the keywords “gestational diabetes mellitus,” “pregnancy glucose intolerance,” “medical nutrition therapy in pregnancy,” “insulin therapy during pregnancy,” “antenatal care,” and “postpartum diabetes screening.” Initially, 32 open-ended patient scenarios were created covering all stages of care, including screening, diagnosis, glycemic targets, lifestyle modifications, pharmacological treatment, intrapartum management, and postpartum follow-up. The questions were developed by clinicians and written in a clear, comprehensible, and clinically detailed manner to reflect how patients typically express their concerns to physicians; they were also subjected to editorial review for grammatical accuracy and consistency of expression. All questions and AI-generated responses were prepared, delivered, and evaluated in English; therefore, standard readability formulas developed for English texts were directly applicable without translation. Content validity assessment and final question set To quantitatively assess the content validity of the questions, the Content Validity Index (CVI) method was applied13,14. Five independent experts with experience in GDM management scored each question on a 1 − 4 Likert scale (1 = “not appropriate at all,” 4 = “completely appropriate”). Each expert had over ten years of clinical experience in gestational diabetes care, including antenatal and perinatal endocrinology clinics, and some had contributed to the development of national diabetes mellitus guideline and others had participated in multicenter research studies on this field. The item-level CVI (I-CVI) value for each item was calculated as the ratio of the number of experts who gave a score of 3 or 4 to the total number of experts. I-CVI ≥ 0.79: The question was accepted without modification. 0.70 ≤ I-CVI < 0.79: The question was revised in terms of wording and content. I-CVI < 0.70: The question was excluded. As a result of the evaluation, seven questions were excluded due to low validity scores, and five questions were revised based on expert feedback. Consequently, a final set of 25 open-ended questions was established. The excluded items were primarily highly technical or overlapping in content (e.g., insulin dose titration, steroid administration, comorbidity-related target adjustments, or twin pregnancies) and were therefore deemed unsuitable for general patient education purposes. This set of questions was divided into six thematic sections (Table 1). The experts who conducted the content evaluation were not involved in the subsequent quality assessment of AI-generated responses, ensuring the impartiality of the evaluation process and minimizing potential bias. Table 1Thematic classification of the gestational diabetes mellitus (GDM) question set.SectionQuestion countThematic focusRepresentative questionA. Definition and Screening/Diagnosis5What GDM is, underlying causes, screening-diagnosis protocols, safety of the OGTT, and alternative tests“What exactly is gestational diabetes mellitus (GDM) and what might be the reasons I developed GDM during my pregnancy?”B. Maternal-Fetal Risks and Prognosis2Effects on mother and fetus, consequences of delayed treatment“What might happen if I delay starting treatment for gestational diabetes?”C. Glycemic Monitoring and Targets5Daily blood-glucose testing frequency, target ranges, glycemic variability, managing hyper- and hypoglycemia“I have gestational diabetes. How often should I check my blood sugar each day, and what target levels should I aim for?”D. Lifestyle and Dietary Management4Medical nutrition therapy, meal-and-snack planning, recommended exercise types and frequency, diet-and-exercise–only control“How should I plan my meals with gestational diabetes, and which foods should I limit or avoid?”E. Pharmacologic Therapy and Safety4Insulin and oral agents, fetal safety, possible side-effects, safety during breastfeeding“Is insulin the only treatment option for gestational diabetes, or can I use diabetes pills instead?”F. Delivery and Post-partum Follow-up5Mode-of-delivery decisions, post-partum glucose surveillance, long-term type 2 diabetes risk, recurrence in future pregnancies, breastfeeding considerations“Can I have a vaginal birth if I have gestational diabetes mellitus, or will I need a C-section?”GDM, gestational diabetes mellitus; OGTT, oral glucose tolerance test. Table 1Thematic classification of the gestational diabetes mellitus (GDM) question set.SectionQuestion countThematic focusRepresentative questionA. Definition and Screening/Diagnosis5What GDM is, underlying causes, screening-diagnosis protocols, safety of the OGTT, and alternative tests“What exactly is gestational diabetes mellitus (GDM) and what might be the reasons I developed GDM during my pregnancy?”B. Maternal-Fetal Risks and Prognosis2Effects on mother and fetus, consequences of delayed treatment“What might happen if I delay starting treatment for gestational diabetes?”C. Glycemic Monitoring and Targets5Daily blood-glucose testing frequency, target ranges, glycemic variability, managing hyper- and hypoglycemia“I have gestational diabetes. How often should I check my blood sugar each day, and what target levels should I aim for?”D. Lifestyle and Dietary Management4Medical nutrition therapy, meal-and-snack planning, recommended exercise types and frequency, diet-and-exercise–only control“How should I plan my meals with gestational diabetes, and which foods should I limit or avoid?”E. Pharmacologic Therapy and Safety4Insulin and oral agents, fetal safety, possible side-effects, safety during breastfeeding“Is insulin the only treatment option for gestational diabetes, or can I use diabetes pills instead?”F. Delivery and Post-partum Follow-up5Mode-of-delivery decisions, post-partum glucose surveillance, long-term type 2 diabetes risk, recurrence in future pregnancies, breastfeeding considerations“Can I have a vaginal birth if I have gestational diabetes mellitus, or will I need a C-section?”GDM, gestational diabetes mellitus; OGTT, oral glucose tolerance test. Thematic classification of the gestational diabetes mellitus (GDM) question set. GDM, gestational diabetes mellitus; OGTT, oral glucose tolerance test. Response collection Between June and July 2025, responses were obtained from ChatGPT-4o, DeepSeek R-1, Gemini 2.5 Pro, and Grok-3.0. All queries were submitted in a single session on an Apple MacBook Air (M2, 2022) with 8 GB RAM, operating on macOS Ventura 13.7.6 and connected to a 100 Mbps fixed-line network. All models were presented with the same set of questions in an identical sequence to ensure comparability of responses. The “reproduce” function was not used, and only the first response was recorded to avoid variability arising from the probabilistic nature of LLM outputs. All responses were generated using the default settings of each platform, as user-level control of hyperparameters such as temperature, top-p, or max tokens was not available in their standard web interfaces. Consequently, 25 input questions were generated and presented to each model (Supplementary File 1). Each LLM produced 25 responses, yielding a total of 100 outputs (Supplementary File 2). The overall study workflow—from patient-style question generation to multi-dimensional evaluation of LLM outputs—is summarized in Fig. 1. Fig. 1Study methodology. Gestational diabetes questions were posed to multiple AI models, and the anonymized responses were evaluated for quality and readability by a panel of endocrinologists. Study methodology. Gestational diabetes questions were posed to multiple AI models, and the anonymized responses were evaluated for quality and readability by a panel of endocrinologists. Reliability and quality assessment Seven endocrinologists with expertise in GDM independently evaluated the model-generated responses according to the current ADA guidelines and established clinical practice as reference standards. None of these reviewers were involved in question development. Before formal scoring, they completed a brief calibration session to ensure consistent interpretation of criteria. To reduce bias, the identity of each LLM was concealed, and responses were anonymized as Model A, Model B, Model C, and Model D. Each response was evaluated for scientific accuracy, clinical appropriateness, and comprehensibility. The primary appraisal tool was the DISCERN instrument, a widely used scale for assessing the reliability and content quality of online health information15,16. DISCERN consists of three sections: Part 1 (8 items) examines the reliability of the information; Part 2 (7 items) assesses the quality of treatment information; and Part 3 (1 item) measures overall usefulness as a treatment resource15,16. Since not all questions were treatment-focused, only Part 1 was applied, as has also been done in previous studies, and evaluated in its modified form (mDISCERN). Items were scored on a 5-point scale (1 = No, 2–4 = Partly, 5 = Yes); total scores were categorized as 8–15 = low, 16–31 = moderate, and 32–40 = high reliability9,16,17. The overall quality and presentation of the responses were then assessed using the single-item Global Quality Scale (GQS) (1–2 = low quality, 3 = moderate quality, and 4–5 = high quality, Table 2)18. GQS ratings were independently performed by seven trained evaluators following a calibration session to standardize scoring criteria. The final GQS score for each response was calculated as the mean of all seven evaluators’ ratings. The inter-rater reliability of total DISCERN scores was assessed using the Intraclass Correlation Coefficient (ICC). Although the mDISCERN scores were not perfectly normally distributed, the total scores were treated as quasi-continuous variables derived from summed ordinal items. Therefore, ICC was retained, as it has been shown to be robust to moderate deviations from normality, particularly in studies with moderate-to-large sample sizes19,20. These metrics provided an objective comparison of how effectively each model answered patient-centred questions on GDM9,21,22. Table 2Overview of mDISCERN, GQS, and readability evaluation criteria.mDISCERN CriteriaScoreAre the aims clear?1–5 pointsDoes it achieve its aims?1–5 pointsIs it relevant?1–5 pointsIs it clear what sources of information were used to compile the publication (other than the author or producer)?1–5 pointsIs it clear when the information used or reported in the publication was produced?1–5 pointsIs it balanced and unbiased?1–5 pointsDoes it provide details of additional sources of support and information?1–5 pointsDoes it refer to areas of uncertanity?1–5 pointsGlobal Quality Scale (GQS)ScoreMeasures the practical guidance value of the response for GDM managementVery low: Offers virtually no actionable advice; of negligible help to patient management.1 pointLow: Omits many key points; guidance is incomplete and of limited clinical use2 pointModerate: Covers the main topics but in shallow detail; only partly helpful for day-to-day care.3 pointGood: Provides most of the essential information; delivers clear, generally reliable recommendations.4 pointExcellent: Supplies comprehensive, step-by-step instructions that can be readily applied in clinical practice.5 pointReadability indices- formulaDescriptionFlesch reading ease score (FRES): 206.835–1015 x (W/S) − 84.6 x (B/W)Produces a score from 0 to 100; higher scores indicate that the text is easier to read.Flesch-Kincaid grade level (FKGL): 0.39 x (W/S) + 11.8 x (B/W) − 15.59Estimates the U.S. school grade level required to comprehend the text.Simple measure of Gobbledygook (SMOG): 1.0430 x √(C) + 3.1291Indicates the educational grade level required for a reader to fully comprehend the text.Gunning fog index (GFI): 0.4 x [(W/S) + 100 x (C*/W)]Higher values indicate greater textual complexity and more advanced reading skills required.Coleman-Liau index (CLI): (0.0588 x L) - (0.296 x S*) − 15.8Expressed as a grade level, showing the estimated educational stage needed for comprehension.DISCERN = standard reliability checklist; mDISCERN = its 8-item short form; GQS = Global Quality Scale. W = total number of words; S = total number of sentences; B = total number of syllables; C = number of polysyllabic words (three or more syllables); C* = complex words excluding proper nouns, words made polysyllabic by adding “-ed” or “-es”, and compound words made of simpler words; L = average number of letters per 100 words; S* = average number of sentences per 100 words. Scoring rules. mDISCERN total 8–40 → ≤ 15 = low, 16–31 = moderate, ≥ 32 = high reliability; each item: 1 = No, 2–4 = Partial, 5 = Yes. GQS ranges 1–5 → 1–2 = low, 3 = moderate, 4–5 = high practical quality. Greater values always indicate better performance on the criterion assessed. Table 2Overview of mDISCERN, GQS, and readability evaluation criteria.mDISCERN CriteriaScoreAre the aims clear?1–5 pointsDoes it achieve its aims?1–5 pointsIs it relevant?1–5 pointsIs it clear what sources of information were used to compile the publication (other than the author or producer)?1–5 pointsIs it clear when the information used or reported in the publication was produced?1–5 pointsIs it balanced and unbiased?1–5 pointsDoes it provide details of additional sources of support and information?1–5 pointsDoes it refer to areas of uncertanity?1–5 pointsGlobal Quality Scale (GQS)ScoreMeasures the practical guidance value of the response for GDM managementVery low: Offers virtually no actionable advice; of negligible help to patient management.1 pointLow: Omits many key points; guidance is incomplete and of limited clinical use2 pointModerate: Covers the main topics but in shallow detail; only partly helpful for day-to-day care.3 pointGood: Provides most of the essential information; delivers clear, generally reliable recommendations.4 pointExcellent: Supplies comprehensive, step-by-step instructions that can be readily applied in clinical practice.5 pointReadability indices- formulaDescriptionFlesch reading ease score (FRES): 206.835–1015 x (W/S) − 84.6 x (B/W)Produces a score from 0 to 100; higher scores indicate that the text is easier to read.Flesch-Kincaid grade level (FKGL): 0.39 x (W/S) + 11.8 x (B/W) − 15.59Estimates the U.S. school grade level required to comprehend the text.Simple measure of Gobbledygook (SMOG): 1.0430 x √(C) + 3.1291Indicates the educational grade level required for a reader to fully comprehend the text.Gunning fog index (GFI): 0.4 x [(W/S) + 100 x (C*/W)]Higher values indicate greater textual complexity and more advanced reading skills required.Coleman-Liau index (CLI): (0.0588 x L) - (0.296 x S*) − 15.8Expressed as a grade level, showing the estimated educational stage needed for comprehension.DISCERN = standard reliability checklist; mDISCERN = its 8-item short form; GQS = Global Quality Scale. W = total number of words; S = total number of sentences; B = total number of syllables; C = number of polysyllabic words (three or more syllables); C* = complex words excluding proper nouns, words made polysyllabic by adding “-ed” or “-es”, and compound words made of simpler words; L = average number of letters per 100 words; S* = average number of sentences per 100 words. Scoring rules. mDISCERN total 8–40 → ≤ 15 = low, 16–31 = moderate, ≥ 32 = high reliability; each item: 1 = No, 2–4 = Partial, 5 = Yes. GQS ranges 1–5 → 1–2 = low, 3 = moderate, 4–5 = high practical quality. Greater values always indicate better performance on the criterion assessed. Overview of mDISCERN, GQS, and readability evaluation criteria. DISCERN = standard reliability checklist; mDISCERN = its 8-item short form; GQS = Global Quality Scale. W = total number of words; S = total number of sentences; B = total number of syllables; C = number of polysyllabic words (three or more syllables); C* = complex words excluding proper nouns, words made polysyllabic by adding “-ed” or “-es”, and compound words made of simpler words; L = average number of letters per 100 words; S* = average number of sentences per 100 words. Scoring rules. mDISCERN total 8–40 → ≤ 15 = low, 16–31 = moderate, ≥ 32 = high reliability; each item: 1 = No, 2–4 = Partial, 5 = Yes. GQS ranges 1–5 → 1–2 = low, 3 = moderate, 4–5 = high practical quality. Greater values always indicate better performance on the criterion assessed. Readability and text feature analysis The comprehensibility and linguistic structure of each response were evaluated using established readability metrics (Flesch Reading Ease (FRES), Flesch–Kincaid Grade Level (FKGL), Gunning Fog Index (GFI), Coleman–Liau Index (CLI), and Simple Measure of Gobbledygook (SMOG))23–25. FRES reflects overall textual fluency (0–100; higher = easier to read), whereas FKGL, GFI and CLI estimate the U.S. school-grade level required for comprehension; SMOG estimates the number of years of education needed to fully understand the text. For general readers, we predefined an optimal profile as FRES > 60 and grade-level scores < 7 on the other indices25–29. Lexical density was calculated from word, sentence, and character counts. All analyses were performed in R (version 2024) and results were compared across the four models; formulas and reference thresholds are presented in Table 2. Statistical analysis Statistical analyses were performed using IBM SPSS Statistics version 27.0 (IBM Corp., Armonk, NY, USA) and Python 3.10 (textstat package) for readability assessments. Readability scores, including Flesch Reading Ease Score (FRES), Flesch–Kincaid Grade Level (FKGL), Gunning Fog Index (GFI), SMOG Index, and Coleman–Liau Index (CLI), were automatically computed. There were no missing data in the dataset. Differences in readability and quality scores among the platforms were examined using the Kruskal–Wallis test, given the non-normal distribution of the data. The use of nonparametric procedures ensured the robustness of statistical inferences. When the omnibus test indicated a statistically significant difference, pairwise comparisons were performed using Dunn’s post-hoc method to identify the specific group contrasts. Correlation analyses were conducted using Spearman’s rho, and Bonferroni correction was applied to control for multiple comparisons. Statistical significance was set at two-tailed p < 0.05 after correction. Ethical considerations Since the study did not include any procedures involving human participants or experimental work on animals, formal ethics committee approval was not required. However, ethical issues related to the use of AI-supported tools in the context of medical education have been taken into consideration, and data confidentiality, impartiality, and scientific ethical principles have been strictly adhered in the study design. Results mDISCERN and GQS outcomes In total, 100 answers produced by four distinct LLMs—ChatGPT, Gemini, Grok, and DeepSeek—were independently reviewed by seven endocrinologists. Each evaluator applied the mDISCERN tool and the GQS, resulting in a dataset of 700 separate ratings (25 questions × 4 LLMs × 7 reviewers). Grok (29.01 ± 6.01) and Gemini (28.49 ± 4.36) achieved the highest mean mDISCERN scores, whereas ChatGPT scored lowest (25.40 ± 4.18). According to the predefined classification (8–15 = low, 16–31 = moderate, 32–40 = high), the mean mDISCERN scores of all four models fell within the moderate reliability category. For GQS, Gemini (4.41 ± 0.66) and Grok (4.27 ± 0.90) received the highest mean ratings, ChatGPT again ranked the lowest mean score (3.86 ± 0.76) (Table 3). Table 3Descriptive quality scores by platform.PlatformDISCERN mean ± SDGQS mean ± SDChatGPT25.40 ± 4.183.86 ± 0.76DeepSeek27.23 ± 4.554.21 ± 0.76Gemini28.49 ± 4.364.41 ± 0.66Grok29.01 ± 6.014.27 ± 0.90Explanation: Mean ± SD values of modified DISCERN total and Global Quality Score (GQS) across four large language models; n = 700 ratings (25 answers × 4 LLMs × 7 raters). Table 3Descriptive quality scores by platform.PlatformDISCERN mean ± SDGQS mean ± SDChatGPT25.40 ± 4.183.86 ± 0.76DeepSeek27.23 ± 4.554.21 ± 0.76Gemini28.49 ± 4.364.41 ± 0.66Grok29.01 ± 6.014.27 ± 0.90Explanation: Mean ± SD values of modified DISCERN total and Global Quality Score (GQS) across four large language models; n = 700 ratings (25 answers × 4 LLMs × 7 raters). Descriptive quality scores by platform. Explanation: Mean ± SD values of modified DISCERN total and Global Quality Score (GQS) across four large language models; n = 700 ratings (25 answers × 4 LLMs × 7 raters). Post-hoc analyses indicated that ChatGPT scored significantly lower than all other models on both mDISCERN and GQS, while Grok outperformed DeepSeek on mDISCERN but not on GQS (Supplementary Table 1). Item-level mDISCERN scores (D1–D8) are provided in Supplementary Table 2. Score distributions across models are illustrated in Fig. 2. Inter-rater reliability for mDISCERN was good, with ICC (A,7) = 0.71(95% CI, 0.55–0.82). Fig. 2Quality outcomes of LLM responses. Mean ± SD values of modified DISCERN total scores (Panel A) and Global Quality Scores (GQS) (Panel B) across four large language models (LLMs). Quality outcomes of LLM responses. Mean ± SD values of modified DISCERN total scores (Panel A) and Global Quality Scores (GQS) (Panel B) across four large language models (LLMs). Readability and text structure outcomes Significant differences were observed among the four LLMs in both readability and structural metrics. Evaluation was performed across nine indicators—FRES, FKGL, SMOG, CLI, GFI, TTR (type–token ratio), along with total words, sentences, and characters—using the Kruskal–Wallis test (p < 0.01 for all). Summary statistics are presented in Table 4, and pairwise contrasts are reported in Supplementary Table 3. DeepSeek achieved the best performance in terms of readability (FRES = 49.5 ± 7.7; FKGL = 11.0 ± 1.7), whereas Grok produced significantly longer and more complex texts (FRES = 27.7 ± 11.5; FKGL = 17.1 ± 3.5). ChatGPT and Gemini showed intermediate readability; while ChatGPT produced the shortest responses. All models had FRES values < 60, indicating that readability was below the recommended threshold for materials intended for a general audience. Descriptive statistics are shown in Table 4; Fig. 3. Table 4Mean ± SD values of readability indices for each platform.PlatformFRESFKGLSMOGGFICLITTRWordsChatGPT39.42 ± 11.9615.86 ± 3.8315.37 ± 2.3218.94 ± 3.9510.90 ± 1.780.584 ± 0.039487.9 ± 118.8DeepSeek 49.53 ± 7.68 11.04 ± 1.65 12.76 ± 1.42 13.88 ± 1.96 10.77 ± 1.95 0.603 ± 0.037 692.9 ± 104.5Gemini42.64 ± 9.3212.03 ± 1.8914.01 ± 1.6215.33 ± 2.24 12.76 ± 1.47 0.450 ± 0.028931.1 ± 189.9Grok 27.66 ± 11.52 17.06 ± 3.52 17.21 ± 2.29 20.44 ± 3.65 13.71 ± 1.32 0.408 ± 0.080 1 536.1 ± 624.7 FRES, Flesch Reading Ease score; FKGL, Flesch–Kincaid Grade Level; SMOG, Simple Measure of Gobbledygook; GFI, Gunning Fog Index; CLI, Coleman–Liau Index; TTR, type–token ratio. Scores are presented as mean ± standard deviation (SD) for 100 answers (25 questions × 4 large language models). Higher FRES and TTR indicate greater readability and lexical diversity, respectively, whereas higher FKGL, SMOG, GFI, and CLI indicate greater text complexity.Significant values are in bold. Table 4Mean ± SD values of readability indices for each platform.PlatformFRESFKGLSMOGGFICLITTRWordsChatGPT39.42 ± 11.9615.86 ± 3.8315.37 ± 2.3218.94 ± 3.9510.90 ± 1.780.584 ± 0.039487.9 ± 118.8DeepSeek 49.53 ± 7.68 11.04 ± 1.65 12.76 ± 1.42 13.88 ± 1.96 10.77 ± 1.95 0.603 ± 0.037 692.9 ± 104.5Gemini42.64 ± 9.3212.03 ± 1.8914.01 ± 1.6215.33 ± 2.24 12.76 ± 1.47 0.450 ± 0.028931.1 ± 189.9Grok 27.66 ± 11.52 17.06 ± 3.52 17.21 ± 2.29 20.44 ± 3.65 13.71 ± 1.32 0.408 ± 0.080 1 536.1 ± 624.7 FRES, Flesch Reading Ease score; FKGL, Flesch–Kincaid Grade Level; SMOG, Simple Measure of Gobbledygook; GFI, Gunning Fog Index; CLI, Coleman–Liau Index; TTR, type–token ratio. Scores are presented as mean ± standard deviation (SD) for 100 answers (25 questions × 4 large language models). Higher FRES and TTR indicate greater readability and lexical diversity, respectively, whereas higher FKGL, SMOG, GFI, and CLI indicate greater text complexity.Significant values are in bold. Mean ± SD values of readability indices for each platform. FRES, Flesch Reading Ease score; FKGL, Flesch–Kincaid Grade Level; SMOG, Simple Measure of Gobbledygook; GFI, Gunning Fog Index; CLI, Coleman–Liau Index; TTR, type–token ratio. Scores are presented as mean ± standard deviation (SD) for 100 answers (25 questions × 4 large language models). Higher FRES and TTR indicate greater readability and lexical diversity, respectively, whereas higher FKGL, SMOG, GFI, and CLI indicate greater text complexity. Significant values are in bold. Fig. 3Readability metrics across four large language models (LLMs). Bar charts illustrate Flesch Reading Ease Score (FRES) and Flesch–Kincaid Grade Level (FKGL) values (left y-axis), while the line represents type–token ratio (TTR) (right y-axis) for ChatGPT, DeepSeek, Gemini, and Grok. Higher FRES indicates greater readability, whereas higher FKGL reflects increased text complexity. TTR represents lexical diversity, with higher values indicating a greater proportion of unique words in the response. Readability metrics across four large language models (LLMs). Bar charts illustrate Flesch Reading Ease Score (FRES) and Flesch–Kincaid Grade Level (FKGL) values (left y-axis), while the line represents type–token ratio (TTR) (right y-axis) for ChatGPT, DeepSeek, Gemini, and Grok. Higher FRES indicates greater readability, whereas higher FKGL reflects increased text complexity. TTR represents lexical diversity, with higher values indicating a greater proportion of unique words in the response. Post-hoc Dunn tests indicated that DeepSeek scored significantly higher than Grok on readability metrics, while Grok produced significantly longer and more complex responses. DeepSeek and Gemini demonstrated higher lexical diversity relative to the other models. Other significant pairwise comparisons are provided in Supplementary Table 3. Correlation analysis outcomes A strong positive correlation was observed between mDISCERN and GQS (ρ = 0.612, p < 0.001), indicating consistency across the two quality assessment tools. Both of these quality metrics were also significantly positively correlated with the length of responses, including word count (DISCERN: ρ = 0.619; GQS: ρ = 0.472, p < 0.001) and character count (DISCERN: ρ = 0.643; GQS: ρ = 0.511, p < 0.001). Among the readability metrics, the Coleman–Liau Index (CLI) demonstrated a moderate positive correlation with DISCERN (ρ = 0.439, p < 0.001). The type–token ratio (TTR), representing lexical diversity, was inversely correlated with both quality scores (DISCERN: ρ = − 0.484; GQS: ρ = − 0.313, p ≤ 0.002), but positively correlated with readability measures (e.g., TTR–FRES: ρ = 0.404, p = 0.001) (Table 5). Table 5Spearman correlation matrix (ρ) among quality, readability and length.Metrics.GQSDISCERNFRESFKGLSMOGGFICLITTRWordsSentencesCharactersGQS1.000.61–0.01–0.18–0.06–0.160.35–0.310.470.510.51DISCERN0.611.00–0.180.000.130.020.44–0.480.620.620.64FRES–0.01–0.181.00–0.89–0.97–0.92–0.760.40–0.200.02–0.30FKGL–0.180.00–0.891.000.910.980.44–0.310.05–0.260.10SMOG–0.060.13–0.970.911.000.950.68–0.390.15–0.090.24GFI–0.160.02–0.920.980.951.000.51–0.320.07–0.220.13CLI0.350.44–0.760.440.680.511.00–0.530.430.400.57TTR–0.31–0.480.40–0.31–0.39–0.32–0.531.00–0.76–0.71–0.81Words0.470.62–0.200.050.150.070.43–0.761.000.880.97Sentences0.510.620.02–0.26–0.09–0.220.40–0.710.881.000.89Characters0.510.64–0.300.100.240.130.57–0.810.970.891.00GQS, Global Quality Score; DISCERN, modified DISCERN total score; FRES, Flesch Reading Ease Score; FKGL, Flesch–Kincaid Grade Level; SMOG, Simple Measure of Gobbledygook; GFI, Gunning Fog Index; CLI, Coleman–Liau Index; TTR, type–token ratio (lexical diversity). Values represent Spearman’s rank correlation coefficients (ρ). Bold values indicate correlations that remained statistically significant after Bonferroni adjustment (two-tailed; adjusted p < 0.05). Positive correlations indicate that both variables increase together, whereas negative correlations indicate that one variable increases as the other decreases. Table 5Spearman correlation matrix (ρ) among quality, readability and length.Metrics.GQSDISCERNFRESFKGLSMOGGFICLITTRWordsSentencesCharactersGQS1.000.61–0.01–0.18–0.06–0.160.35–0.310.470.510.51DISCERN0.611.00–0.180.000.130.020.44–0.480.620.620.64FRES–0.01–0.181.00–0.89–0.97–0.92–0.760.40–0.200.02–0.30FKGL–0.180.00–0.891.000.910.980.44–0.310.05–0.260.10SMOG–0.060.13–0.970.911.000.950.68–0.390.15–0.090.24GFI–0.160.02–0.920.980.951.000.51–0.320.07–0.220.13CLI0.350.44–0.760.440.680.511.00–0.530.430.400.57TTR–0.31–0.480.40–0.31–0.39–0.32–0.531.00–0.76–0.71–0.81Words0.470.62–0.200.050.150.070.43–0.761.000.880.97Sentences0.510.620.02–0.26–0.09–0.220.40–0.710.881.000.89Characters0.510.64–0.300.100.240.130.57–0.810.970.891.00GQS, Global Quality Score; DISCERN, modified DISCERN total score; FRES, Flesch Reading Ease Score; FKGL, Flesch–Kincaid Grade Level; SMOG, Simple Measure of Gobbledygook; GFI, Gunning Fog Index; CLI, Coleman–Liau Index; TTR, type–token ratio (lexical diversity). Values represent Spearman’s rank correlation coefficients (ρ). Bold values indicate correlations that remained statistically significant after Bonferroni adjustment (two-tailed; adjusted p < 0.05). Positive correlations indicate that both variables increase together, whereas negative correlations indicate that one variable increases as the other decreases. Spearman correlation matrix (ρ) among quality, readability and length. GQS, Global Quality Score; DISCERN, modified DISCERN total score; FRES, Flesch Reading Ease Score; FKGL, Flesch–Kincaid Grade Level; SMOG, Simple Measure of Gobbledygook; GFI, Gunning Fog Index; CLI, Coleman–Liau Index; TTR, type–token ratio (lexical diversity). Values represent Spearman’s rank correlation coefficients (ρ). Bold values indicate correlations that remained statistically significant after Bonferroni adjustment (two-tailed; adjusted p < 0.05). Positive correlations indicate that both variables increase together, whereas negative correlations indicate that one variable increases as the other decreases. Discussion GDM management remains insufficiently standardized globally, particularly in low- and middle-income countries, where limited screening infrastructure and disparities in health literacy contribute to underdiagnosis and suboptimal glycemic control30,31. Limited antenatal consultation time and shortages of specialists further restrict access to guideline-based care. LLMs have recently emerged as innovative digital tools with the potential to bridge these gaps by providing readily accessible, patient-oriented medical information32,33. However, no comprehensive study has directly compared different LLM platforms in addressing patient-focused GDM queries in terms of content quality, scientific reliability, and readability. This study systematically evaluates the responses of ChatGPT, Grok, Gemini, and DeepSeek to 25 patient-derived GDM scenarios, using the mDISCERN instrument, Global Quality Scale, and multiple readability indices to assess guideline adherence, educational potential, and the balance between linguistic simplicity and technical depth. In our study, Grok and Gemini achieved higher mDISCERN and GQS scores, indicating better content reliability and overall quality in responding to patient-focused GDM queries, whereas ChatGPT demonstrated comparatively lower performance. This difference may reflect a closer alignment of Grok and Gemini’s responses with established guidelines and the informational needs of GDM patients. Based on the predefined mDISCERN classification (8–15 = low, 16–31 = moderate, 32–40 = high), the mean scores for all four models fell within the moderate reliability range. Although these inter-model differences were statistically significant, they should not be overinterpreted as clinically meaningful, since all models remained within the same reliability category. Moreover, no established minimal clinically important difference exists for mDISCERN or GQS in the current literature, so these variations are best interpreted as relative improvements in information quality rather than absolute clinical differences. This suggests that, although each model was able to produce responses with a reasonable degree of scientific reliability, none met the threshold for high reliability. Similar findings have been reported in studies assessing general-purpose AI chatbots across different medical domains, where overall performance was frequently in the moderate reliability (fair) range. For instance, Dursun et al. evaluated ChatGPT-4, Gemini, and other models in orthodontic patient information and found that all models remained within the moderate mDISCERN category34. In another study, AI models were evaluated for their responses to urinary incontinence–related questions, and, similarly, the reliability level of the generated health information was consistently determined to be moderate35. These results suggest that while LLMs demonstrate potential for supporting patient education, their current responses remain insufficient to fully replace guideline-concordant professional advice. Previous literature suggests that LLMs adapted specifically for the medical field, especially those trained on medically focused data, may offer performance advantages over general-purpose counterparts36. In our readability analyses, DeepSeek produced the most accessible responses according to FRES, whereas Grok generated longer and more complex outputs, potentially more useful for medical experts. However, none of the models met the recommended FRES > 60 threshold for patient education materials37. The cutoff proposed by the American Medical Association (AMA) and the National Institutes of Health (NIH) serves as a general benchmark for the lay population, yet it should be interpreted with caution for pregnant women with gestational diabetes, who represent a highly heterogeneous group in terms of educational background, cognitive load, and health literacy. Thus, a single readability threshold may not equally reflect comprehension levels across all subgroups. The relatively low FRES scores observed in this study may limit the comprehensibility of AI-generated responses, particularly for individuals with low health literacy. Previous research has shown that health literacy among women with gestational diabetes is often inadequate, negatively affecting their self-management abilities and glycemic control38,39. In this context, the inability of LLMs to consistently generate content at the recommended readability level may restrict their potential for unsupervised patient use, especially in vulnerable populations. Another important aspect of this study concerns the linguistic and cultural boundaries of LLM-based health communication. Limiting the analysis to English responses provided methodological consistency and enabled the use of validated readability indices calibrated for English texts (FRES, FKGL, SMOG). However, language not only facilitates communication but also shapes how people perceive health-related concepts. Previous studies have shown that linguistic framing and cultural tone significantly influence how patients perceive health information and medical authority39–41. Patients’ ways of expressing questions, interpreting recommendations, and establishing trust can vary depending on differences in language, culture, and healthcare systems. Therefore, the readability and reliability levels reported here may not fully represent how non-English-speaking populations—such as Turkish-speaking or multilingual communities—perceive AI-generated medical content. In addition, the questions used in this study were developed by Turkish clinicians to reflect patient expressions observed in local clinical settings. While this ensured linguistic realism, it may also have introduced certain cultural nuances—such as indirect expressions, politeness strategies, or contextual references—that differ from Anglo-American communication norms. Similarly, multilingual comparative studies have demonstrated that ChatGPT’s health-related responses vary in empathy, tone, and accuracy depending on the query language42,43. Thus, models trained predominantly on English datasets may produce linguistically fluent but culturally incongruent or contextually incomplete answers. These findings suggest that large language models should be evaluated not only in terms of formal readability but also regarding semantic clarity and cultural adaptability. Future cross-cultural validation studies should examine how translation, idiomatic differences, and health literacy variability influence AI–patient communication. Such research will be crucial for adapting large language models to multilingual healthcare environments and ensuring that AI-supported medical information is accessible and reliable for all populations. A positive correlation was observed between response length and quality scores, indicating that longer responses were often perceived as more comprehensive and detailed, which may have contributed to higher perceived reliability, rather than reflecting a causal effect. This finding is consistent with the literature, which highlights ‘detail and clarity’ as important factors in the perception of quality in LLM-generated responses44. However, lexical diversity (TTR) showed a paradoxical effect in our study. Our findings revealed a negative relationship between word variety (TTR) and quality scores. This suggests that increasing lexical diversity (e.g., contextual or technical words) in texts does not always positively impact perceived quality. While linguistic variety can enhance fluency, excessive terminological variation or ‘’unnecessary technicality’’ may undermine clarity and guideline alignment, negatively affecting evaluators’ perceptions of overall quality. In contrast, Long’s study found that lexical diversity generally had little to no effect on readability, while lexical density and lexical sophistication were the primary drivers. This divergence suggests that, in patient-oriented medical content, lexical diversity may play a more prominent role in readability than observed in academic medical texts45. These findings underscore that in patient-directed contexts, lexical simplicity and consistency may be more important than linguistic richness. From a clinical perspective, our findings highlight both the promise and the limitations of LLMs in GDM education. Gemini and Grok produced responses of comparatively higher quality, indicating their potential as supplementary educational tools. However, the moderate overall reliability and suboptimal readability reinforce the necessity of physician oversight and guideline-based verification before LLM-generated content can be shared with patients. To maximize impact, LLM outputs should be tailored to the health literacy level of the intended audience, using clear and patient-centered language. Evidence suggests that simplified educational materials in diabetes care improve treatment adherence and quality of life46, whereas failing to consider cultural context and digital literacy reduces patient engagement and intervention effectiveness41. When preparing informational responses for patients with GDM, focusing on texts that are both easy to read and specific, reducing unnecessary technical terminology, yet providing comprehensive information may be more effective in optimizing evaluator scores. These findings suggest that LLM-based chatbots have the potential to serve as complementary advisory tools under physician supervision and for continuous use in gestational diabetes care. These technologies offer an important opportunity to bridge information gaps in regions with limited antenatal care access, as they can offer rapid, round-the-clock accessible, and linguistically adaptable responses in e-health environments. However, since this study was restricted to English-language responses, the generalizability of the findings to non-English-speaking populations remains limited. Future research should include multilingual and culturally diverse datasets to confirm the applicability of these results across different healthcare contexts. Content tailored to individuals with GDM, prepared in a simplified and patient-friendly language appropriate for the target audience, can enhance comprehensibility and educational impact, particularly among groups with low health literacy. The literature reports that the use of simplified educational materials in individuals with diabetes significantly improves treatment adherence and quality of life46. Furthermore, it has been reported that when cultural context and digital health literacy levels are not taken into account in digital educational materials, patient participation and intervention effectiveness decrease significantly41. The strengths of our study include the use of patient-perspective-based questions, multidimensional evaluation of both quality and readability, and a comparative analysis of leading contemporary large language models. On the other hand, the limitations of our study include its restriction to English-language content, the lack of evaluation across culturally diverse populations, and the fact that only the first response from each model was recorded for analysis Although this approach enhanced standardization and comparability across LLMs, it did not account for the potential intra-model variability that could emerge from repeated querying. This English-only design was chosen to maintain methodological consistency and to utilize standardized readability metrics that are validated for English texts. However, we acknowledge that linguistic and cultural diversity strongly influences how patients interpret, trust, and apply medical information. Therefore, the generalizability of our findings to non-English-speaking or culturally distinct populations should be interpreted with caution. In addition, although the questions were written from a patient-centered perspective, they were developed by clinicians based on authentic patient expressions observed during consultations rather than by patients themselves. Therefore, the patient perspective was represented indirectly. This was achieved through the use of plain language derived from authentic patient statements and by objectively assessing comprehension levels through readability analyses. Future studies should include direct patient participation and comprehension testing to determine how understandable and educational AI-generated responses are in real-life contexts. Furthermore, research involving patient groups from different cultural backgrounds and using visual materials or multi-step clinical scenarios may more accurately reflect real patient needs. Repeated querying across sessions should also be considered to evaluate the stability and consistency of LLM responses. Finally, another limitation of the study is that readability was not validated through patient-based assessment. This issue should be addressed in future research to ensure that AI-generated materials are adapted to different literacy levels and cultural contexts. Conclusion LLMs hold promise as complementary tools to bridge informational gaps in GDM management, particularly in settings where access to specialized care is limited. However, the moderate overall reliability and suboptimal readability highlight the need for physician oversight and guideline-based verification before LLM-generated content can be shared with patients. While current LLMs provide moderately reliable content with varying readability, their outputs remain insufficient for unsupervised patient use. These systems should be further developed and aligned with clinical guidelines, and integration into physician-supervised care pathways that support patient education. Supplementary Information Below is the link to the electronic supplementary material. Supplementary Material 13 Supplementary Material 13 Supplementary Material 2 Supplementary Material 2 Supplementary Material 3 Supplementary Material 3