Evaluating the performance of ChatGPT in responding to myotonic dystrophy type 1 patient inquiries: a specialist physician-based study Authors: Gülşah Çelik, Hanife Hale Hekim, Meral Bilgilisoy Filiz, Naciye Füsun Toraman Published: 2026 ABSTRACT AbstractBackgroundAdvancements in artificial intelligence have led to the widespread use of large language models such as ChatGPT in healthcare communication. Myotonic dystrophy type 1, a chronic and multisystemic neuromuscular disorder, poses significant challenges for patients in understanding disease progression, symptom management, and future planning. Given the scarcity of specialized resources for rare diseases, AI-based tools may offer valuable support by delivering accessible and relevant information. This study aimed to evaluate ChatGPT-4o’s performance in addressing patient-centered questions about myotonic dystrophy type 1.MethodsThis descriptive methodological study utilized ten frequently asked patient-style questions concerning myotonic dystrophy type 1, posed to ChatGPT-4o in Turkish using a standardized zero-shot prompting approach. A total of 53 specialist physicians in Physical Medicine and Rehabilitation and Neurology assessed the responses using a structured rubric across five domains: accuracy, currency, comprehensiveness, usefulness, and understandability. Each response was scored from 1 to 5.ResultsChatGPT-4o achieved high specialist physicians’ satisfaction across all domains, with over 80% of responses rated ≥ 4. Accuracy received the highest score (85.7%), followed by usefulness and comprehensiveness (above 82%). Currency and understandability were rated slightly lower (81.9% and 81.5%, respectively).ConclusionChatGPT-4o generated responses that were generally accurate, relevant, and understandable, demonstrating promise as an accessible resource for rare disease contexts such as myotonic dystrophy type 1. It also showed strong potential as a supportive informational tool for specialist physicians. Ongoing model refinement and integration with up-to-date clinical data are essential to optimize its performance and ensure its safe, equitable use in healthcare communication.Keywords: ChatGPT, Large language models, Artificial intelligence, Myotonic dystrophy, Expert opinion Introduction Recent advancements in artificial intelligence (AI) and natural language processing (NLP) technologies have facilitated the development of sophisticated language models, or chatbots, capable of generating human-like text [1, 2]. These models, trained on vast datasets, can engage in dynamic, context-aware conversations, offering users rapid access to information, support, and even decision-making assistance in various domains, including healthcare [3]. The ability of machines to process and interpret human language has become a cornerstone of modern AI, largely driven by developments in NLP [4]. This progress has paved the way for the creation of highly capable language-based AI systems, known as large language models (LLMs) [5]. Among these, ChatGPT-4o (OpenAI Incorporated, Mission District, San Francisco, USA) has emerged as a prominent example, designed to simulate coherent and contextually appropriate human dialogue [1, 2]. ChatGPT-4o’s advanced language capabilities have introduced new opportunities in healthcare, particularly in delivering accessible medical information and supporting clinical decision-making [6]. In addition to enhancing general health literacy, such tools can help reduce disparities in healthcare access, especially for individuals in rural or underserved areas who may face barriers such as limited specialist availability, long travel distances, or insufficient health infrastructure [7]. ChatGPT-4o can assist patients in understanding their conditions, managing symptoms, and making informed decisions by providing timely, personalized, and comprehensible information [8]. Its capacity to synthesize complex clinical data and deliver patient-centered explanations highlights its potential to promote early intervention, reduce unnecessary clinical visits, and support cost-effective care [9]. Evaluating ChatGPT-4o’s effectiveness in disseminating health knowledge is thus a key step toward understanding its broader role in strengthening public health systems through digital innovation and inclusive information delivery. Given the complex care needs associated with chronic neuromuscular conditions, myotonic dystrophy type 1 (DM1) stands out as a particularly challenging disorder due to its progressive, multisystemic nature. It affects both skeletal muscles and various organ systems, leading to a broad spectrum of symptoms, including muscle weakness, myotonia, fatigue, pain, and dysphagia. In addition to these physical symptoms, cognitive involvement is frequently observed, particularly affecting attention, executive functions, and information processing, which may influence patients’ ability to understand and manage health-related information [10, 11]. These symptoms significantly impair patients’ ability to carry out daily activities and maintain functional independence [12]. Beyond the physical burden, individuals with DM1 often struggle with psychological issues such as anxiety, depression, frustration, and low self-esteem, which further diminish their overall quality of life [13]. In clinical settings, many patients with DM1 report that the medical information they receive is often insufficient for them to fully understand their condition, manage symptoms effectively, or make informed decisions regarding treatment [14]. Patients express a strong desire for clearer, more individualized guidance and a more active role in the therapeutic process. These observations underscore the importance of improving patient health literacy and empowering individuals with DM1 to feel more in control of their disease management. Importantly, patients often seek information not only about symptom control or disease progression but also about how the disorder may affect their long-term functional status, daily living, and future planning [15, 16]. In this context, conversational AI tools such as ChatGPT-4o may represent a promising solution by delivering accessible, personalized, and context-aware medical information. By addressing patients’ practical concerns and knowledge gaps, such platforms have the potential to enhance self-efficacy, support informed decision-making, and promote more active engagement in the management of chronic conditions like DM1. To contextualize the potential of AI tools like ChatGPT in addressing the informational needs of patients with DM1, it is useful to consider findings from other clinical areas. Recent studies have examined the potential of ChatGPT in various medical contexts, including oncology, neurology, and musculoskeletal disorders. Gummadi et al. demonstrated that ChatGPT provided accurate and comprehensive responses to patient questions about metastatic breast cancer, suggesting its potential to enhance patient education and support [17]. Similarly, Zhenzhu et al. found that ChatGPT-based tools, when guided by clinical protocols, improved both the clarity and empathy of responses in traumatic brain injury rehabilitation [18]. Giray et al. evaluated ChatGPT’s responses on Scheuermann’s kyphosis and concluded that while general knowledge was accurate, treatment-specific recommendations were limited [19]. Likewise, Johnson et al. reported that ChatGPT gave satisfactory answers to common questions about shoulder arthroplasty, though many required further clarification [20]. These studies collectively indicate that while ChatGPT may not substitute clinical judgment, it can function as a supplementary tool to improve patient awareness and engagement, especially in complex chronic diseases that require continuous information support. Despite these applications, ChatGPT’s utility in the context of DM1 remains unexplored. DM1 is a progressive, multisystemic neuromuscular disorder that requires long-term follow-up in specialized centers due to its complex and evolving clinical course. From the time of diagnosis, patients face diverse challenges, ranging from organ involvement to progressive loss of functional capacity, that lead to a continuous need for reliable and accessible information. Given their frequent questions about disease progression, symptom management, and future planning, individuals with DM1 may benefit from AI tools that can supplement healthcare communication. Therefore, this study aims to evaluate ChatGPT-4o’s potential as a supportive resource for addressing the informational needs of DM1 patients. Materials and methods Ethical approval Ethical approval was obtained from the University of Health Sciences Antalya Training and Research Hospital Clinical Research Ethics Committee (Approval Date: January 30, 2025; Approval Number: 2/15). All participating specialist physicians were informed about the study's objectives, and informed consent was obtained prior to their participation. The study was conducted in accordance with the principles of the 2013 Declaration of Helsinki. Study design and participants This descriptive methodological study aimed to evaluate ChatGPT-4o’s performance in addressing patient-centered inquiries related to DM1. Physicians with expertise in neuromuscular disorders from the fields of Physical Medicine and Rehabilitation (PMR) and Neurology conducted the assessments. The patient questions reflected real-world concerns about DM1, encompassing topics such as functional limitations, treatment approaches, disease trajectory, and individual life planning, including daily challenges, rehabilitation strategies, and genetic implications. A total of 53 specialist physicians from Turkey, all within the inclusion range of 35–70 years, each with at least four years of clinical experience, voluntarily participated in the study. This age range was chosen to ensure that participants had sufficient clinical experience to provide expert evaluations. The group included attending specialists, associate professors, professors, and senior residents (research assistants) in the final year of their specialty training, who were considered specialist candidates actively managing neuromuscular patients under supervision. Participants were recruited through non-probability snowball sampling. Data collection was carried out using online forms (Google Forms: Google LLC, Mountain View, CA, USA), which were distributed via email. Questions directed to ChatGPT-4o The ten most frequently asked questions by patients with DM1 were developed by a core group of six senior specialist physicians (three from PMR and three from Neurology) working at the Neuromuscular Diseases Center of Antalya Training and Research Hospital, who routinely manage patients diagnosed with DM1 (Table 1.). These questions were derived from the most frequent concerns expressed by their patients during follow-up visits, covering topics such as disease progression, daily functioning, rehabilitation, and genetic implications. The questions were simulated as if they were asked by patients or their relatives and were submitted to ChatGPT-4o in a manner typical of patient speech. Table 1Questions submitted to ChatGPT-4oQ1My doctor diagnosed me with Myotonic Dystrophy. What kind of disease is it and what are its symptoms?Q2My doctor diagnosed me with Myotonic Dystrophy. How will this disease affect me?Q3Is there a treatment for Myotonic Dystrophy? What treatment methods are available?Q4I am a Myotonic Dystrophy patient. Is physical therapy and rehabilitation beneficial for my muscle weakness?Q5My doctor diagnosed me with Myotonic Dystrophy. Which exercises would be most beneficial for me?Q6Does Myotonic Dystrophy only cause muscle weakness, or does it also affect other organs?Q7My doctor diagnosed me with Myotonic Dystrophy. How will this disease affect my daily life?Q8I am a Myotonic Dystrophy patient. What can I do to reduce the impact of this disease on my daily activities?Q9I am a Myotonic Dystrophy patient. Will this disease prevent me from having children?Q10I am a Myotonic Dystrophy patient. Should I have genetic testing done for my child? Questions submitted to ChatGPT-4o All questions were asked in Turkish, which reflects the language spoken by the majority of the patient population in the clinical context of this study. ChatGPT-4o was accessed through its official platform (https://chat.openai.com) in January 2025. All questions were submitted to the model using a standardized zero-shot prompt. In this approach, the model was not provided with any examples or demonstrations prior to generating a response. Using a standardized zero-shot prompting method is essential for ensuring consistency and comparability across responses. By eliminating prior examples, the model’s performance reflects its generalized reasoning and knowledge retrieval capabilities, rather than task-specific fine-tuning or pattern memorization [21, 22]. Furthermore, to prevent the models from adapting their response strategies based on previous interactions within the same session (i.e., in-context adaptation), a new session was initiated for each individual question by clearing the chat history prior to submission. This approach ensured that each question was evaluated independently, eliminating any potential carryover effects from earlier prompts or responses. Responses were recorded without any word limit, and each question was asked only once to avoid contextual bias and ensure independent generation of responses. However, to ensure accessibility for an international readership, we have provided the English-translated versions of all ten AI-generated responses in Supplementary Table S1. Evaluation of ChatGPT-4o responses The responses generated by ChatGPT-4o were evaluated by specialist physicians using a structured rubric developed through a comprehensive review of the literature on LLM assessments in the medical domain. Five core evaluation dimensions; accuracy, currency, comprehensiveness, usefulness, and understandability, were adapted from previously published frameworks to assess the quality of the information provided [23–25] (Table 2). Each response was independently rated by specialist physicians using a 5-point Likert-type rating system. This rating system was used as part of a rubric-based expert evaluation method. Each evaluation domain was scored separately on an ordinal scale from 1 to 5 for each clinical scenario [23, 24]. The ratings were used descriptively, and no total or composite score was calculated. Table 2The rubric evaluation for specialist physiciansCriteria1 Point2 Points3 Points4 Points5 PointsAccuracyThe information is completely incorrect or missing.The information is mostly incorrect or missing.The information is partially correct but incomplete.The information is mostly correct but has some omissions.The information is completely correct.CurrencyThe response is not up-to-date.The response is very slightly up-to-date.The response is somewhat up-to-date.The response is mostly up-to-date.The response is completely up-to-date.ComprehensivenessThe response covers only one aspect.The response covers very few aspects.The response covers some important aspects.The response covers most important aspects.The response covers all important aspects.UsefulnessThe response is not useful at all.The response provides very little benefit.The response is somewhat beneficial.The response is generally useful.The response is completely useful and applicable.UnderstandabilityThe response is not understandable.The response is difficult to understand.The response is partially understandable.The response is mostly understandable.The response is completely understandable. The rubric evaluation for specialist physicians Ratings of 4 and 5 indicated that the evaluator considered the response satisfactory or highly satisfactory for a given criterion. For descriptive analysis, these two upper categories were combined, consistent with standard practice in expert-based Likert evaluations to enhance interpretability [23, 24, 26]. The “Overall Evaluation Percentage” was calculated as the average proportion of evaluators assigning scores of 4 or higher across all questions and criteria. Additionally, domain-specific index scores were computed by aggregating and averaging ratings within each evaluation category, allowing for a unified summary metric per domain [23]. Although rubric-based scoring is commonly used in educational assessment, its application has expanded to health and clinical research for structured expert evaluations [27–29]. This study was designed as a descriptive methodological evaluation based on expert judgment, consistent with prior validation studies assessing AI-generated medical content. Although a minimum of five raters is generally recommended for content validity and rubric-based assessments, larger expert panels including up to 40 specialists are often used in complex medical studies to strengthen reliability and objectivity [23, 24, 30–33]. In expert-based assessments of this nature, the primary objective is to ensure content validity through the inclusion of a sufficiently diverse and experienced panel rather than achieving a predetermined sample size. Accordingly, a total of 53 specialist physicians from PMR and Neurology were included in the study to provide evaluations of ChatGPT-4o’s responses. This number notably exceeds the commonly cited minimum expert raters recommended for content validity and rubric-based assessments. Moreover, recent studies evaluating ChatGPT’s medical information quality have typically involved 20–60 expert participants, indicating that the sample size used in the present study was appropriate to ensure reliability, objectivity, and generalizability of findings within the expert-based assessment framework [23, 24]. Statistical analysis Statistical analyses were performed with IBM SPSS 26.0 (SPSS Inc., Chicago, USA). The distribution of the continuous variable (age) was assessed through both visual methods (such as histograms and probability plots) and statistical test (Shapiro–Wilk test). The age of participants was analyzed as a continuous variable, while Likert-type rating system were treated as ordinal data. Descriptive statistics were presented as number, percentage, mean±standard deviation and median (minimum; maximum). For each evaluation domain (accuracy, currency, comprehensiveness, usefulness, and understandability), domain-specific index scores were calculated by averaging all individual Likert ratings given by physicians across the ten questions. For example, if a physician rated the ‘accuracy’ of 10 ChatGPT responses as x₁, x₂, …, x₁₀ on a 5-point Likert-type rating system, the Accuracy Index Score for that physician was computed as: This calculation was repeated for all 53 physicians, and the median (min–max) and mean ± SD of these individual index scores was reported in Table 3. The same procedure was applied for all other domains. Given the ordinal nature of the data, median (min–max) values were presented, and mean ± SD values were also reported to facilitate interpretation and comparison across evaluation domains. Table 3Evaluation of ChatGPT’s answers to the questions by the experts according to the scoring rubric in terms of accuracy, currency, comprehensiveness, usefulness, and understandabilityExperts, N = 53Experts with Likert Response of ≥ 4,n (%)Mean ± SDScoreMedian (Min-max) ScoreQuestion 1 Accuracy42 (79.2)4.1 ± 0.84.0 (2.0–5.0) Currency39 (73.6)3.9 ± 0.74.0 (2.0–5.0) Comprehensiveness34 (64.2)3.8 ± 0.84.0 (2.0–5.0) Usefulness39 (73.6)4.0 ± 0.84.0 (2.0–5.0) Understandability38 (71.7)4.0 ± 1.04.0 (1.0–5.0)Question 2 Accuracy47 (88.7)4.3 ± 0.74.0 (3.0–5.0) Currency42 (79.2)4.0 ± 0.74.0 (3.0–5.0) Comprehensiveness38 (71.7)3.9 ± 0.84.0 (2.0–5.0) Usefulness43 (81.1)4.1 ± 0.74.0 (2.0–5.0) Understandability42 (79.2)4.2 ± 0.94.0 (2.0–5.0)Question 3 Accuracy41 (77.4)4.1 ± 0.84.0 (2.0–5.0) Currency40 (75.4)3.9 ± 0.84.0 (2.0–5.0) Comprehensiveness41 (77.4)4.0 ± 0.84.0 (2.0–5.0) Usefulness42 (79.2)4.1 ± 0.84.0 (2.0–5.0) Understandability41 (77.4)4.1 ± 1.04.0 (2.0–5.0)Question 4 Accuracy45 (84.9)4.2 ± 0.74.0 (3.0–5.0) Currency47 (88.7)4.3 ± 0.74.0 (3.0–5.0) Comprehensiveness46 (86.8)4.2 ± 0.74.0 (2.0–5.0) Usefulness47 (88.7)4.3 ± 0.74.0 (2.0–5.0) Understandability45 (84.9)4.3 ± 0.84.0 (2.0–5.0)Question 5 Accuracy48 (90.6)4.3 ± 0.74.0 (2.0–5.0) Currency46 (86.8)4.1 ± 0.74.0 (2.0–5.0) Comprehensiveness47 (88.7)4.2 ± 0.74.0 (2.0–5.0) Usefulness46 (86.8)4.2 ± 0.94.0 (1.0–5.0) Understandability45 (84.9)4.3 ± 0.84.0 (2.0–5.0)Question 6 Accuracy47 (88.7)4.3 ± 0.74.0 (3.0–5.0) Currency44 (83.0)4.2 ± 0.74.0 (3.0–5.0) Comprehensiveness47 (88.7)4.3 ± 0.74.0 (3.0–5.0) Usefulness44 (83.0)4.2 ± 0.74.0 (3.0–5.0) Understandability45 (84.9)4.3 ± 0.85.0 (2.0–5.0)Question 7 Accuracy47 (88.7)4.3 ± 0.74.0 (3.0–5.0) Currency47 (88.7)4.2 ± 0.64.0 (3.0–5.0) Comprehensiveness50 (94.3)4.3 ± 0.64.0 (3.0–5.0) Usefulness48 (90.6)4.3 ± 0.64.0 (3.0–5.0) Understandability49 (92.5)4.4 ± 0.74.0 (2.0–5.0)Question 8 Accuracy46 (86.8)4.2 ± 0.74.0 (3.0–5.0) Currency44 (83.0)4.0 ± 0.64.0 (3.0–5.0) Comprehensiveness43 (81.1)4.1 ± 0.74.0 (3.0–5.0) Usefulness46 (86.8)4.2 ± 0.74.0 (3.0–5.0) Understandability47 (88.7)4.3 ± 0.74.0 (3.0–5.0)Question 9 Accuracy46 (86.8)4.2 ± 0.74.0 (2.0–5.0) Currency42 (79.2)4.0 ± 0.74.0 (2.0–5.0) Comprehensiveness46 (86.8)4.1 ± 0.74.0 (2.0–5.0) Usefulness42 (79.2)4.0 ± 0.84.0 (2.0–5.0) Understandability38 (71.7)4.0 ± 0.94.0 (2.0–5.0)Question 10 Accuracy45 (84.9)4.3 ± 0.84.0 (2.0–5.0) Currency43 (81.1)4.1 ± 0.84.0 (2.0–5.0) Comprehensiveness45 (84.9)4.2 ± 0.84.0 (2.0–5.0) Usefulness44 (83.0)4.2 ± 0.84.0 (2.0–5.0) Understandability42 (79.2)4.2 ± 0.84.0 (2.0–5.0)All QuestionsOverall Index Percentage of Responses Scoring ≥ 4, %*Overall Index Score**Overall Index Score** Accuracy85.74.2 ± 0.64.2 (2.9-5.0) Currency81.94.1 ± 0.64.0 (2.7-5.0) Comprehensiveness82.54.1 ± 0.54.0 (2.7-5.0) Usefulness83.24.2 ± 0.64.1 (2.6-5.0) Understandability81.54.2 ± 0.74.2 (2.0–5.0)*: The percentage of responses scoring ≥ 4 across all ten questions are provided for each of the five evaluation areas**: The overall index scores for all questions are derived by aggregating and averaging the scores from all five areas across each question Evaluation of ChatGPT’s answers to the questions by the experts according to the scoring rubric in terms of accuracy, currency, comprehensiveness, usefulness, and understandability *: The percentage of responses scoring ≥ 4 across all ten questions are provided for each of the five evaluation areas **: The overall index scores for all questions are derived by aggregating and averaging the scores from all five areas across each question The Wilcoxon Signed Ranks Test was used for pairwise comparisons of the overall index scores of evaluation criteria. A p-value of < 0.05 was considered statistically significant. Results Characteristics of specialist physicians A total of 53 specialists in PMR and Neurology participated in the study. The mean age of the participants was 42.5 years (SD = 8.8), with an age range from 36 to 67 years and a median age of 39. The majority were female (n = 42, 79.2%), while 11 participants (20.8%) were male. Regarding academic titles, 20 participants (37.7%) held associate professor or professor positions, 17 (32.1%) were specialist physicians, and 16 (30.2%) were residents (research assistants). Most of the evaluators (n = 42, 79.2%) specialized in PMR, and the remaining 11 (20.8%) were neurologists. ChatGPT-4o was reported as the most commonly used LLM, with 41 participants (77.4%) indicating prior use. Evaluation of ChatGPT-4o responses For each question, specialist physicians rated ChatGPT-4o’s responses as 4 or higher (≥ 4) in all five evaluation categories in more than 70% of cases, with one exception: for Question 1, the proportion of specialist physicians rating ‘’ comprehensiveness ‘’ as ≥ 4 was slightly lower, at 64.2% (Table 3). Across all evaluations, scores of 1 or 2, representing the lowest satisfaction levels, were rare. Only two responses in total received a score of 1, both observed in the Understandability domain of Q1 (n = 1, 1.9%) and the Usefulness domain of Q5 (n = 1, 1.9%). Limited “2” ratings were observed, the highest proportion being 11.3% (n = 6 responses) in the Understandability domain of Q3, followed by 7.5% (n = 4 responses) in the Understandability domain of Q2. In general, ratings of 4 and 5 predominated across all domains (Supp Table 2). Considering the overall index scores calculated by aggregating and averaging the ratings across all questions, accuracy also had the highest proportion of responses rated ≥ 4 (85.7%), while the remaining four domains ranged between 81.5% and 83.2% (Table 3; Fig. 1). Figure 1 visually illustrates the proportion of responses scoring ≥ 4 across all five evaluation domains, highlighting that accuracy was consistently rated highest, whereas currency and comprehensiveness showed slightly lower satisfaction levels. The median overall index scores were relatively close across evaluation domains, with accuracy and understandability both scoring 4.2 (ranges: 2.9–5.0 and 2.0–5.0, respectively), followed by usefulness at 4.1 (2.6–5.0), and currency and comprehensiveness each at 4.0 (2.7–5.0) (Table 4). Pairwise comparisons revealed statistically significant differences between several criteria. Specialist physicians rated accuracy significantly higher than both currency and comprehensiveness (p < 0.001 for both). Currency was rated significantly lower than usefulness and understandability (both p = 0.001). In addition, comprehensiveness received lower scores compared with usefulness (p = 0.024) and understandability (p = 0.017), (Table 4; Fig. 2). “Distribution of overall index percentage of all questions’ responses scoring ≥ 4 across the five evaluation areas (based on experts’ assessment) Table 4Pairwise comparisons of the overall index scores of evaluation criteria by expertsAll Questions Overall Index Score*p values**Median (Min-max) ScoreEvaluation by ExpertsAccuracy4.2 (2.9-5.0)Accuracy vs. Currency: p < 0.001Accuracy vs. Comprehensiveness: p < 0.001Accuracy vs. Usefulness: p = 0.066Accuracy vs. Understandability: p = 0.679Currency vs. Comprehensiveness: p = 0.251Currency vs. Usefulness: p = 0.001Currency vs. Understandability: p = 0.001Comprehensiveness vs. Usefulness: p = 0.024Comprehensiveness vs. Understandability: p = 0.017Usefulness vs. Understandability: p = 0.289Currency4.0 (2.7-5.0)Comprehensiveness4.0 (2.7-5.0)Usefulness4.1 (2.6-5.0)Understandability4.2 (2.0–5.0)*: The overall index scores for all scenarios is derived by aggregating and averaging the scores from all five areas across each scenario**: Wilcoxon Signed Ranks Test Pairwise comparisons of the overall index scores of evaluation criteria by experts Accuracy vs. Currency: p < 0.001 Accuracy vs. Comprehensiveness: p < 0.001 Accuracy vs. Usefulness: p = 0.066 Accuracy vs. Understandability: p = 0.679 Currency vs. Comprehensiveness: p = 0.251 Currency vs. Usefulness: p = 0.001 Currency vs. Understandability: p = 0.001 Comprehensiveness vs. Usefulness: p = 0.024 Comprehensiveness vs. Understandability: p = 0.017 Usefulness vs. Understandability: p = 0.289 *: The overall index scores for all scenarios is derived by aggregating and averaging the scores from all five areas across each scenario **: Wilcoxon Signed Ranks Test Distribution of overall index scores across five evaluation domains Discussion This study aimed to explore the potential of ChatGPT-4o in supporting individuals affected by DM1 through the provision of reliable, comprehensible, and accessible health information. By focusing on AI-generated responses to patient-centered queries, the study contributes to the emerging evidence base regarding the role of LLMs in rare disease communication. Given the complexity and chronic progression of DM1, providing patients with clear and trustworthy information is essential to promote informed decision-making and self-management [13]. These findings offer a valuable basis for understanding how AI-powered tools may enhance clinical communication in rare diseases such as DM1, especially where access to disease-specific information is limited. In this study, we evaluated ChatGPT-4o’s responses to questions posed from a patient perspective, reflecting common concerns and expressions typically voiced by individuals living with DM1. Specialist physician reviewers rated each response across five categories: accuracy, currency, comprehensiveness, usefulness, and understandability. Although there were some statistically significant differences between domains, such as accuracy being rated higher than currency and comprehensiveness, the overall scores were consistently high. Additionally, the percentage of responses rated 4 or higher (≥ 4) was also high: accuracy had the highest rate at 85.7%, followed by usefulness and comprehensiveness (both above 82%), and slightly lower percentages for currency and understandability (81.9% and 81.5%, respectively). This consistency suggests that, overall, the model performed satisfactorily and met specialist physicians’ expectations across domains, regardless of content complexity or emotional sensitivity embedded in the patient-like queries. One possible explanation for the relatively higher scores in the accuracy domain may be related to the well-established medical knowledge base concerning DM1, particularly in areas such as disease progression, multisystem involvement, impact on daily life, and available treatment approaches. Since ChatGPT-4o is trained on extensive and validated medical literature, it is more likely to produce accurate and reliable responses when addressing well-documented aspects of the disease [34, 35]. Many of the questions posed in this study reflected these commonly discussed and clinically recognized topics, allowing the model to retrieve and generate information that aligns closely with current medical consensus. In addition to accuracy, specialist physician reviewers rated the domains of usefulness, comprehensiveness, currency, and understandability at comparably high levels, with minor variations across categories. The elevated scores for usefulness and comprehensiveness indicate that ChatGPT-4o was generally effective in delivering information that aligned with the real-life needs of patients while also reflecting the complex and multisystemic nature of DM1. Usefulness was likely enhanced by the model’s capacity to provide applicable and relatable content, particularly in areas such as daily living and symptom management [36]. This finding is consistent with previous literature, such as the study by Uz et al., which reported high usefulness scores for ChatGPT in conveying information related to rheumatic diseases, emphasizing its value in patient education across chronic conditions [37]. Similarly, Cankurtaran et al. highlighted ChatGPT’s utility in inflammatory bowel disease, suggesting that, despite certain content limitations, it holds promise as an informative tool when appropriately maintained and updated by developers [38]. Comprehensiveness, in turn, was reflected in the model’s ability to address a wide range of disease-related domains, including muscular, respiratory, cardiac, and genetic aspects, providing a broad yet coherent overview that largely aligned with specialist physicians’ expectations. Notably, although comprehensiveness received statistically lower scores than usefulness and understandability, this difference was relatively modest and must be interpreted within the context of generally high ratings across all domains. In fact, the overall index score for comprehensiveness remained high at 82.5%, indicating that the majority of responses were still perceived as satisfactory or better. This suggests that, despite minor comparative differences, the model was able to provide sufficiently detailed and informative content for most evaluators. Similar findings were reported by Yapar et al., who observed slightly reduced scores for comprehensiveness in orthopedic rehabilitation settings, attributing this to the inherent difficulty of conveying nuanced clinical detail through AI models [23]. However, this limitation did not preclude the generation of broadly acceptable content. Conversely, Wang et al. demonstrated ChatGPT’s capacity to deliver well-rounded and satisfactory consultation responses in thyroid eye disease, underscoring its adaptability in multidisciplinary clinical scenarios [39]. Taken together, these results highlight that while there is room for improvement in enhancing clinical nuance and personalization, ChatGPT-4o already demonstrates a strong capacity to generate comprehensive, system-spanning information that supports patient understanding in complex conditions like DM1. Currency, or the ability to reflect up-to-date clinical knowledge, was also evaluated positively by specialist physicians. Although this domain received comparatively lower scores than some others, the ratings remained above the 80% threshold, suggesting that the majority of ChatGPT-4o’s responses were still perceived as timely and relevant. This relatively strong performance may be attributed to the model’s exposure to a broad corpus of medical literature up to its knowledge cut-off, allowing it to deliver answers aligned with widely accepted standards of care. Nevertheless, minor differences in scoring such as accuracy being rated significantly higher than currency, may reflect the inherent challenges of keeping pace with rapidly evolving medical guidelines, particularly in specialized or rare disease contexts like DM1. Similar concerns were raised by Meo et al., who noted that AI models may require ongoing updates to reliably incorporate the most recent scientific developments [40]. However, domain-specific variation may influence this performance. For instance, Toiv et al. reported high levels of perceived currency in gastroenterology-related responses, indicating that up-to-dateness may vary depending on how frequently new evidence emerges in a given field [41]. Overall, these results suggest that ChatGPT-4o maintains a generally satisfactory level of currency, successfully reflecting up-to-date medical knowledge across a wide range of topics. While minor gaps were observed in highly dynamic or specialized areas, the model’s ability to provide timely and relevant information was evident throughout most queries. These findings highlight its potential as a valuable resource for supporting patient education, particularly in situations where access to real-time clinical expertise may be constrained. Continuous refinement of its updating mechanisms may further strengthen its reliability in rapidly evolving medical contexts. Among the evaluated domains, understandability received relatively lower ratings, suggesting that clarity and ease of comprehension may still benefit from targeted refinement, particularly when evaluated by specialist physicians. While ChatGPT-4o generally aimed to present information in a patient-friendly format, certain responses included technical terminology or abstract phrasing that may have limited accessibility for users with lower health literacy. This is consistent with previous findings indicating that healthcare professionals tend to emphasize clinical precision and appropriate use of medical language, which can inadvertently result in lower scores when responses are not explicitly simplified for lay audiences [42]. Moreover, cognitive involvement is a recognized feature of DM1, often manifesting as difficulties in attention, executive functioning, and information processing speed. These neurocognitive features can further hinder patients’ ability to comprehend lengthy or complex explanations, even when the content is medically accurate. In the context of DM1, where the complexity of multisystem involvement poses additional challenges, achieving the right balance between medical accuracy and understandable communication remains critical. These findings underscore the importance of improving linguistic clarity in AI-generated health information, particularly in contexts involving direct patient communication, while also highlighting the model’s potential to convey medical content in an accessible and coherent manner. In light of these findings, it is essential to recognize both the promise and the limitations of AI-assisted tools in patient communication. As AI systems like ChatGPT become increasingly integrated into healthcare communication, ethical and safety issues warrant careful attention. Although these tools can significantly improve access to information, they should be employed as adjuncts to, rather than replacements for, clinical expertise [43]. Risks such as inaccurate guidance, implicit bias, or emotionally insufficient responses remain relevant concerns [44–47]. Moreover, the digital divide may restrict access for certain populations, potentially deepening health disparities [43]. To ensure safe, equitable, and effective implementation, ongoing validation against current clinical standards, regular content updates, and well-defined ethical frameworks are needed. This study provides timely insights into how structured evaluations of LLM performance can inform such policies and guide responsible AI integration in health communication. This study has several limitations that warrant consideration. One limitation is that the prompts provided to ChatGPT used the general term “myotonic dystrophy” rather than explicitly specifying DM1. Although all clinical evaluations and expert-based comparisons in this study were conducted exclusively on patients with clinically confirmed DM1, the questions were intentionally framed using patient-centered language to reflect real-world clinical communication. As patients often do not differentiate between disease subtypes in daily practice, the AI-generated responses may partially reflect information applicable to both DM1 and DM2, which should be considered when interpreting the findings. The patient-centered questions used for evaluation were carefully designed by the research team to reflect common concerns associated with DM1; however, this predefined and fixed set may not fully capture the diversity, spontaneity, or linguistic variability of real-life patient inquiries. In addition, as four of the ten questions focused on treatment, exercise, and rehabilitation, the topic distribution may have been influenced by the authors’ professional background in PMR introducing a potential bias toward therapeutic content rather than broader patient concerns such as fatigue, pain, or prognosis. Moreover, although a standardized zero-shot prompting approach was used to ensure consistency, each question was submitted only once, and the potential variability of ChatGPT-4o’s responses, an inherent feature of LLMs, was not systematically explored. Additionally, the understandability of ChatGPT-4o’s responses was evaluated solely by specialist physicians rather than patients. While the evaluators were instructed to consider how understandable the responses would be for a general patient audience, their professional background may still differ from the cognitive and linguistic capacities of individuals with DM1. It should also be acknowledged that individuals with DM1 may exhibit cognitive and behavioral characteristics such as apathy, reduced initiative, and limited health-seeking behavior, which could decrease their likelihood of actively engaging with digital or AI-based health resources. This behavioral aspect may further limit the real-world applicability of ChatGPT’s outputs for some patients within this population. Future studies incorporating direct patient feedback would provide a more comprehensive assessment of language accessibility and real-world comprehensibility. Furthermore, the evaluation relied exclusively on physician scoring without objective readability analyses, and since all questions were posed in Turkish, the generalizability of findings to other languages and cultural contexts remains uncertain. Another limitation is that the physicians’ prior experience with DM1 was not specifically recorded. This may have influenced individual rating variability. Including such contextual and open-ended feedback questions in future studies could provide deeper insight into low-score evaluations. Finally, given the evolving nature of generative AI models, future updates to ChatGPT-4o may produce different results, limiting the temporal applicability of the current findings. Conclusion Patients with rare neuromuscular conditions such as DM1 often face significant challenges in accessing accurate and comprehensible health information. This study represents the first structured evaluation of a generative AI model’s responses to patient-oriented questions in DM1, assessed by specialist physicians. ChatGPT-4o demonstrated strong performance across key domains such as accuracy, usefulness, and comprehensiveness, suggesting that it can provide reliable and informative content relevant to patient needs. Although slightly lower scores were observed for currency and understandability, these remained within acceptable limits, indicating that the model generally produces clear and contextually appropriate explanations even for complex medical topics. Although several pairwise comparisons among the evaluation domains reached statistical significance, the absolute differences in median scores were small. This may suggest that the observed variations are more likely related to minor perceptual differences among evaluators rather than reflecting clinically meaningful disparities in ChatGPT’s overall response quality. Overall, these findings support the potential role of large language models as supplementary tools for improving patient understanding and health communication in DM1. Future research including direct patient-based evaluations will be important to further validate these results and optimize the practical use of AI-driven communication tools in clinical settings. Acknowledgements We sincerely thank the physiatrists and neurologists for their valuable contributions in evaluating ChatGPT’s responses and providing expert insights. Abbreviations Artificial intelligence Natural language processing Large language models Myotonic dystrophy type 1 Physical Medicine and Rehabilitation Authors' contributions GC contributed to conceptualization, methodology, formal analysis, and investigation, and was a major contributor in writing the original draft as well as reviewing and editing the manuscript. MBF contributed to formal analysis and investigation, and participated in reviewing and editing the manuscript. HHH was responsible for project administration and supervision. NFT contributed to methodology and supervision. All authors read and approved the final manuscript. Funding The authors did not receive support from any organization for the submitted work. Data availability The datasets generated and analyzed during the current study are available from the corresponding author on reasonable request. Declarations Ethics approval and consent to participate This study was approved by the University of Health Sciences Antalya Training and Research Hospital Clinical Research Ethics Committee (Approval Date: January 30, 2025; Approval Number: 2/15). Informed consent was obtained from all participants included in the study. All procedures performed in studies involving human participants were conducted in accordance with the ethical standards of the institutional and/or national research committee and with the 2013 Declaration of Helsinki and its later amendments or comparable ethical standards. Consent for publication Not applicable. Competing interests The authors declare no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References Associated Data Data Availability Statement The datasets generated and analyzed during the current study are available from the corresponding author on reasonable request.