Role-based evaluation of artificial intelligence chatbot responses in orthodontic emergency scenarios: accuracy, readability, understandability, and patient-oriented communication Authors: Mustafa Ozdemir, Aysegul Gulec Published: 2026 ABSTRACT ObjectivesThis study investigated role-based differences in the accuracy, readability, understandability, and internal consistency of chatbot responses to orthodontic emergencies and examined the clinical implications of patient-oriented communication.MethodsTwenty-three standardized orthodontic emergency scenarios were presented to four chatbots— ChatGPT-4o, Claude 3 Opus, Microsoft Copilot, and Gemini 2.5—using patient and orthodontist roles. Response accuracy was evaluated by expert orthodontists and research assistants using a 3-point Likert scale, while readability, understandability, and internal consistency were assessed with the Atesman index, Sonmez formula, and Cronbach’s α. Patient-role responses were descriptively analyzed using predefined communication dimensions to contextualize quantitative findings.ResultsSignificant chatbot-specific role-based differences were observed, with Claude 3 Opus (p = 0.001) and Gemini 2.5 (p = 0.023) showing higher accuracy in the orthodontist role. In the patient role, ChatGPT-4o and Claude 3 Opus showed the highest rates of correct information, while Claude 3 Opus had the highest rate in the orthodontist role. Patient-role responses were significantly more understandable than orthodontist-role responses (p < 0.05). ChatGPT-4o (α = 0.862) and Gemini 2.5 (α = 0.815) showed high internal consistency. Qualitative analysis indicated that patient-oriented responses frequently adopted a reassuring tone and emphasized temporary self-care strategies, potentially influencing perceived urgency in orthodontic emergencies.ConclusionsChatbot performance varied according to user role. Patient-oriented responses were more understandable despite similar readability across models and could influence perceptions of urgency and professional responsibility, highlighting the need for cautious framing of chatbot-generated information for orthodontic emergency guidance.Clinical relevanceChatbots can provide preliminary information in orthodontic emergencies; however, due to limitations in accuracy and consistency, they should be used only as supportive tools and should not replace professional clinical judgment.Supplementary InformationThe online version contains supplementary material available at 10.1186/s12903-026-08836-z. Introduction Artificial intelligence (AI) chatbot-based systems have rapidly gained traction in health care and are increasingly applied in diverse areas from clinical decision support to patient education. After initial training on large datasets, chatbots can be adapted to specific domains via transfer learning and reinforcement learning enabling them to deliver specialized and flexible responses beyond general knowledge. This adaptability make chatbots particularly valuable in health communication by simplifying complex medical information, personalizing content, and improving health literacy [1]. Patient education materials are crucial for ensuring accurate understanding of clinical information and its translation into treatment behavior [2]. Chatbots’ simplicity and alignment with the health literacy levels of the target population increase compliance, especially for long-term treatments such as orthodontics, and reduce risks associated with misinformation [3]. However, to fulfill this role effectively, chatbot-generated data must be regularly evaluated in terms of readability and understandability, especially in populations with varying health literacy [4]. Ensuring that the language and complexity of chatbot responses align with the audience’s comprehension capacity is crucial for practical success and better patient outcomes. Besides patient education, chatbots can also assist clinicians in decision making. Therefore, evaluating chatbot responses in orthodontic emergency scenarios from both the patient and clinician perspective is essential. Investigating variations in chatbot performance according to user role provides valuable insights into how well these systems tailor language and content complexity to diverse audiences, informing improvements in clinical support and patient education. Recent studies have increasingly focused on the accuracy, reliability, and clinical applicability of chatbot-generated information in orthodontics. For example, Kılınç et al. [5] analyzed the responses of various ChatGPT versions to orthodontic clinical questions in terms of readability, content quality, and reliability, highlighting their potential in patient education and clinical decision support. Other studies have compared ChatGPT with chatbots such as Google Bard, examining clinical accuracy and information [6, 7]. These efforts collectively suggest promising roles for chatbots in orthodontic practice while emphasizing the necessity for ongoing evaluation to ensure validation for education, while simultaneously emphasizing the necessity to ensure both reliability and usability in clinical contexts. While previous studies have primarily focused on chatbot accuracy, the influence of user role on information quality and usefulness remains underexplored. How responses to the same orthodontic emergency scenario vary according to user role and how these differences affect information delivery, understandability, clinical guidance, and perceived urgency have not been systematically evaluated in the existing literature. Moreover, accuracy alone does not guarantee that information is readable, understandable, or internally consistent for end users. Therefore, this study introduces a role-based framework to comprehensively evaluate AI chatbot performance in orthodontic emergency scenarios. Beyond accuracy and linguistic quality, the way health-related information is communicated also plays a crucial role in patient interpretation. This communication is particularly relevant in orthodontic emergency contexts, where information delivery could influence perceived urgency and decisions to seek professional care. Patient-oriented guidance that emphasizes reassurance or temporary self-care can alter patients’ perceptions of urgency and professional responsibility, even when the information provided is clinically acceptable. Health communication literature has shown that the reassurance level, perceived urgency, and guidance framing are important factors influencing patient interpretation and care-seeking behavior. However, these communication-related dimensions have not been systematically investigated in evaluations of AI chatbot responses within orthodontic emergency settings. Therefore, in addition to quantitative performance metrics, this study also evaluates patient-oriented communication patterns to provide a more clinically contextualized interpretation of chatbot-generated information. Materials and methods The research protocol was approved by the Gaziantep University Clinical Research Ethics Committee on August 25, 2025 (protocol no: 2025/270). For this study, 23 orthodontic emergency scenarios were developed and the responses provided by four AI chatbots—ChatGPT-4o, Claude 3 Opus, Gemini 2.5, and Microsoft Copilot—to these scenarios were compared. The data collection process was conducted between July 2025, and August 2025. The scenarios were selected based on published research and orthodontic emergency protocols to represent the situations most frequently encountered by patients and clinicians during orthodontic treatment [8, 9]. These scenarios were converted into a standardized, role-based prompt structure to ensure consistent and comparable input across all chatbot models, and the corresponding prompts are presented in detail in the Supplementary Materials (Table S1). No system-level instructions or persona-setting prompts (e.g., ‘act as an orthodontist’) were used prior to the queries. Instead, the intended user role was directly embedded within each scenario. Each scenario was evaluated within the scope of two user roles: the patient and the orthodontist. A total of 46 responses (23 scenarios per role) were obtained from each chatbot. To enhance the reliability of the data, each scenario was presented to the chatbots in triplicate under identical conditions. To prevent the influence of previous conversations on chatbot performance, responses were recorded in independent sessions while maintaining the same prompt structure, with browsing history cleared. Responses were obtained using the publicly available web-based interfaces of each chatbot. No application programming interface (API) was used during data collection. Chatbots were used in their default system settings and without any parameter modifications. The chatbot responses were independently evaluated by two expert orthodontists and two orthodontic research assistant reviewers. The scores obtained from these repeated evaluations were averaged and used for the final analysis. In total, 184 responses from four chatbots were analyzed in the study. The power analysis was performed using G*Power 3.1.9.4 software with a medium effect size (f = 0.25), significance level (α = 0.05), total number of scenarios (n = 23), eight measurement conditions, inter-replication correlation (p = 0.50), and non-sphericity correction (ε = 0.75). The calculation resulted in a statistical power (1–β) of 89.3%, indicating that the 23 selected scenarios would provide sufficient power to test the hypotheses of the study. The accuracy of each response was independently assessed by two expert orthodontists and two orthodontic research assistants, using a 3-point Likert scale: 0 = incorrect, 1 = partially correct, and 2 = correct. Before the evaluations, all raters participated in a structured training session and were calibrated using representative sample responses. For each scenario, the individual scores from the four evaluators were averaged to generate a single accuracy score for statistical analysis. The readability of the responses provided by the chatbots was analyzed with the Atesman index (Fig. 1 shows the formula for the index). High values indicate that the text is easier to read. According to the Atesman index, readability levels are classified as: 1–29 (very difficult), 30–49 (difficult), 50–69 (medium difficulty), 70–89 (easy), or 90–100 (very easy). The understandability of the chatbot responses was evaluated using the Sonmez formula for which values between 0 and 0.0001 indicate that full communication is ensured, values between 0.00099 and 0.0001 suggest that the text is clear and understandable, and values between 0.03 and 0.001 indicate that the text is understandable (Fig. 2) [10]. As the value increases and moves away from 0, the level of understandability decreases. The analysis of all responses was performed using a Python-based text processing tool, and Atesman index and Sonmez scores were calculated for each chatbot and user role. Fig. 1Atesman readability index Atesman readability index Fig. 2Sonmez formula Sonmez formula The internal consistency of the chatbot-generated responses was evaluated using Cronbach’s α coefficients. To ensure a reliable assessment of consistency, the responses were collected across different sessions and at different timepoints for each chatbot. Higher α values indicate greater internal consistency within each chatbot’s output. Qualitative interpretation of patient-oriented communications In addition to quantitative analyses, responses generated from the patient role were examined using a structured qualitative interpretation framework to provide clinical context. Accordingly, three communication dimensions—reassurance level, perceived urgency, and guidance framing—were defined a priori based on contemporary health communication research and recent evaluations of patient-facing chatbot systems [11–14]. Patient-role responses were descriptively reviewed for the presence and emphasis of reassuring expressions, urgency-related wording, and the relative prioritization of professional consultation versus temporary self-care advice. The qualitative component was intended to contextualize quantitative findings rather than generate new themes or theory, and no formal theme or subtheme development process was performed. The qualitative evaluation was performed independently by two expert orthodontists and two orthodontic research assistants following a structured calibration session. Disagreements were resolved after consensus discussions. Qualitative observations and response categorizations were organized using Microsoft Excel without specialized qualitative analysis software. The conceptual framework informing this qualitative interpretation, derived from recent digital health literature, is summarized in Fig. 3. Fig. 3Conceptual framework for qualitative interpretation of patient-oriented communication Conceptual framework for qualitative interpretation of patient-oriented communication Statistical analysis Data analysis was performed using SPSS 26.0 software. Readability (Atesman index) and understandability (Sonmez) scores, along with accuracy scores, were summarized by calculating the mean ± standard deviation values. Numerical and percentage distributions are provided for categorical data. The normality of the data distribution was assessed based on skewness and kurtosis values, and all numerical data followed a normal distribution. For cross-role comparisons, a paired samples T-test was performed, whereas comparisons among chatbots within the same role were analyzed using repeated measures ANOVA followed by Bonferroni post-hoc corrections. An intraclass correlation coefficient (ICC) analysis was performed to determine consistency among evaluators. The reliability of the chatbot responses was evaluated by calculating Cronbach’s α coefficient. A significance level of p < 0.05 was accepted in all statistical analyses. Results The reliability of the ratings provided by the four evaluators (i.e., the two experts and two research assistants) was high (ICC 0.812, 95% CI 0.701–0.892, p < 0.001). Accuracy No significant differences in accuracy were observed when the chatbots were compared while keeping the user role constant. Comparing the performance of each chatbot across the patient and orthodontist roles revealed that ChatGPT-4o and Copilot had similar performance across both roles (p2 > 0.05), whereas Claude 3 Opus (p1 = 0.001) and Gemini 2.5 (p1 = 0.023) produced significantly more accurate responses for the orthodontist role (Table 1). The mean accuracy values suggested that, in both roles, the responses were largely within the ‘partially accurate to accurate’ range (Table 2). Table 1Accuracy scores of responses generated by chatbots according to user role (patient/orthodontist) based on expert evaluations (p < 0.05, p < 0.01)VariableChatbotPatient RoleOrthodontist RoleMean.±S.DMean.±S.Dp 1AccuracyChatGPT-4o1.09 ± 0.441.13 ± 0.520.549Claude 3 Opus1.04 ± 0.531.27 ± 0.49 0.001** Copilot1.00 ± 0.471.10 ± 0.360.129Gemini 2.50.93 ± 0.391.07 ± 0.51 0.023* p 20.5920.107* p < 0.05, ** p < 0.01p 1: Comparison of readability results for each chatbotl in the patient role and orthodontist role (Paired Samples t-Test)p 2: Comparison of readability results for each role across 4 different chatbots (Repeated Measures ANOVA). Three-point accuracy score (0 = incorrect, 1 = partially correct, 2 = correct)” Table 1Accuracy scores of responses generated by chatbots according to user role (patient/orthodontist) based on expert evaluations (p < 0.05, p < 0.01)VariableChatbotPatient RoleOrthodontist RoleMean.±S.DMean.±S.Dp 1AccuracyChatGPT-4o1.09 ± 0.441.13 ± 0.520.549Claude 3 Opus1.04 ± 0.531.27 ± 0.49 0.001** Copilot1.00 ± 0.471.10 ± 0.360.129Gemini 2.50.93 ± 0.391.07 ± 0.51 0.023* p 20.5920.107* p < 0.05, ** p < 0.01p 1: Comparison of readability results for each chatbotl in the patient role and orthodontist role (Paired Samples t-Test)p 2: Comparison of readability results for each role across 4 different chatbots (Repeated Measures ANOVA). Three-point accuracy score (0 = incorrect, 1 = partially correct, 2 = correct)” Accuracy scores of responses generated by chatbots according to user role (patient/orthodontist) based on expert evaluations (p < 0.05, p < 0.01) * p < 0.05, ** p < 0.01 p 1: Comparison of readability results for each chatbotl in the patient role and orthodontist role (Paired Samples t-Test) p 2: Comparison of readability results for each role across 4 different chatbots (Repeated Measures ANOVA). Three-point accuracy score (0 = incorrect, 1 = partially correct, 2 = correct)” Table 2Accuracy level distributions of responses generated by chatbots according to user role based on expert evaluationsRoleChatbotLevel of accuracyIncorrect InformationPartially Correct InformationCorrect InformationNumber%Number%Number%PatientChatGPT-4o14.31982.6313.0Claude 3 Opus313.01773.9313.0Copilot28.71982.628.7Gemini 2.528.72087.014.3OrthodontistChatGPT-4o28.71669.6521.7Claude 3 Opus00.01773.9626.1Copilot00.02191.328.7Gemini 2.528.71878.3313.0Three-point accuracy score (0 = incorrect, 1 = partially correct, 2 = correct) Table 2Accuracy level distributions of responses generated by chatbots according to user role based on expert evaluationsRoleChatbotLevel of accuracyIncorrect InformationPartially Correct InformationCorrect InformationNumber%Number%Number%PatientChatGPT-4o14.31982.6313.0Claude 3 Opus313.01773.9313.0Copilot28.71982.628.7Gemini 2.528.72087.014.3OrthodontistChatGPT-4o28.71669.6521.7Claude 3 Opus00.01773.9626.1Copilot00.02191.328.7Gemini 2.528.71878.3313.0Three-point accuracy score (0 = incorrect, 1 = partially correct, 2 = correct) Accuracy level distributions of responses generated by chatbots according to user role based on expert evaluations Three-point accuracy score (0 = incorrect, 1 = partially correct, 2 = correct) Readability There was no significant difference in readability between the patient and orthodontist roles for any of the chatbots (p1 > 0.05; Table 3). Readability scores ranged from 63.58 to 67.80 across all chatbots and roles. No significant differences were observed among the chatbots in either the patient role or the orthodontist role (p2 > 0.05; Table 3). Furthermore, the distribution of readability categories revealed that, for both user roles, most responses were classified within the ‘moderate difficulty’ and ‘easy’ categories (Table 4). Table 3Readability of chatbots responses according to user rolesVariableChatbotPatient RoleOrthodontist RoleMean.±S.DMean.±S.Dp 1Readability(Atesman Index)ChatGPT-4o67.80 ± 7.1566.92 ± 12.000.722Claude 3 Opus63.58 ± 12.1067.45 ± 11.420.131Copilot64.97 ± 9.0364.79 ± 10.810.947Gemini 2.566.32 ± 11.8665.52 ± 9.850.779p 20.2360.718p < 0.05, * p < 0.01p 1: Comparison of readability results for each chatbot in the patient role and orthodontist role (Paired Samples t-Test)p 2: Comparison of readability results for each role across 4 different chatbots (Repeated Measures ANOVA)Atesman readability index: 1–29: very difficult; 30–49: difficult; 50–69: medium difficulty; 70–89: easy; 90–100: very easy Table 3Readability of chatbots responses according to user rolesVariableChatbotPatient RoleOrthodontist RoleMean.±S.DMean.±S.Dp 1Readability(Atesman Index)ChatGPT-4o67.80 ± 7.1566.92 ± 12.000.722Claude 3 Opus63.58 ± 12.1067.45 ± 11.420.131Copilot64.97 ± 9.0364.79 ± 10.810.947Gemini 2.566.32 ± 11.8665.52 ± 9.850.779p 20.2360.718p < 0.05, * p < 0.01p 1: Comparison of readability results for each chatbot in the patient role and orthodontist role (Paired Samples t-Test)p 2: Comparison of readability results for each role across 4 different chatbots (Repeated Measures ANOVA)Atesman readability index: 1–29: very difficult; 30–49: difficult; 50–69: medium difficulty; 70–89: easy; 90–100: very easy Readability of chatbots responses according to user roles Readability (Atesman Index) p < 0.05, * p < 0.01 p 1: Comparison of readability results for each chatbot in the patient role and orthodontist role (Paired Samples t-Test) p 2: Comparison of readability results for each role across 4 different chatbots (Repeated Measures ANOVA) Atesman readability index: 1–29: very difficult; 30–49: difficult; 50–69: medium difficulty; 70–89: easy; 90–100: very easy Table 4Readability level distributions of chatbots generated responses according to user rolesRoleChatbotReadability (Atesman Index) CategoriesDifficultModerately difficultEasyVery easyNumber%Numberı%Number%NumberPatientChatGPT-4o00.01221.11139.300,0Claude 3 Opus360.01424.6517.9150.0Copilot120.01526.3725.000.0Gemini 2.5120.01628.1517.9150.0OrthodontistChatGPT-4o114.31223.11031.300.0Claude 3 Opus342.91223.1825.000.0Copilot114.31528.8618.81100.0Gemini 2.5228.61325.0825.000.0Atesman readability index: 1–29: very difficult; 30–49: difficult; 50–69: medium difficulty; 70–89: easy; 90–100: very easy Table 4Readability level distributions of chatbots generated responses according to user rolesRoleChatbotReadability (Atesman Index) CategoriesDifficultModerately difficultEasyVery easyNumber%Numberı%Number%NumberPatientChatGPT-4o00.01221.11139.300,0Claude 3 Opus360.01424.6517.9150.0Copilot120.01526.3725.000.0Gemini 2.5120.01628.1517.9150.0OrthodontistChatGPT-4o114.31223.11031.300.0Claude 3 Opus342.91223.1825.000.0Copilot114.31528.8618.81100.0Gemini 2.5228.61325.0825.000.0Atesman readability index: 1–29: very difficult; 30–49: difficult; 50–69: medium difficulty; 70–89: easy; 90–100: very easy Readability level distributions of chatbots generated responses according to user roles Atesman readability index: 1–29: very difficult; 30–49: difficult; 50–69: medium difficulty; 70–89: easy; 90–100: very easy Understandability When understandability scores were compared across user roles irrespective of chatbot type, responses generated in the patient role were significantly more understandable than those generated in the orthodontist role (p < 0.05). A significant difference among chatbots was identified only in the patient role (p2 < 0.05; Table 5). Claude 3 Opus was more understandable than Gemini 2.5 in the patient role (p = 0.030). Furthermore, in the patient role, the highest distributions were observed in the ‘text is understandable’ category. In the orthodontist role, all responses across all chatbots were classified solely within the ‘text is understandable’ category (Table 6). Table 5Understandability of chatbots generated responses acording to user rolesVariableChatbotPatient RoleOrthodontist RoleMean.±S.DMean.±S.Dp 1Understandability (Sonmez Formula)ChatGPT-4o10.0053 ± 0.00370.0088 ± 0.0056 0.010** Claude 3 Opus20.0031 ± 0.00250.0079 ± 0.004 0.015* Copilot30.0048 ± 0.00450.0088 ± 0.006 0.000** Gemini 2.540.0057 ± 0.00460.0094 ± 0.0052 0.024* p 2Post Hoc(Bonferroni) Test 0.049* 1—2, p:0.054 2–4, p:0.030 3–4, p:1.000 1–3, p:1.0000.447-* p < 0.05, **p < 0.01p 1: Comparison of Understandability results for patient and orthodontist roles for each chatbot (Paired Samples t-test)p 2: Comparison of Understandability results for each role across four different chatbots (Repeated Measures ANOVA)Sonmez understandability formula: 0–0.0001: Full communication is ensured, 0.00099–0.0001: The text is clear and understandable, 0.03–0.001: The text is understandable Table 5Understandability of chatbots generated responses acording to user rolesVariableChatbotPatient RoleOrthodontist RoleMean.±S.DMean.±S.Dp 1Understandability (Sonmez Formula)ChatGPT-4o10.0053 ± 0.00370.0088 ± 0.0056 0.010** Claude 3 Opus20.0031 ± 0.00250.0079 ± 0.004 0.015* Copilot30.0048 ± 0.00450.0088 ± 0.006 0.000** Gemini 2.540.0057 ± 0.00460.0094 ± 0.0052 0.024* p 2Post Hoc(Bonferroni) Test 0.049* 1—2, p:0.054 2–4, p:0.030 3–4, p:1.000 1–3, p:1.0000.447-* p < 0.05, **p < 0.01p 1: Comparison of Understandability results for patient and orthodontist roles for each chatbot (Paired Samples t-test)p 2: Comparison of Understandability results for each role across four different chatbots (Repeated Measures ANOVA)Sonmez understandability formula: 0–0.0001: Full communication is ensured, 0.00099–0.0001: The text is clear and understandable, 0.03–0.001: The text is understandable Understandability of chatbots generated responses acording to user roles p 2 Post Hoc (Bonferroni) Test 0.049* 1—2, p:0.054 2–4, p:0.030 3–4, p:1.000 1–3, p:1.000 0.447 - * p < 0.05, **p < 0.01 p 1: Comparison of Understandability results for patient and orthodontist roles for each chatbot (Paired Samples t-test) p 2: Comparison of Understandability results for each role across four different chatbots (Repeated Measures ANOVA) Sonmez understandability formula: 0–0.0001: Full communication is ensured, 0.00099–0.0001: The text is clear and understandable, 0.03–0.001: The text is understandable Table 6Understandability categories of chatbots responses according to user rolesRoleChatbotSonmez CategoryThe text is understandableThe text is clear and understandableFull communication is ensuredNumber%Number%Number%PatientChatGPT-4o2226.2116.700.0Claude 3 Opus1922.6350.0150.0Copilot2125.0233.300Gemini 2.52226.200.0150.0OrthodontistChatGPT-4o2325.000.000.0Claude 3 Opus2325.000.000.0Copilot2325.000.000.0Gemini 2.52325.000.000.0Sonmez understandability formula: 0–0.0001: Full communication is ensured, 0.00099–0.0001: The text is clear and understandable, 0.03–0.001: The text is understandable Table 6Understandability categories of chatbots responses according to user rolesRoleChatbotSonmez CategoryThe text is understandableThe text is clear and understandableFull communication is ensuredNumber%Number%Number%PatientChatGPT-4o2226.2116.700.0Claude 3 Opus1922.6350.0150.0Copilot2125.0233.300Gemini 2.52226.200.0150.0OrthodontistChatGPT-4o2325.000.000.0Claude 3 Opus2325.000.000.0Copilot2325.000.000.0Gemini 2.52325.000.000.0Sonmez understandability formula: 0–0.0001: Full communication is ensured, 0.00099–0.0001: The text is clear and understandable, 0.03–0.001: The text is understandable Understandability categories of chatbots responses according to user roles Sonmez understandability formula: 0–0.0001: Full communication is ensured, 0.00099–0.0001: The text is clear and understandable, 0.03–0.001: The text is understandable Internal consistency of chatbots ChatGPT-4o (α = 0.862) and Gemini 2.5 (α = 0.815) showed good internal consistency. In contrast, Copilot (α = 0.547) showed poor internal consistency, whereas Claude 3 Opus (α = 0.298) exhibited unacceptable internal consistency (Table 7). Table 7Cronbach’s alpha results and consistency levels of chatbotsGroupCronbach’s AlfaConsistency LevelChatGPT-4o0.862Good internal consistentGemini 2.50.815Good internal consistentCopilot0.547Poor internal consistentClaude 3 Opus0.298UnacceptableCronbach’s α: Internal consistency coefficient. Values between 0.80–0.89 indicate good internal consistency, 0.70–0.79 indicate acceptable consistency, 0.6–0.69 indicate questionable consistency, 0.5–0.59 indicate poor consistency, less than 0.5 indicate unacceptable consistency Table 7Cronbach’s alpha results and consistency levels of chatbotsGroupCronbach’s AlfaConsistency LevelChatGPT-4o0.862Good internal consistentGemini 2.50.815Good internal consistentCopilot0.547Poor internal consistentClaude 3 Opus0.298UnacceptableCronbach’s α: Internal consistency coefficient. Values between 0.80–0.89 indicate good internal consistency, 0.70–0.79 indicate acceptable consistency, 0.6–0.69 indicate questionable consistency, 0.5–0.59 indicate poor consistency, less than 0.5 indicate unacceptable consistency Cronbach’s alpha results and consistency levels of chatbots Cronbach’s α: Internal consistency coefficient. Values between 0.80–0.89 indicate good internal consistency, 0.70–0.79 indicate acceptable consistency, 0.6–0.69 indicate questionable consistency, 0.5–0.59 indicate poor consistency, less than 0.5 indicate unacceptable consistency Qualitative interpretation of patient-oriented communication patterns Patient-role responses were descriptively analyzed in accordance with the predefined qualitative interpretation framework to identify communication patterns related to reassurance level, perceived urgency, and guidance framing. Overall, chatbot responses commonly employed a reassuring tone and emphasized temporary self-care strategies. Perceived urgency varied across scenarios, with several responses framing orthodontic emergencies as manageable in the short term and professional consultation as non-immediate. Guidance framing frequently prioritized interim self-care, with professional evaluation presented as advisable rather than urgent. An overview of these communication patterns across scenarios is presented in Table 8. Table 8Scenario-based qualitative interpretation of patient-oriented chatbot responsesScenario (Patient question)Common communication pattern of responsesPotential clinical interpretation (Risk label)1. A bracket has come offReassuring tone, temporary self-care advice, appointment recommendationPossible delay in professional consultation2. A band has come off my toothEmphasis on being common and easily manageablePotential unwanted tooth movement3. A metal piece on the back of my tooth broke offTemporary at-home relief emphasizedProlongation of soft-tissue irritation4. The end of my wire is poking my cheekOrthodontic wax and home remedies prioritizedPersistence of mucosal trauma5. The appliance in my mouth broke and cannot be usedReassuring and solvable framingReduced treatment effectiveness6. The rubber chain came outPerceived as non-urgentDelay in tooth movement7. A thin wire is poking my cheek/lipTemporary at-home measures emphasizedRecurrent or prolonged soft-tissue irritation8. The wire behind my teeth became looseCalming tone and observation adviceLoss of retention control9. My gums are bleedingNormalization and hygiene reassurancePossible overlooked gingival condition10. It hurts when I close my mouthFramed as temporary sensitivityDelayed mechanical evaluation11. A small piece on the inside of my tooth came offHome care advice with appointment suggestionFunctional underestimation12. A screw in my mouth has come looseCalm tone with clinical follow-up recommendationProgressive inflammation risk13. My rubber bands have worn outShort-term tolerance impliedReduced compliance14. I have noise/pain in my jawEmphasized as common or transientDelayed TMD evaluation15. The area around my screw is swollenTemporary care with follow-up adviceInflammation progression16. I have not been wearing my retainers regularlyMotivational and advisory toneRelapse risk17. Attachments from my clear aligners have fallen offReassuring, reattachment possibleReduced force control18. My aligner is irritating my cheekHome relief measures prioritizedMucosal irritation19. The edge of my aligner crackedAlternative aligner use suggestedTreatment delay20. I have not been wearing my elastics regularlyImportance noted but gently framedProlonged treatment21. My braces are coming out and I cannot wear themReassuring tone with appointment adviceTemporary treatment cessation22. I cannot brush my teeth properlySupportive educational guidanceIncreased caries and periodontal risk23. I feel pain around my braces when brushingNormal sensitivity emphasizedCompromised oral hygiene Table 8Scenario-based qualitative interpretation of patient-oriented chatbot responsesScenario (Patient question)Common communication pattern of responsesPotential clinical interpretation (Risk label)1. A bracket has come offReassuring tone, temporary self-care advice, appointment recommendationPossible delay in professional consultation2. A band has come off my toothEmphasis on being common and easily manageablePotential unwanted tooth movement3. A metal piece on the back of my tooth broke offTemporary at-home relief emphasizedProlongation of soft-tissue irritation4. The end of my wire is poking my cheekOrthodontic wax and home remedies prioritizedPersistence of mucosal trauma5. The appliance in my mouth broke and cannot be usedReassuring and solvable framingReduced treatment effectiveness6. The rubber chain came outPerceived as non-urgentDelay in tooth movement7. A thin wire is poking my cheek/lipTemporary at-home measures emphasizedRecurrent or prolonged soft-tissue irritation8. The wire behind my teeth became looseCalming tone and observation adviceLoss of retention control9. My gums are bleedingNormalization and hygiene reassurancePossible overlooked gingival condition10. It hurts when I close my mouthFramed as temporary sensitivityDelayed mechanical evaluation11. A small piece on the inside of my tooth came offHome care advice with appointment suggestionFunctional underestimation12. A screw in my mouth has come looseCalm tone with clinical follow-up recommendationProgressive inflammation risk13. My rubber bands have worn outShort-term tolerance impliedReduced compliance14. I have noise/pain in my jawEmphasized as common or transientDelayed TMD evaluation15. The area around my screw is swollenTemporary care with follow-up adviceInflammation progression16. I have not been wearing my retainers regularlyMotivational and advisory toneRelapse risk17. Attachments from my clear aligners have fallen offReassuring, reattachment possibleReduced force control18. My aligner is irritating my cheekHome relief measures prioritizedMucosal irritation19. The edge of my aligner crackedAlternative aligner use suggestedTreatment delay20. I have not been wearing my elastics regularlyImportance noted but gently framedProlonged treatment21. My braces are coming out and I cannot wear themReassuring tone with appointment adviceTemporary treatment cessation22. I cannot brush my teeth properlySupportive educational guidanceIncreased caries and periodontal risk23. I feel pain around my braces when brushingNormal sensitivity emphasizedCompromised oral hygiene Scenario-based qualitative interpretation of patient-oriented chatbot responses Discussion This study provides a comprehensive evaluation of ChatGPT-4o, Claude 3 Opus, Copilot, and Gemini 2.5 chatbots and their performance in orthodontic emergency scenarios in terms of accuracy, readability, understandability, and consistency from two perspectives: the patient and the orthodontist. The findings show the decisive influence of user role on response accuracy and linguistic characteristics. Moreover, these results highlight the context-sensitive language generation capacity of chatbots and provide a strong methodological contribution to the literature regarding their reliability and role-based performance variations in orthodontic emergency management. Although similar communication dimensions have been extensively examined in health communication and risk communication literature, this study is among the first to integrate reassurance level, perceived urgency, and guidance framing into a structured qualitative framework for interpreting patient-oriented chatbot responses in orthodontic emergency contexts. The accuracy of chatbot-generated responses in orthodontic emergency scenarios was predominantly classified as ‘partially correct’, consistent with previous reports [6, 7], indicating that these systems cannot yet be considered solely as a reliable source for clinical decision-making processes. When the influence of user role on accuracy was examined, only Claude 3 Opus and Gemini 2.5 produced significantly more accurate responses for the orthodontist role. However, when overall accuracy levels were compared across chatbots, no significant differences were observed, indicating a similar reliability profile among the systems in emergency settings. The findings of our study partly align with those of Erdem et al., who reported Gemini and ChatGPT-4o as the more accurate chatbots [15]. However, the methodological differences between the two studies—particularly regarding the scoring systems employed—are noteworthy. The use of a 0–2 scoring scale in our study is consistent with the approach adopted by Dogan et al. [16] supporting the methodological coherence of this evaluation framework. Broader scoring ranges might assign points to details that are not clinically decisive in emergency contexts, thereby creating an inflated perception of accuracy. In contrast, the narrow 0–2 scale used in this study allows for a more meaningful assessment of how well responses meet essential clinical requirements. Considering that broader scales might overemphasize elements with limited clinical relevance and generate a misleading sense of reliability in emergency assessments, the 0–2 scoring system offers a more balanced and sensitive measurement approach. These findings suggest that, despite having an acceptable baseline performance, current chatbots should not be used as standalone decision-support tools for orthodontic emergency management. The readability levels of chatbot-generated responses indicate no significant difference between patient and orthodontist roles. However, the distributions reveal a linguistic differentiation between the two roles. While chatbots in the patient role adopt a simpler and more understandable style, the concentration of technical terminology and professional expressions in the orthodontist role increased the textual difficulty (Table S2). This qualitative finding suggests that chatbots develop a language strategy appropriate for the target audience, a finding that is consistent with published studies. Kılınç and Mansız have reported that the presence of comprehensive and technically detailed texts prepared in the field of orthodontics has a detrimental effect on the readability of chatbot responses [5]. Demirsoy et al. reported that the complexity of treatment content, including clear aligners, lingual orthodontics, and temporomandibular joint hinders the simplification of language in chatbot responses, thereby compromising readability [17]. As the technical details increase, the linguistic load becomes more substantial and the readability declines. The role-based linguistic differentiation observed in our study is consistent with these findings, indicating that linguistic complexity is influenced not only by content type but also by the perspective of the targeted role. In the patient role, the use of simplified language and limited technical content results in increased readability. Conversely, in the orthodontist role, the texts are characterized by an increased professional knowledge load and terminology density, leading to a more challenging reading experience. The understandability levels of chatbot-generated responses differed significantly depending on the targeted user role (Table 5). The lower understandability observed in the orthodontist role compared with the patient role could be associated with the greater technical content and longer explanatory structures of responses generated for this role. In contrast, responses produced for the patient role were presented in a shorter, more direct, and easier-to-understand language. For the role-based analysis, Claude 3 Opus was more understandable than Gemini 2.5 in the patient role (p = 0.030), while no significant difference was observed for the orthodontist role (p2 = 0.447). This finding indicates that the level of technical content in the orthodontist role is generally similar across chatbots and that this technical emphasis reduces the understandability of the responses (Table S2). In orthodontic emergencies, AI chatbots could provide supportive patient guidance outside clinical hours; however, role-appropriate language adaptation is essential to ensure clarity and appropriate interpretation of information. In this context, although responses generated in the orthodontist role appear appropriate for professional use, they could require additional linguistic adaptation and simplification for direct patient communication. These findings show that the development of user role-sensitive language generation strategies is of crucial importance for the effective use of chatbots as supportive tools in clinical practice. When the chatbot models were compared, ChatGPT-4o and Gemini 2.5 demonstrated good internal consistency, Copilot showed poor internal consistency, and Claude 3 Opus exhibited unacceptable internal consistency. Similarly, in the evaluation of general orthodontic questions conducted by Metin et al., the ChatGPT-4o chatbot produced more stable responses [18]. Despite differences in the content evaluated in these two studies, the emergence of similar consistency patterns suggests that consistency levels might be more closely related to chatbot-specific characteristics rather than to the type of content evaluated. The qualitative analysis of patient-role responses revealed distinct communication patterns characterized by reassuring language, variable urgency signaling, and guidance framing that frequently emphasized temporary self-care ahead of immediate professional consultation. These findings justify the use of a structured qualitative interpretation framework in addition to quantitative analyses, as the clinical implications of chatbot-generated information extend beyond factual accuracy. Consistent with recent digital health literature, reassuring communication can enhance patient understanding and trust but might simultaneously attenuate perceived urgency and delay care-seeking behavior. Moreover, the tendency to frame orthodontic emergencies as manageable in the short term and to present professional consultation as advisable rather than urgent suggests that chatbot communication style can influence patients’ perceptions of risk and professional responsibility. The prioritization of interim self-care strategies further indicates that patient-oriented chatbot responses could encourage autonomous management rather than prompt clinical evaluation. Collectively, these findings highlight the importance of evaluating not only the accuracy but also the communicative framing of chatbot responses, as subtle linguistic features could meaningfully affect patient interpretation and clinical decision making in orthodontic emergency contexts. Limitations This study provides a multidimensional evaluation of chatbot performance in orthodontic emergency scenarios; however, several limitations should be acknowledged. First, the findings are limited to the available versions of four chatbots during the study period, and rapid system updates could restrict the generalizability of the results to other chatbots or newer versions of the same models. Second, although the 23 orthodontic emergency scenarios represent the most frequently reported conditions in the literature, they do not encompass rare or complex clinical situations. Third, despite high inter-rater agreement in the accuracy assessments, the scoring process inherently involved a degree of subjectivity due to the interpretation of clinical content. Moreover, the study focused on the evaluation of chatbot responses based on content and linguistic characteristics and did not directly assess their impact on clinical decision making or real-time patient management. The qualitative interpretation of patient-oriented communication patterns was descriptive and did not quantify patient behavior or clinical outcomes. Lastly, the reliance on predefined communication dimensions and standardized scenarios could limit the generalizability of the qualitative findings to real-world clinical interactions. Conclusions In conclusion, chatbot-generated responses to orthodontic emergencies showed limited reliability and should not be considered as standalone tools for clinical decision making. The findings highlight that response characteristics vary depending on user role, particularly in terms of communication style and understandability. While chatbots can support patient information needs, especially outside clinical settings, their use requires cautious interpretation. In particular, patient-oriented responses can influence perceptions of urgency with reassuring language and emphasis on temporary self-care. These results underscore the importance of role-sensitive communication strategies and reinforce that chatbot-generated information should only complement, not replace, professional clinical judgment. Supplementary Information Supplementary Material 1. Supplementary Material 1. Supplementary Material 2. Supplementary Material 2.