AI-powered simulated patients with automated feedback for enhancing headache history-taking skills: a convergent mixed-methods study Authors: Kridipaka Sindhvananda, Kewalin Ruengwattanachot, Surachai Leksuwankun, Thanakit Pongpitakmetha, Akarin Hiransuthikul, Totsapol Surawattanawong, Prakit Anukoolwittaya, Poonnakarn Panjasriprakarn Published: 2026 ABSTRACT BackgroundMedical students often struggle with headache disorders, which are common but diagnostically challenging due to limited clinical exposure and feedback during early clinical training. This study evaluated the feasibility and effectiveness of an AI-powered simulated patient (AISP) chatbot in enhancing history-taking and clinical reasoning for three primary headache disorders—migraine, tension-type, and cluster headache—among medical students at a tertiary university hospital in Thailand.MethodsWe developed a simulated patient platform that provided automated, rubric-based feedback across three headache cases. Third-year medical students who had completed pre-clerkship requirements participated in a mixed-methods experimental study with a pre-post design. The study consisted of a 1-week self-study phase followed by a pre-test Objective Structured Clinical Examination (OSCE), a 1-week washout period, and a 1-week AISP practice phase followed by a post-test OSCE. Quantitative outcomes included changes in OSCE scores, Clinical Reasoning Indicator-History Taking (CRI-HT) scores, usability/satisfaction measured using the Chatbot Usability Questionnaire (CUQ). Focus group discussions (FGDs) explored participants’ learning experiences, perceived system limitations, and recommendations for future implementation. Qualitative data were analyzed thematically.ResultsTwenty-five medical students participated. OSCE scores increased by 22.5 points (p < 0.001, Cohen’s d = 1.93), and CRI-HT scores increased by 8.8 points (p < 0.001, Cohen’s d = 1.81). CUQ findings indicated high usability. Students commonly cited the platform’s intuitive interface and informative feedback, although some noted the chatbot’s responses were overly robotic. FGDs further highlighted three themes: (1) convenient, structured practice support learning; (2) dialogue felt realistic but was limited for practicing communication skills; and (3) students recommended greater case variety and integration into the curriculum.ConclusionsAn AISP platform with automated, rubric-based feedback was feasible and associated with improvements in headache history-taking and clinical reasoning. It serves as a valuable complement to traditional teaching by enabling structured, repeated practice with timely feedback.Supplementary InformationThe online version contains supplementary material available at 10.1186/s12909-026-09511-1. Article highlights / Key findings An AI-powered simulated patient with automated feedback was used to support headache history-taking practice among medical students.Students showed improvement in OSCE performance and clinical reasoning scores after using the system.Learners with lower baseline performance demonstrated larger gains, narrowing the performance gap between baseline groups.Usability findings and focus group data suggested that the system supported structured learning and confidence building, although emotional realism remained limited. An AI-powered simulated patient with automated feedback was used to support headache history-taking practice among medical students. An AI-powered simulated patient with automated feedback was used to support headache history-taking practice among medical students. Students showed improvement in OSCE performance and clinical reasoning scores after using the system. Students showed improvement in OSCE performance and clinical reasoning scores after using the system. Learners with lower baseline performance demonstrated larger gains, narrowing the performance gap between baseline groups. Learners with lower baseline performance demonstrated larger gains, narrowing the performance gap between baseline groups. Usability findings and focus group data suggested that the system supported structured learning and confidence building, although emotional realism remained limited. Usability findings and focus group data suggested that the system supported structured learning and confidence building, although emotional realism remained limited. Introduction Headache disorders are among the most common neurological conditions and contribute substantially to the global burden of neurological disability [1, 2]. Despite their high prevalence, formal training in headache medicine remains insufficient across both undergraduate and postgraduate medical curricula [3–5]. Surveys from multiple countries have identified persistent barriers, including constrained curriculum time, limited specialist teaching capacity, and variability in instructional approaches [6, 7]. For instance, a survey by the American Headache Society reported that medical students receive only a small amount of dedicated teaching in headache medicine [6], and residency programs often report gaps in formal headache training that negatively affect patient care [7]. Together, these findings highlight a need for more accessible and scalable educational approaches to help address the gap between the burden of headache disorders and current training exposure [3–7]. Addressing this gap is educationally challenging because the diagnosis of primary headache disorders relies almost entirely on comprehensive history-taking [8]. Accurate diagnosis of migraine, for example, requires recognition of key clinical hallmarks through careful patient interview rather than laboratory or imaging investigations [8, 9]. Medical students often have limited and inconsistent opportunities to practice this structured history-taking in clinical settings, where patient encounters are unpredictable in timing and case variety [10, 11]. Standardized patients (SPs) are widely recognized as the gold standard for teaching history-taking and communication skills in a safe, feedback-driven environment [12]. However, SP programs require actor recruitment, training, and supervision, which limit their scalability for large cohorts and frequent repetition [13]. Virtual patient technologies have emerged as a potential complement to SP-based training, with systematic reviews demonstrating that they can improve clinical reasoning and knowledge acquisition in a variety of medical education contexts [14–16]. Recent advances in artificial intelligence (AI) have demonstrated increasing roles in headache research and clinical care, including applications in diagnosis, monitoring, and personalized management [17–19]. Although AI-based systems can improve efficiency in headache medicine, they cannot replace the human elements of clinical care that remain essential to the doctor-patient relationship [17]. AI-powered simulated patients (AISPs) built on large language models (LLMs) can support interactive clinical conversations through natural language processing and speech recognition [20–22]. These systems allow learners to practice history-taking, receive structured feedback, and repeat cases in a flexible and low-pressure setting [22]. Prior studies suggest that chatbot-based training with structured feedback can improve history-taking skills and clinical reasoning [23], and that chatbot-based environments can support self-directed practice in health professions education [24, 25]. Recent work suggests that LLM-based simulated patients can provide educationally meaningful interactions and may serve as scalable complements to traditional SP-based teaching [16, 21, 22, 26]. Given that headache remains frequently underdiagnosed in primary care [27] and that management gaps persist in clinical headache care [28, 29], headache medicine represents an ideal domain for implementing and evaluating AISP-based learning. Although LLM-based simulated patients have been studied for history-taking training in other clinical domains [22, 30, 31], evidence specific to headache-focused history-taking education remains limited. An AISP platform designed for headache cases could provide a structured environment in which students practice the systematic questioning needed for differential diagnosis, while receiving immediate, rubric-based performance feedback grounded in the principles of deliberate practice [32]. This study therefore evaluated the feasibility and effectiveness of an AI-powered simulated patient chatbot in improving headache history-taking and clinical reasoning among medical students at a tertiary university hospital in Thailand. Specifically, the study aimed (1) to evaluate the effectiveness of Cortex®, a custom-built AISP chatbot, in improving headache history-taking performance and clinical reasoning among undergraduate medical students using a single-group pre-post design, and (2) to explore challenges and opportunities for integrating AISP into medical education through usability assessment and focus group interviews. Methods Enrollment and study population Third-year medical students who had completed pre-clerkship requirements and the Foundation of Clinical Competency course at the Faculty of Medicine, Chulalongkorn University were voluntarily recruited to participate in this mixed-methods, quasi-experimental study between March and April 2025. During the Foundation of Clinical Competency course, students received training in the principles of history-taking and basic diagnosis, including coverage of headache patient history-taking. The study was conducted as part of a four-week Student-Selected Component (SSC) elective module designed to introduce students to applications of artificial intelligence in healthcare education and clinical neurology. Human ethics and consent to participate This study was approved by the Institutional Review Board of the Faculty of Medicine, Chulalongkorn University, Bangkok, Thailand (IRB Approval No. 338/2025), and was conducted in accordance with the Declaration of Helsinki. All participants provided written informed consent prior to any study procedures. Clinical trial number: not applicable. Intervention: AISP chatbot (Cortex) Cortex is a custom-built, voice-enabled web application that simulates clinical encounters for history-taking practice. The system is powered by GPT-4o (OpenAI, San Francisco, CA, USA) and employs a dual-prompt architecture. The first prompt governs patient simulation: it constrains the model’s responses to a predefined patient script to reduce off-script generation (i.e., hallucination risk) and maintain clinical consistency across interactions. The second prompt drives the automated evaluation module, which generates rubric-based feedback after each session. Students could interact with the system through both voice input (using integrated speech recognition) and text input. Students selected one of three primary headache cases (migraine, tension-type headache, or cluster headache). Each case was created by neurologists and independently reviewed by three board-certified neurologists for clinical accuracy. Case content and diagnostic features were developed in accordance with the International Classification of Headache Disorders, 3rd edition (ICHD-3) [33]. Each case included standardized demographic details and a predefined clinical script. The chatbot responded only to user-initiated questions and generated responses constrained by the script. After each session, students recorded their diagnostic impression and received automated, rubric-based feedback, including a comparison with the reference diagnosis (Fig. 1). Full application details, system prompts, and the technical architecture are provided in Supplementary Material 1. Fig. 1Workflow of the AI-powered simulated patient chatbot (Cortex). (1) Learners select a simulated patient profile, each representing a predefined headache case. (2) The chatbot conducts an interactive history-taking dialogue through text or voice input. (3) After the session, the system provides automated rubric-based feedback, including item-level evaluation, diagnostic accuracy, and an overall performance score Workflow of the AI-powered simulated patient chatbot (Cortex). (1) Learners select a simulated patient profile, each representing a predefined headache case. (2) The chatbot conducts an interactive history-taking dialogue through text or voice input. (3) After the session, the system provides automated rubric-based feedback, including item-level evaluation, diagnostic accuracy, and an overall performance score Study procedure The study comprised three phases: baseline preparation and assessment, AISP intervention, and post-intervention assessment (Fig. 2). Fig. 2Study protocol diagram. The study followed a single-group pre-post design over four weeks. In week 1, students completed self-directed study using written materials on headache history-taking and diagnosis, followed by a baseline eight-minute Objective Structured Clinical Examination (OSCE) with standardized patients. After a one-week washout period, students entered the AISP intervention phase and completed one week of headache history-taking practice using the Cortex chatbot while continuing self-directed study. A post-intervention OSCE, matched in structure and scoring to the baseline assessment, was then administered. Afterward, students completed a usability questionnaire and participated in focus group discussions. Black boxes indicate OSCE assessment time points. Study protocol diagram. The study followed a single-group pre-post design over four weeks. In week 1, students completed self-directed study using written materials on headache history-taking and diagnosis, followed by a baseline eight-minute Objective Structured Clinical Examination (OSCE) with standardized patients. After a one-week washout period, students entered the AISP intervention phase and completed one week of headache history-taking practice using the Cortex chatbot while continuing self-directed study. A post-intervention OSCE, matched in structure and scoring to the baseline assessment, was then administered. Afterward, students completed a usability questionnaire and participated in focus group discussions. Black boxes indicate OSCE assessment time points. During the baseline preparation and assessment phase, students received written study materials on headache history-taking and diagnosis. Following a one-week self-directed study period using these materials, students completed a baseline eight-minute OSCE with standardized patients. Each OSCE station was assessed by a single neurologist examiner using a standardized rubric-based checklist and the Clinical Reasoning Indicator-History Taking (CRI-HT) scale [34]. The three neurologist examiners (PA, TP, and TS) participated in a one-hour orientation program prior to the examination to familiarize themselves with the OSCE format, competency assessment procedures, and scoring guidelines. In addition, each standardized patient was trained and validated in history delivery for at least one hour prior to the examination. Assessors were not blinded to study phase but followed the standardized rubric to minimize potential scoring bias. All OSCE sessions were video- or audio-recorded for quality assurance purposes. After a one-week washout period, students entered the AISP intervention phase and were given access to the Cortex platform. They were encouraged to complete at least nine chatbot-based history-taking practice sessions over one week while continuing to review the same study materials. System logs were used to describe chatbot usage during the intervention period. In the post-intervention assessment phase, students completed a second OSCE matched in structure and scoring to the baseline assessment, enabling within-participant comparison of performance. At the end of the course, students completed a usability questionnaire and participated in focus group discussions (FGDs) facilitated by a researcher who was not involved in student assessment. Data collection Three validated instruments were used. History-taking performance was evaluated using a rubric-based OSCE checklist that assessed completeness and structure. The OSCE scripts and scoring checklists were developed by a headache specialist (PA), and two neurologists (TP and TS) reviewed the content, provided feedback on the appropriateness of the scoring criteria, and contributed to refining the assessment. Diagnostic reasoning during history-taking was measured using the CRI-HT scale, a validated tool that evaluates the relevance and reasoning behind history-taking questions [34]. Unlike completeness-focused checklists, the CRI-HT assesses how well students prioritize and sequence questions to support diagnostic thinking. Usability and user satisfaction were assessed using the Chatbot Usability Questionnaire (CUQ), a validated 16-item instrument designed for conversational user interfaces [35]. The CUQ includes eight positively and eight negatively worded items and yields a score on a 100-point scale, allowing benchmarking against the System Usability Scale. The questionnaire was administered digitally in its original English-language form. Sample size calculation This study was powered for a single-group pre-post comparison using a paired t-test (two-tailed; α = 0.05; power = 80%). Prior meta-analytic evidence on educational chatbots reported an overall medium-to-large positive effects on educational outcomes, with a pooled effect size of 0.789 across 32 studies [36]. To be conservative, we assumed a standardized mean change of d = 0.67. The required sample size was 20 completers. Allowing for 20% attrition, the target enrollment was increased to 25 participants, providing at least 80% power to detect the prespecified effect. Statistical analysis Quantitative data analysis Quantitative variables are presented as mean (SD) or median (IQR), as appropriate, and categorical variables as number (%). Normality was assessed using the Shapiro-Wilk test and visual inspection of distributions. No extreme outliers were identified, and no observations were excluded. As all participants completed all assessments, there were no missing data and no imputation was required. Pre- and post-intervention outcomes were compared using paired t-tests. Effect sizes were calculated as Cohen’s d for paired samples. Pearson’s correlation coefficient was used to examine the association between OSCE and CRI-HT scores. To explore differential effects according to baseline performance, participants were stratified a priori by pre-test OSCE score using a 60% cut-off, corresponding to the minimum passing score. Mean gains (post-test minus pre-test) were compared between subgroups using independent-samples t-tests for both OSCE and CRI-HT outcomes. No adjustment was made for multiple comparisons, and subgroup analyses were considered exploratory. Statistical significance was defined as a two-sided p value < 0.05. All analyses were performed using IBM SPSS Statistics for Mac, version 28. Qualitative data analysis Qualitative data was analyzed using a descriptive thematic approach to explore students’ perceptions and experiences [37]. Data were collected using a semi-structured interview guide developed by the research team (Supplementary Material 2). FGDs were conducted in a private room at the Chulalongkorn Healthcare Advanced Multi-Profession Simulation Center (CHAMPS). Each session included four to six students and continued until thematic saturation was reached. Audio recordings were transcribed verbatim. Two researchers independently coded the transcripts using an initial codebook derived from the interview guide, supplemented by inductive coding to identify emergent themes. Any discrepancies in coding were resolved through discussion among the research team. Thematic analysis was used to develop themes related to learning experiences, system usability, and recommendations for implementation. Coding and theme development were conducted manually to support close engagement with the data. Trustworthiness To enhance credibility and dependability, two researchers independently coded the transcripts and met regularly to discuss and resolve discrepancies. Member checking was conducted with a subset of participants to confirm the accuracy of the interpretations and the thematic structure. The qualitative researchers had no prior relationship with the participants and no role in student assessment, thereby reducing the potential for power imbalance and social desirability bias. Results Participant characteristics A total of 25 third-year medical students participated and completed all study procedures. The majority of participants were male (23/25, 92%), and the mean age was 20.7 years (SD 0.5). Baseline characteristics, baseline performance, and chatbot usage metrics are summarized in Table 1. Table 1Participant characteristics, baseline performance, and chatbot useCharacteristicOverall (n = 25)Age, years, mean (SD)20.7 (0.5)Male sex, n (%)23 (92)Third-year medical students, n (%)25 (100)OSCE pre-test score (0–100), mean (SD)67.9 (10.8)CRI-HT pre-test score (0–40), mean (SD)28.2 (5.2)Low baseline OSCE performance (≤ 60), n (%)5 (20)High baseline OSCE performance (> 60), n (%)20 (80)Chatbot sessions, median (IQR)11 (9–14)Chatbot sessions, range3–20Total chatbot use time, min, median (IQR)97.2 (72.8-123.1)Total chatbot use time, min, range34.6-326.4Total chatbot use time, h, cohort total44.1Abbreviations: CRI-HT Clinical Reasoning Indicator–History Taking, IQR interquartile range, OSCE Objective Structured Clinical Examination, SD standard deviation Table 1Participant characteristics, baseline performance, and chatbot useCharacteristicOverall (n = 25)Age, years, mean (SD)20.7 (0.5)Male sex, n (%)23 (92)Third-year medical students, n (%)25 (100)OSCE pre-test score (0–100), mean (SD)67.9 (10.8)CRI-HT pre-test score (0–40), mean (SD)28.2 (5.2)Low baseline OSCE performance (≤ 60), n (%)5 (20)High baseline OSCE performance (> 60), n (%)20 (80)Chatbot sessions, median (IQR)11 (9–14)Chatbot sessions, range3–20Total chatbot use time, min, median (IQR)97.2 (72.8-123.1)Total chatbot use time, min, range34.6-326.4Total chatbot use time, h, cohort total44.1Abbreviations: CRI-HT Clinical Reasoning Indicator–History Taking, IQR interquartile range, OSCE Objective Structured Clinical Examination, SD standard deviation Participant characteristics, baseline performance, and chatbot use Abbreviations: CRI-HT Clinical Reasoning Indicator–History Taking, IQR interquartile range, OSCE Objective Structured Clinical Examination, SD standard deviation For prespecified subgroup analyses, participants were stratified by baseline OSCE performance using a 60% cut-off, corresponding to the minimum passing score. The low-performer subgroup (n = 5, 20%) had pre-test OSCE scores ≤ 60%, whereas the high-performer subgroup (n = 20; 80%) had scores > 60%. Quantitative findings Pre- and post-intervention performance Participants demonstrated improvements in both clinical performance and clinical reasoning scores following the intervention (Table 2; Fig. 3A,B). Mean OSCE score increased from 67.9 (SD 10.8) at pre-test to 90.4 (SD 6.6) at post-test (mean difference 22.5; 95% CI 17.7 to 27.3; p < 0.001; Cohen’s d = 1.93). Mean CRI-HT score also increased from 28.2 (SD 5.2) at pre-test to 37.0 (SD 2.2) at post-test (mean difference 8.8; 95% CI 6.8 to 10.8; p < 0.001; Cohen’s d = 1.81). Table 2Pre-test and post-test performance outcomes and subgroup analyses by baseline OSCE scoreOutcome/groupPre-test, mean (SD)Post-test, mean (SD)Mean difference95% CIP valueEffect sizePrimary outcomes (overall, n = 25)OSCE total (0–100)67.9 (10.8)90.4 (6.6)22.517.7 to 27.3< 0.0011.93CRI-HT total (0–40)28.2 (5.2)37.0 (2.2)8.86.8 to 10.8< 0.0011.81Secondary subgroup analyses by baseline OSCE scoreOSCE total (0–100)Low performer (≤ 60), n = 553.4 (6.3)90.2 (4.7)36.827.6 to 46.0< 0.0014.98High performer (> 60), n = 2071.5 (8.4)90.5 (7.1)19.014.4 to 23.5< 0.0011.96Low − High gain difference––17.98.2 to 27.5< 0.0011.92CRI-HT total (0–40)Low performer (≤ 60), n = 523.6 (3.8)36.0 (2.9)12.48.2 to 16.6< 0.0013.69High performer (> 60), n = 2029.4 (5.0)37.2 (2.0)7.95.6 to 10.1< 0.0011.64Low − High gain difference––4.6–0.2 to 9.30.0581.00Abbreviations: CI confidence interval, CRI-HT Clinical Reasoning Indicator–History Taking, OSCE Objective Structured Clinical Examination, SD standard deviationMean difference represents post-test minus pre-test. “Low − High gain difference” indicates the between-group difference in gain scores Table 2Pre-test and post-test performance outcomes and subgroup analyses by baseline OSCE scoreOutcome/groupPre-test, mean (SD)Post-test, mean (SD)Mean difference95% CIP valueEffect sizePrimary outcomes (overall, n = 25)OSCE total (0–100)67.9 (10.8)90.4 (6.6)22.517.7 to 27.3< 0.0011.93CRI-HT total (0–40)28.2 (5.2)37.0 (2.2)8.86.8 to 10.8< 0.0011.81Secondary subgroup analyses by baseline OSCE scoreOSCE total (0–100)Low performer (≤ 60), n = 553.4 (6.3)90.2 (4.7)36.827.6 to 46.0< 0.0014.98High performer (> 60), n = 2071.5 (8.4)90.5 (7.1)19.014.4 to 23.5< 0.0011.96Low − High gain difference––17.98.2 to 27.5< 0.0011.92CRI-HT total (0–40)Low performer (≤ 60), n = 523.6 (3.8)36.0 (2.9)12.48.2 to 16.6< 0.0013.69High performer (> 60), n = 2029.4 (5.0)37.2 (2.0)7.95.6 to 10.1< 0.0011.64Low − High gain difference––4.6–0.2 to 9.30.0581.00Abbreviations: CI confidence interval, CRI-HT Clinical Reasoning Indicator–History Taking, OSCE Objective Structured Clinical Examination, SD standard deviationMean difference represents post-test minus pre-test. “Low − High gain difference” indicates the between-group difference in gain scores Pre-test and post-test performance outcomes and subgroup analyses by baseline OSCE score Abbreviations: CI confidence interval, CRI-HT Clinical Reasoning Indicator–History Taking, OSCE Objective Structured Clinical Examination, SD standard deviation Mean difference represents post-test minus pre-test. “Low − High gain difference” indicates the between-group difference in gain scores Pre-test and post-test performance outcomes overall and by baseline subgroup. Panels A and B show paired pre-test and post-test scores for OSCE (0–100) and CRI-HT (0–40), respectively. Points represent individual participants, and lines connect paired observations; mean values with 95% CIs are overlaid. Panels C and D show change scores (post-test minus pre-test) according to baseline OSCE subgroup (low performers: pre-test OSCE ≤ 60; high performers: pre-test OSCE > 60). Box plots show the median and interquartile range, with individual data points overlaid. Mean differences with 95% CIs and P values are annotated in each panel. Abbreviations: CI, confidence interval; CRI-HT, Clinical Reasoning Indicator-History Taking; IQR, interquartile range; OSCE, Objective Structured Clinical Examination Subgroup analysis by baseline performance In prespecified secondary analyses stratified by baseline OSCE score using a 60% cut-off, both subgroups showed improvement in OSCE and CRI-HT scores (Table 2; Fig. 3C,D). For OSCE, the low-performer subgroup (n = 5) improved from 53.4 (SD 6.3) at pre-test to 90.2 (SD 4.7) at post-test (mean difference 36.8; 95% CI 27.6 to 46.0; p < 0.001; Cohen’s d = 4.98), whereas the high-performer subgroup (n = 20) improved from 71.5 (SD 8.4) to 90.5 (SD 7.1) (mean difference 19.0; 95% CI 14.4 to 23.5; p < 0.001; Cohen’s d = 1.96). The mean gain was greater in the low-performer subgroup, with a between-group difference of 17.9 points (95% CI 8.2 to 27.5; p < 0.001; Cohen’s d = 1.92). For CRI-HT, the low-performer subgroup (n = 5) improved from 23.6 (SD 3.8) at pre-test to 36.0 (SD 2.9) at post-test (mean difference 12.4; 95% CI 8.2 to 16.6; p < 0.001; Cohen’s d = 3.69), whereas the high-performer subgroup (n = 20) improved from 29.4 (SD 5.0) to 37.2 (SD 2.0) (mean difference 7.9; 95% CI 5.6 to 10.1; p < 0.001; Cohen’s d = 1.64). Although the low-performer subgroup showed a larger mean gain, the between-group difference in CRI-HT gain was not statistically significant (difference in gain 4.6; 95% CI -0.2 to 9.3; p = 0.058; Cohen’s d = 1.00) (Table 2; Fig. 3D). Despite differences in baseline performance, post-test scores were similar between subgroups (OSCE: 90.2 [SD 4.7] vs 90.5 [SD 7.1], p = 0.94; CRI-HT: 36.0 [SD 2.9] vs 37.2 [SD 2.0], p = 0.28), suggesting that the performance gap narrowed following the intervention. Chatbot usability Usability outcomes are summarized in Table 3. The mean CUQ score was 75.8 (SD 10.9), indicating generally favorable perceived usability and an overall positive user experience. Most participants agreed or strongly agreed with positive usability statements, including feeling welcome during initial setup (21/25, 84%), ease of navigation (23/25, 92%), the chatbot’s ability to understand their input (21/25, 84%), the usefulness and appropriateness of its responses (21/25, 84%), and overall ease of use (24/25, 96%). The most frequently endorsed usability concern was that the chatbot felt too robotic (12/25, 48%). All other negative usability concerns were endorsed by fewer than 16% of participants. Table 3Chatbot Usability Questionnaire (CUQ) scores and selected item endorsementsMeasureOverall (n = 25)CUQ total score (0–100), mean (SD)75.8 (10.9)Selected CUQ item endorsements, n (%)Felt welcome during initial setup21 (84)Easy to navigate23 (92)Understood me well21 (84)Responses were useful, appropriate, and informative21 (84)Very easy to use24 (96)Seemed too robotic12 (48)Abbreviations: CUQ Chatbot Usability Questionnaire, SD standard deviationItem endorsement indicates a response of agree or strongly agree (Likert score 4 or 5). For the negatively worded item “Seemed too robotic,” endorsement indicates a usability concern Table 3Chatbot Usability Questionnaire (CUQ) scores and selected item endorsementsMeasureOverall (n = 25)CUQ total score (0–100), mean (SD)75.8 (10.9)Selected CUQ item endorsements, n (%)Felt welcome during initial setup21 (84)Easy to navigate23 (92)Understood me well21 (84)Responses were useful, appropriate, and informative21 (84)Very easy to use24 (96)Seemed too robotic12 (48)Abbreviations: CUQ Chatbot Usability Questionnaire, SD standard deviationItem endorsement indicates a response of agree or strongly agree (Likert score 4 or 5). For the negatively worded item “Seemed too robotic,” endorsement indicates a usability concern Chatbot Usability Questionnaire (CUQ) scores and selected item endorsements Abbreviations: CUQ Chatbot Usability Questionnaire, SD standard deviation Item endorsement indicates a response of agree or strongly agree (Likert score 4 or 5). For the negatively worded item “Seemed too robotic,” endorsement indicates a usability concern Qualitative findings Analysis of the focus group discussions identified three themes: (1) structured practice and feedback supported skill development; (2) the simulation was sufficiently realistic for cognitive rehearsal but limited by technical and interpersonal constraints; and (3) students recommended broader case variation and curriculum integration. Theme 1: Structured practice and feedback supported skill development Students described the platform as useful for practicing structured history-taking and organizing questions in a logical sequence. A commonly reported benefit was the ability to practice independently at convenient times, which reduced reliance on peers for role-play. As one student noted, “I could practice from anywhere, and more importantly, anytime, without bothering anyone.” Another added, “I even practiced at 1 a.m., no need to schedule with friends.” A third echoed, “It felt like I always had a patient available to practice with, no need to find a practice partner.” Students also valued the structured feedback provided after each session. They reported that the detailed evaluations helped them identify gaps in content and reasoning and focus on specific areas for improvement. One student commented, “I liked how the feedback clearly pointed out what I missed; it helped me improve immediately.” Another noted, “There are things I often forget, like asking about neurological deficits when taking a headache history.” A third commented, “The feedback helped me understand how to organize my questions better to make the flow smoother.” In addition, repeated practice was frequently linked to increased confidence. Many students reported feeling better prepared for OSCEs after several sessions with the chatbot. One student stated, “After practicing two to three times, I felt much more confident, completely ready for the exam.” Another shared, “My confidence increased 100%. At first, I didn’t even know how to start.” One student also linked repeated practice to perceived performance improvement: “I scored 54 on the first round and 81 on the second, and I felt more confident.” Theme 2: Realism and limitations of the system Students generally perceived the simulation as realistic, particularly in conversational flow and consistency. Several appreciated that the chatbot did not provide unsolicited information, which encouraged them to ask questions systematically in a manner similar to real clinical encounters. As one student noted, “The AI felt like a real patient. It didn’t give hints, just like a real encounter.” Another commented, “The AI voice was pleasant and sounded like a real patient.” A third added, “Sometimes when I said something, the AI interpreted it differently, but in the end, the answer was still on point.” At the same time, students also identified important limitations. The most commonly reported concern involved speech recognition errors, particularly when pauses or word spacing led to misinterpretation. Some students reported asking appropriate questions but not receiving credit, only recognizing the issue after reviewing the feedback. As students reported, “Some words were misinterpreted; for example, when I paused, it cut off the sentence and led to a misunderstanding.” Another said, “Sometimes the system mispronounces Thai words.” Students also pointed to limitations in areas of communication that depend on non-verbal cues and emotional responsiveness. Although the chatbot was viewed as useful for cognitive rehearsal, they felt that it did not reproduce important aspects of real consultations. One student stated, “It still can’t replace real patients because there’s no eye contact or genuine tone of voice.” Another said, “In real exams, we feel more nervous. The system doesn’t train us to handle that kind of pressure.” Other students similarly noted, “It seems to help with knowledge, but it doesn’t yet develop the soft skills needed for real patient interactions” and “It can’t replace real exams or learning directly, but it’s great for skill practice.” Theme 3: Suggestions and future implementation Students expressed strong interest in the continued use of the AISP and offered several suggestions for improvement. One common request was greater case diversity and randomization. Participants noted that repeated exposure to the same scenarios reduced challenge over time and suggested a “random case mode” in which contextual details such as name, age, and background would vary. As one student stated, “I’d like the system to randomize cases without telling us in advance; it would make things more challenging.” Another added, “Using the same disease but changing the name, age, or context would make the practice more engaging.” A third commented, “Repeating the same case becomes too familiar. I’d really like to see true randomization.” Students also supported integrating the AISP into the formal curriculum, particularly for OSCE preparation and the introductory pre-clerkship module that prepares students for clinical rotations. They felt it could supplement lectures and workshops by enabling individualized practice outside class time. As students explained, “It’s perfect for OSCE exam practice; it feels like talking with a real patient,” and “I’d like it to be used alongside lectures: learn from the instructor first, then practice with the AI.” Another noted, “There’s not enough time in workshops, but I can practice multiple times with the chatbot.” Finally, students described the platform as motivating. Some were motivated by score improvement, whereas others used the system to build confidence in clinical conversations. Several indicated they would continue using the platform if additional cases were available or when they felt less confident about a condition. As students reported, “I used it because I wanted a perfect score,” “My motivation was to improve my skills and feel more confident talking to real patients,” and “I’d keep using it if there were new cases or if I still didn’t feel confident with a condition.” Discussion This convergent mixed-methods study found that training with the Cortex AISP chatbot was associated with improved performance on both OSCE checklist scores and CRI-HT clinical reasoning scores, alongside high usability ratings among third-year medical students. The combination of improved completeness and improved reasoning suggests that the intervention supported not only broader coverage of key history-taking elements but also the prioritization and sequencing of questions relevant to headache diagnosis. These quantitative findings were complemented by qualitative data indicating that students valued the structured, repeatable practice environment and the immediacy of automated feedback. Deliberate practice as a framework for understanding performance gains The observed improvements can be interpreted through the lens of deliberate practice, which emphasizes structured, goal-directed repetition with feedback as a central mechanism for skill acquisition in complex performance domains [32, 38]. Cortex incorporated several core features of deliberate practice: repeated opportunities for focused history-taking, clearly defined task goals, immediate rubric-based feedback, and the opportunity to refine performance across multiple attempts. This repeated cycle of practice, feedback, and revision is consistent with the deliberate-practice framework and may help explain the magnitude of improvement observed in this study. Importantly, the median number of chatbot sessions was 11 (IQR 9–14), suggesting that many students engaged in repeated practice throughout the intervention period, consistent with the self-directed engagement emphasized in deliberate practice. Comparison with existing evidence on LLM-based simulated patients Our findings are broadly consistent with emerging evidence that LLM-based and other virtual patient systems can support history-taking, clinical decision-making, and learner confidence in medical education [22, 23, 26, 39]. More broadly, prior literature suggests that virtual patient approaches can support knowledge and clinical reasoning across diverse medical educational settings, providing a foundation for newer AI-driven approaches [15, 16, 39]. Our study extends this literature to a domain-specific application—headache history-taking—where the structured, repetitive nature of diagnostic questioning may be particularly well suited to AISP-based training. The effect sizes observed in our study were large; however, they should be interpreted cautiously because the single-arm pre-post design cannot exclude alternative explanations such as concurrent learning, test familiarity, or maturation. Lower-performing students showed disproportionately larger gains The prespecified subgroup analyses revealed that students with lower baseline OSCE scores demonstrated larger gains, reaching post-test performance similar to that of their higher-performing peers. This pattern is consistent with a catch-up effect and may reflect principles of mastery learning, in which learners continue to practice until they reach a defined standard of competence [40]. The AISP platform supported this type of self-paced, criterion-referenced practice: students with greater room for improvement could repeat cases and receive targeted feedback across multiple attempts. Although the between-group difference in CRI-HT gain did not reach statistical significance (p = 0.058), the direction and magnitude of the effect were consistent with the OSCE findings. These results suggest that AISP-based training may be particularly beneficial for learners who require additional structured practice, although the small subgroup size (n = 5 in the low-performer group) warrants cautious interpretation. AISPs support cognitive rehearsal but do not replace human clinical interactions Usability findings support the feasibility of implementing AISP-based training in medical education. CUQ scores indicated favorable perceived usability, and students highlighted ease of navigation and the usefulness of performance feedback. At the same time, nearly half of participants described the chatbot as occasionally robotic, and some reported speech recognition errors in Thai-language interactions. These concerns identify important targets for technical refinement, particularly in speech-to-text accuracy and conversational prosody in non-English language contexts. Students also noted that the platform was less suitable for practicing interpersonal skills that rely on non-verbal cues, emotional responsiveness, and the affective dimensions of the doctor-patient relationship. Taken together, these findings suggest that AISPs are best suited to structured, repeatable practice of history-taking and clinical reasoning, rather than to the full range of interpersonal and relational skills required in real patient encounters [12, 41]. Implications for curricular integration in headache education From an educational perspective, AISPs may help increase access to structured practice when faculty time, patient availability, or standardized patient capacity is limited [13]. This is particularly relevant for clinical domains such as headache medicine, where dedicated teaching time is often limited and clinical exposure during training may be inconsistent [3, 7]. Participants in our study recommended expanding case variety and integrating the platform into early clinical training and OSCE preparation. These recommendations align with recent literature suggesting that AI-supported tools can be integrated into medical education in ways that complement, rather than replace, traditional teaching approaches [39, 42, 43]. Limitations and future directions This study has several limitations that should be considered when interpreting the findings. First, the single-group pre-post design cannot exclude alternative explanations for the observed improvements, including test familiarity, concurrent self-study, maturation, or increased motivation during the elective module. Without a control group, the specific contribution of the AISP intervention cannot be isolated from these potential confounders. Second, the study was conducted at a single institution with a small sample of 25 students, and the low-performer subgroup included only five participants, limiting statistical power and the generalizability of the subgroup analyses. Third, each OSCE station was assessed by a single neurologist examiner. Although all examiners underwent a standardized orientation and used a rubric-based checklist, formal inter-rater reliability testing was not performed, and scoring variability between examiners cannot be fully excluded. Fourth, OSCE assessors were not blinded to study phase, which may have introduced bias despite the use of standardized rubrics. Fifth, outcomes were measured shortly after the intervention, so longer-term retention and transfer of skills to real clinical encounters remain unknown. Sixth, only three simulated headache scenarios were included, which may limit the generalizability of the findings across the wider spectrum of headache presentations. Finally, the study was conducted with Thai medical students in a specific curricular context, and the findings may not be directly transferable to other settings or learner populations. Future studies should employ randomized controlled designs with active comparison groups to better estimate the independent effect of AISP training. Multi-institutional studies with larger and more diverse samples would strengthen external validity. The case library should be expanded to include a broader range of headache presentations, secondary headache red-flag scenarios, and other clinical domains to reduce familiarity effects and improve generalizability. Longitudinal follow-up is also needed to assess skill retention over time and transfer to real clinical settings. In addition, implementation research should examine feasibility at scale, including faculty workload, cost-effectiveness, and how AISP-based training can be integrated into existing curricula and assessment frameworks. Conclusion In this study, the AI-powered simulated patient platform Cortex, built on a large language model with automated rubric-based feedback, was feasible and associated with meaningful improvements in headache history-taking and clinical reasoning among medical students. Consistent with the principles of deliberate practice, the platform enabled structured, self-paced, and repeatable practice with immediate performance feedback. Students with lower baseline performance showed larger gains, suggesting a potential catch-up effect that may support more equitable learning outcomes. Although emotional authenticity and interpersonal skill training remain limitations of current AISP technology, refinement of conversational design, expansion of case variety, and thoughtful integration into medical curricula may strengthen the role of AISPs as scalable adjuncts for clinical skills training. Controlled studies are needed to confirm these findings and evaluate the long-term educational impact of AISP-based training. Supplementary Information Supplementary Material 1: Detailed procedure for development of the AISP chatbot including the prompts. Supplementary Material 1: Detailed procedure for development of the AISP chatbot including the prompts. Supplementary Material 2: Semi-structured interview guide used for qualitative data collection. Supplementary Material 2: Semi-structured interview guide used for qualitative data collection.