Integrating large language models into prostate cancer training: evidence from comparative benchmarking and a pilot randomized trial Authors: Xuhao Liu, Minfeng Chen, Yi Cai Published: 2026 ABSTRACT BackgroundLarge language models (LLMs) are increasingly explored as tools for medical education. However, evidence remains limited regarding their pedagogical quality and real-world utility in prostate cancer teaching within urology residency training.MethodsWe conducted a two-phase study. Phase 1 benchmarked three LLMs (ChatGPT-4o, DeepSeek R1, and Gemini 2.0) on a structured 40-item prostate cancer education question bank using standardized prompts and blinded expert ratings. Phase 2 implemented the top-performing model in a pilot randomized teaching trial among urology residents (n = 34) using stratified block randomization based on a pre-admission standardized test score, with allocation concealment implemented through a centralized web-based system. Both groups received identical offline instruction with a time-matched lecture structure. The control group committed to avoiding LLM use for course-related questions during the teaching period.ResultsDeepSeek R1 ranked highest in expert ratings, with clearer advantages on higher-order and innovation-oriented questions. In the pilot randomized teaching trial (n = 34), the AI-assisted group achieved higher closed-book examination scores than controls (68.47 ± 12.78 vs. 57.91 ± 10.47; MD 10.56, 95% CI 2.39–18.73; p = 0.013). Improvements were most evident in Multiple-Choice Questions(MCQs) (MD 9.27, 95% CI 4.37–14.17; p < 0.001) and research items (MD 3.32, 95% CI 1.72–4.92; p < 0.001), whereas Multidisciplinary Team(MDT) case analysis showed no clear difference. Student and instructor feedback was generally positive.ConclusionLLM-assisted teaching was associated with higher knowledge-based examination performance, especially for MCQ-style and innovation-focused content, while effects on MDT reasoning remain uncertain. These preliminary findings suggest that carefully guided LLM use may support residency teaching, but larger multicenter studies and structured verification workflows are needed to confirm effectiveness and generalizability. Introduction Prostate cancer is among the most prevalent malignancies encountered in urology residency training, and its teaching requires residents to integrate rapidly evolving guideline recommendations with multidisciplinary decision-making and patient-centered communication [1]. Effective instruction must span foundational physiology and pathology, multimodal imaging, therapeutic strategies, and MDT-oriented case reasoning. In practice, however, training cohorts are heterogeneous in baseline proficiency, curricular demands are extensive, and classroom time is constrained. As a result, many clinically relevant, in-depth questions cannot be resolved in real time, which may hinder trainees—particularly postgraduate learners—from developing the comprehensive competencies expected in standardized training [2, 3]. Large language models (LLMs) have been increasingly adopted across professional domains and are now being explored as interactive tools in medical education [4, 5]. Models such as ChatGPT, DeepSeek, Gemini, and Grok can generate structured explanations, retrieve and synthesize information, and support simulated dialogue, suggesting potential value as on-demand “tutors” for resident learning [6]. However, medical education requires more than producing correct answers: it also involves promoting clinical reasoning, adapting explanations to learner level, and reinforcing humanistic and ethical dimensions of care [7]. Current evidence remains insufficient on whether LLMs reliably deliver these pedagogical qualities, and many evaluations emphasize factual correctness while under-assessing higher-order cognitive support, humanistic care, and educational safety concerns such as overconfidence or misleading guidance [8]. Moreover, head-to-head comparisons using standardized educational blueprints and blinded expert assessment are still limited, and evidence translating model performance into measurable learning outcomes in controlled teaching designs is particularly scarce in prostate cancer education. To address these gaps, we used a two-phase “benchmark-to-classroom” design: first benchmarking three LLMs for prostate cancer education with blinded expert assessment, then testing the best model in a pilot randomized teaching trial among urology residents using examination performance and post-course surveys as outcomes.Together, these findings aim to inform educators not only which LLM characteristics matter for instructional quality, but also how LLM assistance may translate into learning outcomes in specialty residency training. Methods Study design, setting, and reporting standards This study comprised two sequential components: (i) a head-to-head benchmarking evaluation of LLMs for prostate cancer medical education using a structured subjective-question bank, and (ii) a pilot randomized teaching trial applying the best-performing LLM to routine prostate cancer teaching and evaluating educational outcomes. The teaching component was designed and reported with reference to CONSORT principles for randomized educational trials, with participant flow documented using a CONSORT-style diagram(Fig. 1). Fig. 1Flowchart of the study procedure. Flowchart illustrating recruitment of urology residents, application of inclusion or exclusion criteria, stratified block randomization to AI-assisted versus control instruction, completion of the intervention, and inclusion in the final examination and questionnaire analyses Flowchart of the study procedure. Flowchart illustrating recruitment of urology residents, application of inclusion or exclusion criteria, stratified block randomization to AI-assisted versus control instruction, completion of the intervention, and inclusion in the final examination and questionnaire analyses Development of the question bank and prompt template Blueprint and content domains A 40-item subjective question bank was developed to reflect authentic prostate cancer teaching scenarios. Questions were organized across five educational domains—basic knowledge, clinical application, humanistic care, practical skills, and innovative developments—covering urology, anatomy, physiology and pathophysiology, pathology, medical imaging, oncology, and medical humanities relevant to prostate cancer education. To capture different levels of cognition, items were mapped to three cognitive levels (Table 1). All items were designed to be open-ended to elicit explanatory, teaching-oriented answers rather than single-fact retrieval. Table 1Detailed Information for 40 questions. The detailed question stems for the 40 questions, along with their respective cognitive levels and specialized, are comprehensively listed Table 1Detailed Information for 40 questions. The detailed question stems for the 40 questions, along with their respective cognitive levels and specialized, are comprehensively listed Detailed Information for 40 questions. The detailed question stems for the 40 questions, along with their respective cognitive levels and specialized, are comprehensively listed Prompt structure and response requirements To minimize ambiguity and enhance instructional alignment, each LLM prompt was standardized into four components: (1) an introductory role-setting section placing the model as a clinical mentor for prostate cancer education; (2) explicit answer requirements; (3) logical sequencing requirements to encourage structured reasoning; and (4) the question stem (Fig. 2). Responses were required to align with contemporary authoritative guidelines, and to present content in a form suitable for clinical teaching. Fig. 2Process chain and sample prompts for interacting with the LLM. The interaction process is divided into three parts: input, reasoning with evidence, and output. The figure illustrates the complete reasoning process and response of the LLM to a clinical question Process chain and sample prompts for interacting with the LLM. The interaction process is divided into three parts: input, reasoning with evidence, and output. The figure illustrates the complete reasoning process and response of the LLM to a clinical question LLM selection and response acquisition Three widely used, high-profile LLMs were evaluated: OpenAI ChatGPT-4o, DeepSeek R1, and Google Gemini 2.0, selected as representative systems available during the study period. All 40 questions were submitted to each LLM between February 20 and 28, 2025, using identical prompt text across models. Questions were administered sequentially from Q1 to Q40, with item difficulty increasing progressively according to the preset blueprint. All LLMs outputs were captured verbatim in a structured dataset, preserving original formatting without post-editing (Fig. 2). All models were accessed at their highest publicly available versions during the study period via their official user interfaces. No external tools, browsing features, or plug-ins were enabled. For each item, the standardized prompt was entered verbatim, and the full output was recorded without alteration. To minimize carryover effects, each question was administered in an independent session with the conversation context cleared before the next item. Expert evaluation of LLM responses Rating rubric and scoring rules To evaluate whether LLM-generated responses were suitable for prostate cancer teaching, we developed a five-dimension rubric covering correctness, inspiration, adaptability, conciseness, and humanistic care. These dimensions were selected to reflect both the quality of medical information and its educational value in clinical teaching. The construction of the rubric was informed by established principles of health information quality assessment, cognitive learning objectives, and clinical communication education: correctness and conciseness were guided by criteria for reliable, evidence-based, and understandable health information; inspiration and adaptability were aligned with cognitive learning objectives from factual recall to application; and humanistic care was informed by patient-centered clinical communication frameworks [9–11]. Responses were scored on a 0–5 Likert scale for each dimension, with higher scores indicating better performance. Correctness assessed factual accuracy, clinical completeness, internal consistency, and whether recommendations were guideline-concordant and clinically appropriate. Inspiration referred to the extent to which a response moved beyond factual recall by connecting related concepts, stimulating critical thinking, supporting transfer to clinical reasoning, and encouraging further inquiry or self-directed learning. Adaptability evaluated whether the depth, structure, and explanation style of the response matched the cognitive level and educational purpose of the question. Conciseness assessed whether the response was clear, focused, and efficient for learner comprehension without unnecessary repetition. Humanistic care referred to both learner-supportive communication and patient-centered values, including encouragement, respectful tone, empathy, shared decision-making, attention to patient preferences, privacy, and professionalism where applicable. Rater training, blinding, and adjudication Three urology experts with substantial clinical teaching experience independently scored all responses. Before formal scoring, raters reviewed the rubric and conducted a brief calibration exercise on a small set of pilot responses (not included in the analysis) to align scoring interpretations. Raters were aware of the question content but blinded to the identity of the responding LLM. If the absolute discrepancy among the three raters was ≥ 2 points on any dimension for a given response, a fourth expert performed adjudication; the adjudicator’s score was taken as the final determination for that dimension. Pilot randomized teaching trial: teaching application and evaluation Participants enrollment The best-performing LLM identified in Expert evaluation of LLM responses was selected for teaching application. Participants were urology trainees at Xiangya Hospital, Central South University, recruited at the start of a prostate cancer teaching block. This teaching trial enrolled urology residents who met all of the following inclusion criteria: (1) registered trainees in a urology residency program; (2) full exposure to the teaching intervention, defined as attending all scheduled instructional modules delivered during the study period; (3) ability to complete follow-up assessments, operationalized as participation in the end-of-course examination with submission of a valid answer sheet and completion of one valid post-course survey questionnaire; (4) provision of written informed consent; and (5) access to required learning resources, defined as ownership of at least one device capable of accessing the institutional learning platform and the assigned LLM tools. Trainees were excluded if they: (1) were not enrolled in a urology training track; (2) had substantial non-attendance during the study period or did not attend the summative examination (including mid-course withdrawal or absence from the final assessment); or (3) had previously completed a prostate cancer curriculum prior to enrollment. Potentially eligible residents were approached at the start of the prostate cancer teaching block and provided with standardized study information. Eligibility was verified by the study coordinator using attendance records and training registration documents. Participants were informed that participation was voluntary and would not affect academic evaluation beyond routine course requirements. Attendance was monitored throughout the study, and examination participation was confirmed on the scheduled assessment date. Survey responses were considered valid if all mandatory items were completed; incomplete or duplicate submissions were excluded from the questionnaire analysis. A CONSORT flow diagram was used to report participant screening, exclusion reasons, allocation, follow-up, and inclusion in the outcome analyses(Fig. 1).This randomized teaching trial was designed as a single-center pilot study to evaluate feasibility and to provide preliminary effect estimates to inform future multicenter trials. Sample size was therefore cohort-constrained, and all eligible residents in the teaching block were invited to participate. Randomization and allocation concealment Eligible participants were randomly assigned in a 1:1 ratio to the LLM-assisted group or the control group using stratified block randomization based on a pre-admission standardized test score routinely administered prior to program entry. Participants were stratified into high-score and low-score strata using the cohort mean score as the cutoff. Within each stratum, an independent researcher (not involved in teaching, enrollment, or outcome assessment) generated a computer-based randomization sequence using randomly permuted blocks to maintain an approximately 1:1 allocation throughout enrollment. Allocation concealment was ensured via a centralized, web-based assignment system. After written informed consent and eligibility verification, the study coordinator entered each participant’s identification code and stratum into the online system, which then returned the group assignment. The randomization sequence was stored within the system and was not accessible to the coordinator, instructors, or outcome assessors, thereby preventing foreknowledge or prediction of upcoming allocations. Because of the nature of the educational intervention, participants and instructors could not be blinded to group assignment. However, blinding was implemented where feasible. In Phase 1, expert raters were blinded to the identity of the responding LLMs. In Phase 2, outcome assessment was conducted through a standardized computer-based grading workflow: MCQs were graded automatically, and case-analysis responses were anonymized before scoring so that graders could not access students’ names, personal information, or group allocation. Intervention and control procedures To ensure a strictly time-matched teaching dose, both groups received identical offline instruction with the same instructor team, teaching materials, and Q&A frequency. Each lecture lasted 45 min and followed an identical structure: (i) 25 min of instructor-led core teaching and (ii) a fixed 20-minute Structured Question Review and Discussion (SQRD) segment. The SQRD segment was delivered in both groups using the same facilitation approach, with instructors clarifying misconceptions and guiding structured clinical reasoning. In the LLM-assisted group, trainees used the selected LLM for structured pre-class preparation and post-class consolidation. Before each lecture, trainees submitted uncertainties about upcoming content to the LLM and generated structured summaries and draft answers. During SQRD, trainees presented their LLM-generated questions and outputs; instructors provided real-time feedback, corrected errors, and prompted deeper reasoning, with emphasis on guideline-concordant interpretation and verification of key claims. In the control group, trainees completed the same pre-class preparation using standard learning resources without LLM support. During SQRD, trainees presented questions and their own prepared responses derived from these standard resources; instructors used the same facilitation approach to address knowledge gaps and strengthen reasoning. To minimize contamination and maintain adherence to the control condition, control-group residents signed a written commitment before the intervention agreeing not to use any LLM tools for course-related questions during the teaching period. They were instructed to use only standard learning resources, including lecture materials, textbooks, and guideline documents, for course preparation and consolidation. This requirement was reinforced by instructors during the teaching block. MDT case discussions were conducted as part of the department’s routine clinical teaching workflow. After completion of the lecture modules, trainees from both groups participated together in the same MDT case discussions (60 min per session), and LLM use was not permitted during MDT discussions. This design was intended to mirror real-world practice and ensured that MDT exposure and instructor contact during MDT were identical across groups. Summative examination and questionnaires At course completion, all participants undertook a uniform closed-book summative examination (total score 0–100) comprising multiple-choice questions (MCQs) and case analyses. The summative examination questions were developed by the course instructors according to the predefined teaching syllabus and learning objectives of the prostate cancer teaching block. The questions were not generated by any LLM. Except for some basic conceptual items that necessarily reflected standard medical knowledge, the examination items were original and were developed specifically for this course assessment. Before the examination, the questions were not uploaded to any online platform and were not submitted as prompts to ChatGPT-4o, DeepSeek R1, Gemini 2.0, or any other LLM. MCQs were graded automatically through a standardized computer-based scoring process. Case-analysis responses were anonymized and graded independently by two instructors who could not access students’ names, personal information, or group allocation. The final case-analysis score was calculated as the mean of the two ratings. Immediately after the examination, trainees and instructors completed post-course questionnaires. General course feedback was collected from all trainees. Items specifically evaluating the experience of LLM-assisted learning were completed and analyzed only among trainees in the LLM-assisted group. Instructor questionnaires assessed perceived changes in trainee preparation, reasoning, and feasibility of integrating LLM tools into future teaching. The post-course student and instructor questionnaires were purpose-built for this teaching trial and developed following established best practices for survey design and educational questionnaire construction [12, 13]. Items were aligned to prespecified implementation-relevant constructs (perceived learning support, reasoning support, engagement, usability of prompting, and future adoption intention) and were informed by recent survey-based evaluations of LLM use in medical education [14, 15]. To support interpretability, we prioritized content- and response-process validity evidence through expert review for relevance, clarity, and redundancy, consistent with contemporary validity arguments used in healthy education questionnaire development [16, 17]. Survey reporting in the manuscript follows the CROSS recommendations for transparency in survey studies [18]. Outcomes The primary outcome was performance on a standardized closed-book final examination administered at course completion, reported as total score (range 0–100). Secondary outcomes included examination completion time, sub-scores by question difficulty (easy, medium or hard) and question type (e.g., clinical scenarios, research advances, MDT-related items), and post-course perceptions measured using structured student and instructor questionnaires. All outcomes and analysis plans were specified prior to outcome assessment. Statistical Analysis Analyses were performed using RStudio (2024.04.2–764) and IBM SPSS Statistics (v25.0). Continuous variables are reported as mean ± SD or median (IQR), and categorical variables as n (%). For Phase 1, the overall expert-rated instructional quality score served as the primary benchmarking metric for model selection. Dimension-, cognitive-level-, and domain-specific comparisons were treated as exploratory pattern analyses; given multiple comparisons, their p values were interpreted as nominal and descriptive. For Phase 2, the prespecified primary outcome was the total score on the 100-point closed-book summative examination. The primary analysis reported between-group mean difference (MD) with 95% confidence interval (CI) and a two-sided p value. Examination subscores, completion time, and questionnaire responses were analyzed as secondary outcomes, with difficulty-stratified, content-stratified, and item-level questionnaire analyses interpreted as exploratory. Secondary and exploratory findings were interpreted mainly by effect sizes, CIs, and consistency of patterns. A two-sided p < 0.05 was considered statistically significant for the primary outcome. Results Comparative evaluation of LLMs Across the 40-item subjective question bank, blinded expert ratings revealed a consistent performance gradient among the three models across the five evaluation dimensions (Fig. 3). Inter-rater reliability for the overall mean score was good, with an ICC [2, 3] of 0.856 (95% CI :0.765–0.908) based on the three blinded raters’ pre-adjudication scores. Fig. 3Comparative violin charts illustrating the performance of three LLMs across different levels of question difficulty and medical subspecialty domains. A and B display the models’ performance in terms of conciseness and insightfulness on memorize -level questions. C evaluates humanistic care in comprehension-level responses, while D highlights the models’ ability to provide insightful guidance in application-level scenarios. E focuses on the accuracy of responses related to basical concepts, and F presents a comparative summary of performance in research advances At the memorization level, differences were limited, with DeepSeek scoring higher than Gemini on inspiration (4.67 vs. 4.10, p = 0.003), while correctness and adaptability showed no substantial between-model differences (p = 0.581 and p = 0.182)(Fig. 3A). In parallel, conciseness differed across models, with ChatGPT exceeding Gemini (4.60 vs. 3.87, p < 0.001), and Gemini exceeding ChatGPT on humanistic care (4.27 vs. 3.70, p = 0.006)(Fig. 3B). At the comprehension level, model separation remained minimal, with only a small difference detected in humanistic care between ChatGPT and Gemini (4.17 vs. 4.21, p = 0.018)(Fig. 3C). At the application level, clearer differentiation emerged for reasoning-oriented outputs, where DeepSeek showed higher inspiration than ChatGPT (4.42 vs. 4.10, p = 0.005)(Fig. 3D). Comparative violin charts illustrating the performance of three LLMs across different levels of question difficulty and medical subspecialty domains. A and B display the models’ performance in terms of conciseness and insightfulness on memorize -level questions. C evaluates humanistic care in comprehension-level responses, while D highlights the models’ ability to provide insightful guidance in application-level scenarios. E focuses on the accuracy of responses related to basical concepts, and F presents a comparative summary of performance in research advances Across disciplinary objectives, DeepSeek demonstrated advantages on domains most relevant to higher-order learning. For basic concepts, DeepSeek exceeded Gemini in correctness (4.80 vs. 4.44, p = 0.006)(Fig. 3E). For research and new developments, DeepSeek received higher inspiration ratings than both comparators (4.67 vs. 3.83, p = 0.015 for each comparison)(Fig. 3F). Given that Phase 1 involved multiple dimensions and stratified contrasts, p values are reported as nominal and interpreted primarily as pattern evidence. Overall model selection for Phase 2 followed the prespecified criterion of highest expert-rated instructional suitability across the full benchmarking framework (i.e., across dimensions, cognitive levels, and disciplinary objectives), rather than any single panel or isolated comparison(Fig. 4). Fig. 4Bar chart of the comprehensive scores across different dimensions for the three LLMs. The average scores of the three different LLMs across various dimensions are displayed in the figure, where red bars represent ChatGPT 4o, green bars represent Gemini 2.0, and blue bars represent DeepSeek R1 Bar chart of the comprehensive scores across different dimensions for the three LLMs. The average scores of the three different LLMs across various dimensions are displayed in the figure, where red bars represent ChatGPT 4o, green bars represent Gemini 2.0, and blue bars represent DeepSeek R1 Because Phase 1 involved multiple rubric dimensions, cognitive-level strata, and disciplinary objectives, p values from these comparisons were interpreted as nominal and descriptive. These analyses were used to characterize model-performance patterns and support model selection based on the overall expert-rated instructional quality score, rather than to provide confirmatory evidence for isolated pairwise differences. Pilot randomized teaching trial: baseline comparability, examination performance, and post-course evaluations Baseline characteristics A total of 34 urology trainees were randomized to the LLM-assisted group (n = 17) or the control group (n = 17). Baseline characteristics were well balanced between groups. Mean age was 23.41 ± 1.33 years in the LLM-assisted group versus 23.76 ± 1.48 years in controls (p = 0.580). Baseline academic ability, measured by the pre-admission standardized test score, was comparable. Training-year distribution and prior AI use frequency did not differ meaningfully between groups (Table 2). Table 2Baseline characteristics of urology residents by study group. Baseline characteristics include age, training year, baseline standardized test score, and prior AI use frequency by study groupCharacteristicLLM-assisted group (n = 17)Control group (n = 17)Total(n = 34)P valueAge, years23.41 ± 1.3323.76 ± 1.4823.59 ± 1.400.580Training year, n (%) Year 110(58.82%)6(35.29%)16(47.06%) Year 27(41.18%)11(64.71%)18(52.94%) Year 30 (0.0%)0 (0.0%)0 (0.0%)Pre-admission standardized test score69.11 ± 13.4368.53 ± 13.8368.82 ± 13.430.951Prior AI use frequency, n (%)Low (< 1 time/week)13(76.47%)16(94.12%)29(85.30%)Medium (2–3 times/week)4(23.53%)1(5.88%)5(14.70%)High (≥ 4 times/week)0 (0.0%)0 (0.0%)0 (0.0%) Table 2Baseline characteristics of urology residents by study group. Baseline characteristics include age, training year, baseline standardized test score, and prior AI use frequency by study groupCharacteristicLLM-assisted group (n = 17)Control group (n = 17)Total(n = 34)P valueAge, years23.41 ± 1.3323.76 ± 1.4823.59 ± 1.400.580Training year, n (%) Year 110(58.82%)6(35.29%)16(47.06%) Year 27(41.18%)11(64.71%)18(52.94%) Year 30 (0.0%)0 (0.0%)0 (0.0%)Pre-admission standardized test score69.11 ± 13.4368.53 ± 13.8368.82 ± 13.430.951Prior AI use frequency, n (%)Low (< 1 time/week)13(76.47%)16(94.12%)29(85.30%)Medium (2–3 times/week)4(23.53%)1(5.88%)5(14.70%)High (≥ 4 times/week)0 (0.0%)0 (0.0%)0 (0.0%) Baseline characteristics of urology residents by study group. Baseline characteristics include age, training year, baseline standardized test score, and prior AI use frequency by study group 10 (58.82%) 6 (35.29%) 16 (47.06%) 7 (41.18%) 11 (64.71%) 18 (52.94%) 13 (76.47%) 16 (94.12%) 29 (85.30%) 4 (23.53%) 1 (5.88%) 5 (14.70%) Summative examination outcomes All trainees completed the 100-point closed-book final examination. The LLM-assisted group achieved higher total scores than controls (68.47 ± 12.78 vs. 57.91 ± 10.47), corresponding to a mean difference (MD) of 10.56 points (95% CI, 2.39–18.73; p = 0.013) (Table 3). In exploratory secondary analyses, the between-group difference was most evident in the multiple-choice component performance on the multiple-choice component (MD 9.27, 95% CI 4.37–14.17; p < 0.001), whereas MDT case analysis scores were similar between groups (MD 1.29, 95% CI − 3.52 to 6.10). Notably, the LLM-assisted group showed advantages on higher-demand content: difficult questions (34 points) demonstrated an MD of 3.71 (95% CI 0.06–7.36; p = 0.046), and research questions (21.5 points) showed an MD of 3.32 (95% CI 1.72–4.92; p < 0.001). Moderate-difficulty questions also favored the LLM-assisted group (MD 4.94, 95% CI 0.52–9.36; p = 0.029). In contrast, differences for easy questions and clinically oriented items overall did not reach conventional statistical significance, with confidence intervals indicating that both a small benefit and no effect remained plausible. Examination completion time was comparable between groups (Table 3). Table 3Comparison of performance between students in different groups. The finish times, total scores, scores on questions of varying difficulty levels, and scores on different question types for the two groups of students were thoroughly comparedGroupTotalAI-assistednon-AI-assistedMD(95%CI)P vauleFinish Time(min)63.83 ± 12.7067.00 ± 19.8160.67 ± 12.776.33, 95% CI 5.39–18.050.277ScoresTotal Points(100 points)63.19 ± 12.7068.47 ± 12.7857.91 ± 10.4710.56, 95% CI 2.39–18.730.013MCQ(50 points)35.07 ± 8.3639.71 ± 7.1730.44 ± 6.869.27, 95% CI 4.37–14.17< 0.001MDT(50 points)28.12 ± 6.8128.76 ± 6.9627.47 ± 6.801.29, 95% CI − 3.52–6.100.587Easy Questions(22.5 points)20.37 ± 2.9021.32 ± 1.7919.41 ± 3.481.91, 95% CI − 0.05–3.870.053Moderate Questions(43.5 points)27.68 ± 6.6930.14 ± 7.3225.20 ± 5.074.94, 95% CI 0.52–9.360.029Difficult Questions(34 points)15.15 ± 5.4717.00 ± 5.6013.29 ± 4.803.71, 95% CI 0.06–7.360.046Clinical Questions(78.5 points)50.59 ± 11.0754.00 ± 12.5946.97 ± 8.157.03, 95% CI − 0.43–14.490.055New Reserach Questions(21.5 points)12.60 ± 1.3514.26 ± 1.3510.94 ± 2.893.32, 95% CI 1.72–4.92< 0.001* The examination time is limited to 120 min** MCQ: Multiple Choice Questions*** MDT: Multi-Disciplinary Treatment Questions Table 3Comparison of performance between students in different groups. The finish times, total scores, scores on questions of varying difficulty levels, and scores on different question types for the two groups of students were thoroughly comparedGroupTotalAI-assistednon-AI-assistedMD(95%CI)P vauleFinish Time(min)63.83 ± 12.7067.00 ± 19.8160.67 ± 12.776.33, 95% CI 5.39–18.050.277ScoresTotal Points(100 points)63.19 ± 12.7068.47 ± 12.7857.91 ± 10.4710.56, 95% CI 2.39–18.730.013MCQ(50 points)35.07 ± 8.3639.71 ± 7.1730.44 ± 6.869.27, 95% CI 4.37–14.17< 0.001MDT(50 points)28.12 ± 6.8128.76 ± 6.9627.47 ± 6.801.29, 95% CI − 3.52–6.100.587Easy Questions(22.5 points)20.37 ± 2.9021.32 ± 1.7919.41 ± 3.481.91, 95% CI − 0.05–3.870.053Moderate Questions(43.5 points)27.68 ± 6.6930.14 ± 7.3225.20 ± 5.074.94, 95% CI 0.52–9.360.029Difficult Questions(34 points)15.15 ± 5.4717.00 ± 5.6013.29 ± 4.803.71, 95% CI 0.06–7.360.046Clinical Questions(78.5 points)50.59 ± 11.0754.00 ± 12.5946.97 ± 8.157.03, 95% CI − 0.43–14.490.055New Reserach Questions(21.5 points)12.60 ± 1.3514.26 ± 1.3510.94 ± 2.893.32, 95% CI 1.72–4.92< 0.001* The examination time is limited to 120 min** MCQ: Multiple Choice Questions*** MDT: Multi-Disciplinary Treatment Questions Comparison of performance between students in different groups. The finish times, total scores, scores on questions of varying difficulty levels, and scores on different question types for the two groups of students were thoroughly compared Finish Time (min) Total Points (100 points) MCQ (50 points) MDT (50 points) Easy Questions (22.5 points) Moderate Questions (43.5 points) Difficult Questions (34 points) Clinical Questions (78.5 points) New Reserach Questions (21.5 points) * The examination time is limited to 120 min ** MCQ: Multiple Choice Questions *** MDT: Multi-Disciplinary Treatment Questions Post-course trainee and instructor evaluations LLM-specific questionnaires were completed by trainees in the LLM-assisted group (n = 17) and instructors (n = 10) (Tables 4 and 5). Student feedback was uniformly positive for core learning benefits: all students reported improved understanding of complex concepts, better knowledge integration, and willingness to recommend continued LLM use; nearly all also endorsed enhanced clinical reasoning (16/17 positive). More variability was observed for implementation-related items, including increased in-class engagement (13/17 positive) and ease of formulating effective prompts (15/17 positive)(Table 4). Instructor responses were similarly favorable: all endorsed improved class preparedness and perceived improved assessment performance, with most supporting structured clinical reasoning and future adoption of LLM tools. Views were more mixed regarding whether LLM use promoted guideline consultation and whether reviewing LLM interactions reliably helped identify learning gaps (Table 5). Table 4Summary of student post-course questionnaire results (n = 17)QuestionStrongly DisagreeDisagreeNeutralAgreeStrongly AgreeThe LLM helped me better understand complex medical concepts710I was able to use the LLM to clarify my doubts before class1286Interacting with the LLM encouraged me to engage more actively in class467The LLM supported my ability to summarize and integrate knowledge89The logical structure of LLM responses helped me improve clinical reasoning1511I found it easy to phrase effective questions for the LLM1169The feedback discussions based on my LLM use improved my learning habits458I would recommend continued use of LLMs in future courses89 Table 4Summary of student post-course questionnaire results (n = 17)QuestionStrongly DisagreeDisagreeNeutralAgreeStrongly AgreeThe LLM helped me better understand complex medical concepts710I was able to use the LLM to clarify my doubts before class1286Interacting with the LLM encouraged me to engage more actively in class467The LLM supported my ability to summarize and integrate knowledge89The logical structure of LLM responses helped me improve clinical reasoning1511I found it easy to phrase effective questions for the LLM1169The feedback discussions based on my LLM use improved my learning habits458I would recommend continued use of LLMs in future courses89 Summary of student post-course questionnaire results (n = 17) Table 5Summary of instructor post-course questionnaire results (n = 10)QuestionStrongly DisagreeDisagreeNeutralAgreeStrongly AgreeStudents were better prepared for class when using the LLM46The quality of student questions improved through LLM pre-class use244The LLM helped students build more structured clinical reasoning145I observed improvements in students’ performance on assessments10Using the LLM encouraged students to consult updated guidelines1243Reviewing LLM interactions helped identify students’ learning gaps334Using the LLM helped reduce teaching workload without compromising quality244I am willing to incorporate LLM tools in future teaching235 Table 5Summary of instructor post-course questionnaire results (n = 10)QuestionStrongly DisagreeDisagreeNeutralAgreeStrongly AgreeStudents were better prepared for class when using the LLM46The quality of student questions improved through LLM pre-class use244The LLM helped students build more structured clinical reasoning145I observed improvements in students’ performance on assessments10Using the LLM encouraged students to consult updated guidelines1243Reviewing LLM interactions helped identify students’ learning gaps334Using the LLM helped reduce teaching workload without compromising quality244I am willing to incorporate LLM tools in future teaching235 Summary of instructor post-course questionnaire results (n = 10) Discussion AI-assisted teaching-positioned alongside blended online and offline instruction widely adopted during the COVID-19 pandemic and the long-standing Problem-Based Learning (PBL) and Case-Based Learning (CBL) pedagogies- represents a promising direction for clinical education. PBL engages learners through student-centered inquiry driven by open-ended clinical problems, whereas CBL uses authentic patient cases in faculty-facilitated small groups([5, 19–20]). During the pandemic, many institutions transitioned these modalities into hybrid formats, showing that technology-enhanced teaching can sustain learning outcomes and may improve student satisfaction and elements of clinical reasoning compared with lecture-only models [8]. Within this broader evolution, our study contributes a benchmark-to-classroom workflow: a head-to-head comparison of ChatGPT 4o, DeepSeek R1, and Gemini 2.0 for prostate cancer teaching prompts, followed by implementation of the selected model (DeepSeek R1) in professional-track master’s teaching to evaluate real-world educational impact. Our Phase 1 benchmarking aligns with the broader rationale for AI-augmented instruction—enhancing learner autonomy through interactive support that deepens understanding and facilitates application to clinical problem-solving [21, 22]. While accuracy is the minimal requirement for educational use [23], our findings reinforce that educational adequacy also depends on completeness and clinical salience. Even when overall correctness is high, clinically meaningful omissions can persist, as illustrated by biopsy technique details and pathology subtype coverage, where DeepSeek more consistently captured key points expected by experts [24, 25]. Beyond factual correctness, effective clinical teaching requires insight, guidance, and humanistic engagement that support inquiry and deeper learning [2, 23, 26]. In our benchmark, models varied in their ability to provide usable “teaching scaffolds” such as connecting anatomy to biopsy strategy or using analogies to clarify nuanced concepts, which may make LLM outputs more instructionally actionable during pre-class preparation and in-class discussion [24–29]. Humanistic care also warrants scrutiny from both learner and patient-centered perspectives [30, 31], particularly because professionalism elements may be underemphasized unless explicitly prompted and supervised [32]. Finally, because postgraduate trainees are expected to engage frontier evidence and develop research thinking [33], model performance on innovation-oriented questions is educationally consequential; DeepSeek’s relative strength on such items-including discussions that differentiate mCRPC treatment considerations and encourage exploration of resistance mechanisms- suggests potential value for research-minded learning behaviors [34]. Phase 2 provided preliminary classroom evidence that LLM-assisted teaching was associated with higher examination performance under controlled instructional exposure. In the pilot randomized trial, the AI-assisted group achieved higher closed-book summative examination scores than controls (MD 10.56, 95% CI 2.39–18.73). The gain was concentrated in MCQs (MD 9.27, 95% CI 4.37–14.17) and research/new development items (MD 3.32, 95% CI 1.72–4.92), whereas MDT case analysis showed no clear difference (MD 1.29, 95% CI 3.52–6.10). This pattern is pedagogically informative: LLM assistance may preferentially support learning activities that rely on rapid knowledge retrieval, synthesis, and structured summarization.The particularly strong advantage in research development items may also reflect the explicit emphasis on recent evidence in the LLM-assisted workflow. Because the prompts encouraged trainees to explore guideline-related reasoning, emerging literature, and research directions, students in the AI-assisted group may have engaged more frequently and more systematically with recent developments than controls. Therefore, this finding should be interpreted as the effect of an LLM-supported, prompt-guided learning process rather than LLM access alone.By contrast, MDT case analysis typically requires richer contextual reasoning, explicit justification, and trade-off-based decision-making grounded in guideline-concordant evidence. The confidence interval for MDT spans no effect, suggesting that LLM support alone may be insufficient to improve complex case reasoning without additional instructional scaffolding, such as structured reasoning templates, explicit requirements to cite guideline statements, and facilitated debrief sessions that translate LLM outputs into clinically defensible arguments. The LLM-assisted workflow may also have increased cumulative exposure to the learning material. Although classroom teaching time was matched, trainees in the LLM-assisted group generated structured summaries and draft answers that could be revisited during self-study. Thus, the observed score differences may reflect the combined effect of LLM support, structured study materials, and repeated exposure, rather than LLM access alone. Because out-of-class study time was not objectively measured, future studies should include platform logs, study-time tracking, or active controls using equivalent non-LLM study guides. Beyond exam outcomes, our findings have practical implications for implementation. Among trainees who received LLM-assisted teaching, feedback on LLM-specific learning support was generally positive, particularly for perceived comprehension, knowledge integration, and clinical reasoning support. However, responses regarding prompt formulation were more variable, suggesting that prompt literacy should be explicitly supported in future implementations. This dispersion highlights prompt literacy as a teachable competency: some learners may require explicit instruction in how to ask clinically meaningful questions, specify context and constraints, and request structured outputs. Instructor evaluations were similarly favorable yet more mixed regarding whether LLMs use promotes guideline consultation and whether reviewing LLMs interactions reliably identifies learning gaps. These signals matter for governance: without a structured workflow, LLM-assisted learning risks drifting toward convenience rather than evidence-based practice. Future implementations should therefore embed explicit guardrails, including mandatory cross-checking against authoritative guidelines, encouraging citation-aware outputs, and teacher-led feedback sessions that reinforce verification and clinical accountability [30–32]. Study limitations Several limitations should be considered when interpreting these findings. First, Phase 2 was conducted as a single-center pilot randomized teaching trial with a modest sample size (n = 34). The single-center setting also limits generalizability to other institutions, specialties, learner groups, and teaching cultures. Second, participants and instructors could not be blinded to group assignment because of the nature of the educational intervention. This may have introduced expectation effects or performance bias, particularly because trainees knew whether they were using an LLM-supported learning workflow. In addition, because participants used personal learning devices outside supervised classroom settings, adherence to the control condition could not be objectively verified through usage logs. Although written commitment and repeated instructions were used to reduce contamination, residual off-protocol LLM use cannot be completely excluded. Third, the outcomes mainly focused on short-term educational performance, especially a written summative examination at course completion. Therefore, the Phase 2 findings should be interpreted as preliminary and hypothesis-generating rather than as definitive evidence of effectiveness. Fourth, Phase 1 relied on expert judgment applied to a fixed 40-item prostate cancer question bank. Although raters were blinded to model identity and inter-rater reliability for the overall mean score was good, subjective scoring may still be influenced by differences in rater expectations, interpretation of the rubric, or preference for particular response styles. Also, Multiple comparisons were conducted in Phase 1 across rubric dimensions, cognitive levels, and disciplinary objectives, and several secondary and exploratory outcomes were examined in Phase 2. Although these analyses help describe performance patterns and possible mechanisms, they increase the risk of chance findings. Therefore, Phase 1 pairwise comparisons and Phase 2 secondary findings should be interpreted as exploratory and hypothesis-generating, rather than as definitive confirmatory evidence. Finally, the finding that DeepSeek R1 outperformed ChatGPT-4o and Gemini 2.0 in Phase 1 requires careful interpretation. All prompts were written and administered in English, so the result cannot simply be attributed to Chinese prompt wording. However, the broader study environment was embedded in a Chinese residency education context. The question bank was designed for local prostate cancer teaching, and the expert raters were Chinese urology educators familiar with local clinical training expectations. These contextual factors may have influenced judgments of instructional usefulness, inspiration, adaptability, and humanistic care. The standardized prompt structure and five-dimensional rubric may also have favored models that generated more organized, teacher-like, and pedagogically expansive responses. Moreover, the content focus on prostate cancer, guideline-related reasoning, and research development topics may have aligned particularly well with DeepSeek R1’s response style in this study. Therefore, DeepSeek R1’s apparent advantage should be understood as context- and task-dependent rather than as evidence of general superiority across all specialties, languages, learner levels, or assessment formats. Larger multicenter studies with prespecified endpoints, repeated LLM output sampling, longer follow-up, and performance-based outcomes are needed to confirm generalizability and educational impact. Conclusion This study suggests that integrating a carefully selected LLM into prostate cancer teaching for urology trainees was associated with higher knowledge-based examination performance, particularly in MCQ-style and innovation-oriented content, while benefits for MDT case reasoning remain uncertain. Overall, LLMs may serve as promising adjuncts for rapid knowledge retrieval and structured synthesis, but translating support into complex clinical reasoning likely requires structured scaffolding and verification workflows. Larger, multicenter studies with longer follow-up and performance-based endpoints are warranted to confirm effectiveness, generalizability, and real-world educational impact.