A novel resident physician examination using clinical simulation video to assess clinical competence in Japan: a cross-sectional study Authors: Kiyoshi Shikino, Yuji Nishizaki, Koshi Kataoka, Sho Fukui, Daiki Yokokawa, Taro Shimizu, Yu Yamamoto, Kazuya Nagasaki, Hiroyuki Kobayashi, Yasuharu Tokuda Published: 2024 ABSTRACT AbstractBackgroundThe general medicine in-training examination (GM-ITE) assesses physicians’ clinical knowledge. This study expanded on findings from a previous pilot study to assess the relationship between general medicine in-training examination (GM-ITE) scores and the diagnostic skills of resident physicians in Japan by employing an innovative clinical simulation video (CSV-IE).MethodsThis multicenter cross-sectional study included 4,677 resident physicians who took the GMITE between January 17 and 30, 2023. Participants watched the CSV-IE, depicting an emergency room scenario, and provided a diagnosis. The CSV-IE depicts an emergency case and provides a diagnosis. Discrimination indices were used to assess the CSV-IE’s effectiveness across clinical competence domains, and multilevel logistic regression was used to analyze physician- and hospital-level factors associated with correct diagnoses.ResultsCorrect diagnoses were provided by 470 participants (10.0%). The CSV-IE demonstrated high discriminatory power across all assessed domains, including basic clinical knowledge (DI = 0.44), symptomatology and clinical reasoning (DI = 0.31), physical examination and clinical procedure (DI = 0.35), and knowledge about the disease (DI = 0.25), supporting its utility as an effective assessment tool. In the multivariable analysis, factors associated with a higher likelihood of providing a correct CSV-IE diagnosis included a higher annual number of emergency outpatients (adjusted odds ratio: 1.025; 95% confidence interval [CI]: 1.003–1.047; P = .0230) and being in a higher postgraduate year (adjusted odds ratio: 1.387; 95% CI: 1.104–1.742; P = .005). Conversely, resident physicians at university hospitals were less likely to provide a correct CSV-IE response (adjusted odds ratio: 0.624; 95% CI: 0.435–0.896; P = .0107).ConclusionsCSV-IE modules may provide an integrative and realistic evaluation of clinical competence, addressing limitations of traditional MCQ-based assessments by offering contextualized, real-world scenarios that require dynamic decision-making and diagnostic reasoning.Supplementary InformationThe online version contains supplementary material available at 10.1186/s12909-024-06395-x.Keywords: Assessment, Clinical competence, Clinical simulation video, General medicine in-training examination, Postgraduate medical education Background Japan’s medical schools follow a six-year curriculum, comprising four years of preclinical and two years of clinical education, followed by a two-year postgraduate residency program [1–3]. This residency provides hands-on experience in key medical fields, honing essential skills for treating prevalent diseases regardless of specialty [4, 5]. The Japan Institute for Advancement of Medical Education Program (JAMEP) developed the general medicine in-training examination (GMITE) to assess resident physicians’ clinical knowledge, similar to the US internal medicine residency examination (IM-ITE) [6]. The GMITE objectively assesses clinical competency and provides practical feedback on areas for improvement [7]. It is a two-hour assessment comprising 80 multiple-choice questions (MCQs) [8]. It encompasses a broad spectrum of medical knowledge and competencies aligned with the clinical training objectives set by Japan’s Ministry of Health, Labour and Welfare (MHLW) [9, 10]. The GM-ITE is structured to evaluate four principal domains: basic clinical knowledge, symptomatology and clinical reasoning, physical examination and clinical procedure, and disease knowledge [11]. Approximately one-third of resident physicians take this annual voluntary exam [12, 13]. The traditional assessment of clinical competencies through MCQs is valuable; however, it may not encompass the full scope of a clinician’s diagnostic process in real-world practice [14, 15]. In clinical settings, resident physicians must navigate complex problem-solving and decision-making processes [9]. Video simulation emerges as an effective assessment tool, providing contextualized, real-world scenarios where residents must dynamically apply their knowledge, mirroring actual patient interactions [16–18]. However, limitations in time and resources necessitate balancing MCQs with other assessments to ensure comprehensive evaluation [19]. Therefore, this study developed a clinical simulation video (CSV) called “innovative examination” (CSVIE) for the GMITE to assess resident physicians’ clinical competency in a real-world setting. The CSV-IE features a medical interview and physical examination with a patient and their family in an emergency room [20]. To provide context, a pilot study was conducted in the academic year 2021 with 56 resident physicians to initially explore the relationship between GM-ITE scores and CSV-IE performance, which demonstrated significant correlations [20]. Building upon these insights, this study further expanded on the GM-ITE framework by integrating the CSV-IE. The present study builds upon a prior pilot study to further examine the relationship between GM-ITE scores and diagnostic skills among a larger and more diverse cohort of Japanese resident physicians. This study aims to evaluate the effectiveness and educational impact of the CSV-IE in assessing clinical competencies across various training environments. The study also includes an assessment of the realism and instructional value of the CSV-IE through anonymous surveys. Most participants found the CSV-IE more suitable than traditional methods for clinical competence assessment and reported increased learning motivation, suggesting its effectiveness in enhancing diagnostic assessments and educational outcomes in postgraduate medical education. Methods Study design, setting, and participants We conducted a multicenter cross-sectional study on the GM-ITE for the academic year 2022 (GM-ITE 2022). The GMITE uses a methodology similar to the IM-ITE [4, 21, 22]. In Japan, GM-ITE 2022 was included in a large number of residency programs in teaching hospitals, but resident physicians in other hospitals also voluntarily participated. Thus, 9,011 resident physicians from 662 hospitals throughout Japan participated in GM-ITE 2022 between January 17 and January 30, 2023. After its last question, the participants were asked to watch the CSV and provide a correct diagnosis for a simulated patient. Then, they completed the questionnaire survey regarding their demographics and residency program. We included all resident physicians who participated in GM-ITE 2022. We excluded those who did not undergo GM-ITE 2022 at the study sites, did not participate in or answer the CSV question, or did not provide informed consent for study participation. Procedures Innovative examination using high-quality video simulation In this study, we wrote a script depicting a simulated clinical interaction. The approximately five-minute video, filmed from a resident physician’s perspective, depicted a new patient and his family in an emergency room (Fig. 1). The patient’s main symptom was fainting, which turned out to be due to an acute pulmonary thromboembolism, posing a challenging diagnostic scenario for clinical reasoning. The CSV-IE was specifically designed to measure competencies in several key domains: clinical reasoning, diagnostic accuracy, patient interaction, and the resident physicians’ ability to integrate clinical findings. The simulated scenario was designed to evaluate resident physicians’ ability to recognize critical signs (e.g., tachypnea, jugular venous distension), engage in effective clinical decision-making under time pressure, and demonstrate proficiency in synthesizing information from both verbal and non-verbal cues. “Innovative examination” clinical simulation video (CSV-IE) He exhibited physical signs including tachypnea, jugular venous distension, and a right lower limb fracture; during auscultation, actual patient heart sounds were presented. This case was selected based on objectives in Japanese clinical residency training. The resident physician conducted a medical interview and examination, asking and answering questions while the camera recorded the patient’s and their family members’ verbal and non-verbal responses. Professional actors coached by the medical supervisors played their roles effectively. A professional television production company filmed the video and added effects (e.g., heart sounds). Three of the authors and three JAMEP medical supervisors oversaw the video production. The study participants watched the CSV immediately after completing the GMITE, then answered the following CSVIE question: Q. Please state the most likely diagnosis for this patient (free text). Q. Please state the most likely diagnosis for this patient (free text). Following the pilot study [20], multiple disease name patterns meaning “acute pulmonary thromboembolism” (e.g., acute pulmonary embolism, or pulmonary thromboembolism) were accepted as correct answers. Two authors independently assessed the answers and discussed, identified, and agreed on them. We measured the interrater reliability with the κ coefficient (0.8–1.0 = almost perfect, 0.6–0.8 = substantial, 0.4–0.6 = moderate, and 0.2–0.4 = fair) [23]. Discrimination indices The 80 GM-ITE questions cover four main categories: basic clinical knowledge (8 questions), symptomatology and clinical reasoning (18 questions), physical examination and clinical procedure (18 questions), and disease knowledge (36 questions). For total and categorical results, we calculated the discrimination indices (DI) of each GM-ITE and CSV-IE question. DI was defined based on the following Eq. (1) [24]: where a is the number of correct answers in the top 25th percentile; b is the number of incorrect answers in the top 25th percentile; c is the number of correct answers in the bottom 25th percentile; d is the number of incorrect answers in the bottom 25th percentile. The range of φ is −1 ≤ φ ≤ 1. Questions are considered unreliable if this index is below 0. A DI of ≥ 0.20 indicates that the question has high discriminatory power, and ≥ 0.40 indicates that the question is a substantially good measure. Other variables We used the following information from the GM-ITE questionnaire: gender, postgraduate year (PGY-1 or PGY-2), hospital type (community or university hospital), rotation in general medicine department, duration of the internal medicine rotation, number of monthly night shifts, average number of assigned inpatients, self-study time per day, and categorized weekly duty hours. Additionally, we collected hospital-level information from the database for residency training hospitals in Japan, including the number of total hospital beds, physicians, nurses, and emergency outpatients per year. Statistical analyses We conducted the analyses using SAS version 9.4 for Windows (SAS Institute Inc., Cary, NC, USA) in accordance with STROBE guidelines. Statistical significance was set at P < .05. We applied a multilevel logistic regression model to examine the factors associated with correct responses on the CSV-IE. This model enables analysis of both individual- and hospital-level factors while accounting for the hierarchical data structure with resident physicians nested within hospitals. This model was selected to allow for hospital-level clustering effects, acknowledging that variations in training environments and clinical resources across hospitals may impact individual performance on the CSV-IE. The multilevel model included both physician-level and hospital-level covariates. Covariate selection was informed by prior research and theoretical frameworks, focusing on factors identified as relevant to clinical competency outcomes. These covariates included hospital characteristics, such as the number of emergency outpatients per year, and physician-specific factors, including postgraduate year, general medicine rotation experience, duration of internal medical training, and average weekly duty hours [25–27]. By adjusting for these variables, we aimed to estimate the independent effect of each factor on the probability of achieving a correct CSV-IE response, thereby minimizing potential confounding biases. Finally, the discrimination index (DI) for each domain in the GM-ITE and CSV-IE was calculated to assess the discriminatory power of the test items. A DI of ≥ 0.20 indicated high discriminatory power, while a DI of ≥ 0.40 indicated a strong measure of competency. Results Demographic details A total of 4,677 resident physicians from 563 teaching hospitals in Japan were included in the analyses (Fig. 2). Study flow Of the total, 3,210 (68.6%) participants were men, and 2,357 (50.4%) were PGY-1 (Table 1). The majority (n = 3,842; 82.2%) were from community hospitals. A total of 470 (10.0%) participants provided correct diagnosis. The mean (± SD) GMITE score of all 4,677 participants was 45.77 ± 8.13. GM-ITE scores in the correct CSV-IE group were significantly higher than in the incorrect CSV-IE group (50.03 ± 8.54 vs. 45.29 ± 7.94, p < .0001). The interrater reliability with the κ coefficient of the CSV-IE was 0.92. Table 1Baseline characteristics of resident physicians according to the CSV-IE resultsInnovative Examination(high-quality patient-simulated video)CorrectIncorrectn = 470n = 4,207Mean (SD)p-valueGM-ITE score50.03 (8.54)45.29 (7.94)< 0.0001Hospital-level InformationMean (SD)p-valueTotal hospital beds545 (201)557 (223)0.2997Number of physicians187 (139)203 (166)0.0506Number of nurses544 (228)548 (237)0.7528Number of emergency outpatients per year9150 (4970)8240 (5070)0.0002n (%)p-valueHospital type< 0.0001 Community Hospital420 (89.4)3422 (81.3) University Hospital50 (10.6)785 (18.7)Resident Physician-level Informationn (%)p-valueGender0.4366 Male330 (70.2)2880 (68.5) Female140 (29.8)1327 (31.5)PGY< 0.0001 1195 (41.5)2162 (51.4) 2275 (58.5)2045 (48.6)General Medicine rotation0.0032 No227 (48.3)2332 (55.4) Yes243 (51.7)1875 (44.6)Internal Medicine rotation (month)0.0250 0–5107 (22.8)1153 (27.4) 6–10290 (61.7)2579 (61.3) 11–1566 (14.0)418 (9.9) 16–206 (1.3)40 (1.0) ≥ 211 (0.2)17 (0.4)ED Duty per month< 0.0001 09 (1.9)112 (2.7) 1–251 (10.8)738 (17.5) 3–5342 (72.8)2971 (70.6) ≥ 667 (14.3)371 (8.8) Unknown1 (0.2)15 (0.4)Average number of assigned inpatients0.8398 0–4170 (36.2)1609 (38.3) 5–9238 (50.6)2092 (49.7) 10–1441 (8.7)320 (7.6) ≥ 1512 (2.6)97 (2.3) Unknown9 (1.9)89 (2.1)Self-study time per day (minutes)0.2350 06 (1.3)73 (1.7) 1–30193 (41.1)1929 (45.9) 31–60199 (42.3)1591 (37.8) 61–9054 (11.5)482 (11.5) ≥ 9118 (3.8)132 (3.1)Duty-hour per week (hours)0.0329 Category 1 (< 60)210 (44.7)2147 (51.0) Category 2 (60–79)187 (39.8)1479 (35.2) Category 3 (≥ 80)73 (15.5)581 (13.8)CSV-IE Innovative Examination Clinical Simulation Video, GM-ITE General Medicine In-Training Examination Baseline characteristics of resident physicians according to the CSV-IE results CSV-IE Innovative Examination Clinical Simulation Video, GM-ITE General Medicine In-Training Examination DI results The DI revealed that all domains had discrimination indices greater than 0.2 (Table 2). The DI for the entire CSVIE portion of the GMITE was 0.25, 0.44 for basic clinical knowledge, 0.31 for symptomatology and clinical reasoning, 0.35 for physical examination and clinical procedure, and 0.25 for disease knowledge, indicating high discriminatory power in all domains. Table 2Discrimination index of innovative examination and GM-ITEDomains of questions(item number)Basic clinicalknowledgeSymptomatologyandClinical reasoningPhysical examinationandClinical procedureDiseaseKnowledgeTotalNumber of items818183680DI for CVS-IE0.440.310.350.250.25DI for GM-ITE(Mean, SD)0.30 ± 0.170.39 ± 0.100.38 ± 0.090.32 ± 0.100.28 ± 0.11A discrimination index ≥ 0.20 indicates that the question has a high discriminatory power, and a discrimination index of > 0.40 indicates that the question is a very good measure of the subject’s qualificationsGM-ITE General Medicine In-Training Examination Discrimination index of innovative examination and GM-ITE DI for GM-ITE (Mean, SD) A discrimination index ≥ 0.20 indicates that the question has a high discriminatory power, and a discrimination index of > 0.40 indicates that the question is a very good measure of the subject’s qualifications GM-ITE General Medicine In-Training Examination Factors associated with the correct answer in CVS-IE As shown in Supplementary file 1, the univariable logistic regression analysis showed that the correct CSV-IE group was associated with a high number of emergency outpatients per year (odds ratio [OR]: 1.033; 95% confidence interval [CI]: 1.015–1.051; P < .0001), university hospital (OR: 0.519; 95% CI: 0.383–0.703; P < .0001), PGY-2 (OR: 1.491; 95% CI: 1.229–1.809; P < .0001), general medicine rotation (OR: 1.331; 95% CI: 1.100–1.611; P = .0033), internal medicine rotation (11–15 months) (OR: 1.701; 95% CI: 1.227–2.358; P = .0014), night shifts per month (≥ 6) (OR: 2.247; 95% CI: 1.086–4.650; P = .0291), and weekly duty hours (60–79 h) (OR: 1.293; 95% CI: 1.050–1.591; P = .0155). In the multivariable analysis, a higher number of emergency outpatients per year (adjusted OR: 1.025; 95% CI: 1.003–1.047; P = .0230) and being in the PGY-2 (adjusted OR: 1.387; 95% CI: 1.104–1.742; P = .005) were independently associated with a higher likelihood of correctly answering the CSV-IE question (Table 3). Conversely, training at a university hospital was associated with a lower likelihood of a correct CSV-IE answer (adjusted OR: 0.624; 95% CI: 0.435–0.896; P = .0107), indicating that despite the rigorous training environment, university hospital residents may face challenges that impact their CSV-IE performance. Table 3Factors associated with the CSV-IE in multilevel logistic regression analysisHospital-level InformationAdjusted Odds Ratio(95% CI)p-valueNumber of emergency outpatients per year1.025 (1.003, 1.047)0.0230Hospital type Community Hospital(reference) University Hospital0.624 (0.435, 0.896)0.0107Resident Physician-level InformationAdjusted Odds Ratio(95% CI)p-valuePGY 1(reference) 21.387 (1.104, 1.742)0.0050General Medicine rotation No(reference) Yes1.198 (0.973, 1.475)0.0887Internal Medicine rotation (month) 0–5(reference) 6–101.084 (0.836, 1.406)0.5438 11–151.420 (0.975, 2.066)0.0672 16–201.357 (0.544, 3.386)0.5121 ≥ 210.521 (0.067, 4.074)0.5344ED Duty per month 0(reference) 1–20.727 (0.342, 1.545)0.4070 3–51.023 (0.499, 2.099)0.9503 ≥ 61.404 (0.647, 3.049)0.3910 Unknown0.699 (0.080, 6.101)0.7457Duty-hour per week (hours) Category 1 (< 60)(reference) Category 2 (60–79)1.120 (0.902, 1.391)0.3031 Category 3 (≥ 80)0.977 (0.722, 1.322)0.879795% CI 95% confidence interval, CSV-IE Innovative Examination Clinical Simulation Video Factors associated with the CSV-IE in multilevel logistic regression analysis Adjusted Odds Ratio (95% CI) 95% CI 95% confidence interval, CSV-IE Innovative Examination Clinical Simulation Video Discussion The residency program is a critical training phase in medical education, bridging formal instruction with clinical application. This study demonstrates the utility of CSV-IE as an innovative, real-world assessment tool that effectively measures resident physicians’ diagnostic capabilities, which are not fully captured by traditional MCQ-based assessments. The CSV-IE simulates real-world diagnostic scenarios, allowing for a more comprehensive assessment of clinical reasoning, patient interaction, and decision-making skills, which are essential in emergency and other high-pressure medical environments. Our multivariable analysis identified distinct characteristics of the correct CSV-IE group. Resident physicians from community hospitals, which have a higher number of emergency outpatient visits, and those in their PGY-2 were significantly more likely to answer correctly. This outcome likely reflects the relevance of the emergency scenario, which involved a simulated patient presenting with syncope. These findings suggest that resident physicians who are well-trained in emergency rooms exhibit superior performance in accurately diagnosing such conditions. The balance between clinical workload and educational resources is an essential component of residency training, influencing resident physicians’ clinical knowledge. A suitable mix of emergency department duties, inpatient care responsibilities, and access to evidence-based resources, such as online learning platforms, is associated with improved knowledge among resident physicians [10]. These aspects complement our study findings that the experiential learning environment significantly affects residents’ performance on assessments such as the CSV-IE, reflecting the practical application of their training and education. Furthermore, a notable discrepancy exists in the performance outcomes between resident physicians trained in university hospitals versus community-based hospitals [28], highlighting the need to standardize training programs to bolster overall clinical education and bridge competency gaps. Our study, with its extensive scale and broader scope, supports these insights and suggests that such disparities can indicate areas requiring curricular enhancements. By advocating uniform educational standards, we can strive for a comprehensive improvement in resident physicians’ training and assessment, ensuring that they receive a consistent and high-quality educational experience regardless of the training institution. Our results, as well as those of our previous pilot investigation [20], affirm the utility of CSV-IE-based assessments in evaluating multiple domains of clinical competence. This study expanded on the pilot by engaging a substantially larger cohort of resident physicians, thereby providing a more robust analysis of the CSV-IE’s effectiveness. Conversely, the present study broadened the scope to include thousands of resident physicians, providing a more generalizable understanding of the CSV-IE’s role in medical education. Both studies underscore the multidimensional nature of the CSV-IE-based assessment capabilities; however, this study offers a larger-scale validation of these findings. The present findings suggest that the CSV-IE could reliably assess clinical competencies across a range of training environments and educational contexts. The concept of extrapolation in Kane’s validity framework [29] is crucial for understanding how assessment outcomes relate to real-life clinical competencies. While the GM-ITE provides a structured evaluation of theoretical knowledge across multiple domains, the CSV-IE adds a practical dimension, offering insights into residents’ abilities to navigate complex, real-world diagnostic scenarios. Beyond assessment, the CSV-IE provides added value as a feedback tool for medical educators and curriculum developers. By highlighting specific competencies required in real-world emergency scenarios—such as rapid recognition of critical signs and effective clinical reasoning under pressure—the CSV-IE can identify skill gaps that may not be evident in traditional MCQ-based assessments. The GM-ITE, designed as a formative assessment, is a structured feedback tool to assess and improve residents’ foundational knowledge across domains [8]. The GM-ITE and CSV-IE provide complementary feedback, with the GM-ITE focusing on knowledge acquisition and the CSV-IE targeting applied clinical skills. The combined insights from both assessments can inform curriculum adjustments, ensuring a balanced emphasis on theoretical knowledge and practical competencies essential for clinical practice in alignment with Kane’s validity framework. The CSV-IE assesses specific clinical competencies that align more closely with real-life performance, such as recognizing critical symptoms, engaging in clinical reasoning under time pressure, and synthesizing patient information from both verbal and non-verbal cues [30]. By extrapolating performance in the CSV-IE to anticipated clinical practice, the assessment more accurately infers a resident’s readiness for real-world situations, enhancing the validity of the inferences made about the residents’ competency, based on the assessment results. Their implications are profound as they highlight the crucial role of hands-on, scenario-based training in emergency room settings that community hospitals often provide. This aspect of training appears to contribute significantly to the clinical acumen required in real-world emergency care. The observed effectiveness of resident physicians in their PGY-2 could indicate that the maturation of clinical skills is particularly pronounced at this training stage. This could inform the structuring of future residency programs to optimize the learning curve. The strength of ratings regarding different components of clinical reasoning indicates that although MCQs are effective in diagnosis, management, and treatment, they are limited in hypothesis generation, problem representation, and diagnostic justification [9]. This indicates that CSV-IE-based test modules could provide a more accurate measure of participants’ clinical knowledge and abilities than the GMITE. The CSV-IE demonstrated a more comprehensive assessment of clinical reasoning by incorporating non-verbal cues, patient interaction, and real-time decision-making. This module’s high DIs across all domains suggest a more nuanced evaluation of clinical competencies compared to the GM-ITE alone. Notably, the CSV-based assessment could simultaneously examine multiple domains, potentially reducing the number of questions required without compromising the breadth of skills assessed. Limitations Our study has certain limitations. First, the participants’ response rate (52.9%), while substantial, represents only a fraction of all resident physicians in Japan. It should be noted that 35% of the target population is in university hospitals and 65% in community hospitals [31]. Conversely, our study’s participant base was predominantly from non-university hospitals (82.2%), with a smaller proportion from university hospitals. This distribution could influence our findings’ generalizability. Increasing the sample size could enhance validity, but it would also amplify the administrative load of scoring—a crucial factor for the feasibility of large-scale CSV-IE implementation. Innovative solutions, such as automated scoring algorithms and selective sampling, could address this scalability issue. The second limitation is the authenticity of the CSV-IE. While efforts were made to create realistic clinical scenarios, some physical findings (apex pulsation, or Rivero-Carvallo sign) could not be adequately simulated, potentially impacting the examinees’ performance. Future research should explore integrating real patient interactions within CSV-IE modules to enhance authenticity and reduce production costs. Additionally, using a single simulation scenario in the CSV-IE may introduce a risk of situation-specific results [32], potentially limiting the assessment’s ability to evaluate a wider range of clinical competencies. This single emergency scenario—focused on diagnosing acute pulmonary thromboembolism—primarily tests emergency diagnostic skills, which may not fully reflect resident physicians’ broader diagnostic and communication skills across diverse clinical contexts. Including multiple scenarios in future iterations of the CSV-IE could provide a more comprehensive measure of clinical competence by assessing a broader array of skills, such as differential diagnosis and patient interaction in various medical settings. This adjustment could further enhance the CSV-IE’s value as a holistic assessment tool. Finally, the CSV-IE in this study was designed for an emergency outpatient setting, and our survey did not measure whether resident physicians had “active” rather than “observational” case experiences in these settings. Some resident physicians primarily learn by observing diagnostic procedures and medical techniques rather than directly engaging in patient care. The association of improved CSV-IE scores with factors such as employment in community hospitals, being in PGY-2, and the volume of emergency department visits at the medical institution highlights the need for future surveys that assess these variables. Conclusions The CSV-IE offers a promising complement to traditional MCQ assessments in residency training, providing a more integrative and realistic evaluation of clinical competence. Future studies should refine the CSV-IE to balance feasibility with authenticity for advancing medical education assessments. Several steps could be considered to fully leverage CSV-IE’s potential benefits and evolve medical education assessments. Supplementary Information Supplementary Material 1. Factors associated with the CSV-IE in univariable logistic regression analysis. Acknowledgements We thank the members of the JAMEP for their valuable assistance and Dr. Kentaro Sakamaki from the Faculty of Health Data Science at Juntendo University for his statistical guidance. We also thank Editage for the English language review. Abbreviations confidence interval clinical simulation video innovative examination clinical simulation video discrimination index general medicine in-training examination internal medicine residency examination Japan Institute for Advancement of Medical Education Program multiple-choice questions odds ratio postgraduate year Professional and Linguistic Assessments Board Strengthening the Reporting of Observational Studies in Epidemiology Authors’ contributions KS had full access to all the study data and took responsibility for the integrity and accuracy of the data analysis. KS, YN, SF, DY, and YT: study concept and design; HK, TS, and YY: data acquisition, analysis, and interpretation; KS: manuscript drafting; YN: critical content revisions; KS: statistical analysis; YN and YT: administrative, technical, and material support; YN, SF, DY, YY, HK, TS, and YT: supervision. Additionally, YN contributed equally to this work and should be considered a co-corresponding author. Funding This work was supported by the Health, Labour, and Welfare Policy Grants of Research on Region Medical (21IA2004) from Japan’s Ministry of Health, Labour, and Welfare. The funder had no role in the design and conduct of the study, collection, management, analysis, and interpretation of the data, preparation, review, or approval of the manuscript, and decision to submit the manuscript for publication. Data availability The dataset used and/or analyzed during this study is available from the corresponding author upon reasonable request. Declarations Ethics approval and consent to participate This study was approved by the Ethical Review Committee of the Japan Organization for Advancing Medical Education (approval no. 23–26). All participants provided informed consent before participating in the study. This study was conducted according to the ethical standards and principles of the Declaration of Helsinki. Consent for publication Informed consent was obtained from all participants for the publication of identifying information in an online open-access publication. All participants read and signed the informed consent document before participating in the study. To ensure confidentiality, all participant data were anonymized prior to analysis. No compensation was provided to the participants for their involvement in this study. In accordance with ethical standards and journal policy, we have obtained explicit informed consent from all actors appearing in the video material associated with this study. The actors have acknowledged and agreed that the video will be published as part of the study’s material. Competing interests JAMEP was involved in collecting and managing data as the GM‐ITE administrative organization. It did not participate in designing and conducting the study, data analysis and interpretation, preparation, review, or approval of the manuscript, and the decision to submit the manuscript for publication. Dr. Nishizaki received an honorarium from JAMEP as the GM ITE project manager. Dr. Tokuda is the director of the JAMEP. Dr. Kobayashi received an honorarium from the JAMEP as a speaker for the JAMEP lecture. Dr. Shikino received an honorarium from the JAMEP as a reviewer of GM ITE. Dr. Shikino, Dr. Shimizu, and Dr. Yamamoto received honoraria from the JAMEP as exam preparers of GM ITE. No other authors possess any competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Contributor Information Kiyoshi Shikino, Email: kshikino@gmail.com. Yuji Nishizaki, Email: ynishiza@juntendo.ac.jp. References Associated Data Supplementary Materials Supplementary Material 1. Factors associated with the CSV-IE in univariable logistic regression analysis. Data Availability Statement The dataset used and/or analyzed during this study is available from the corresponding author upon reasonable request.