Script concordance tests production optimization using enhanced interactive training: randomized, single-blind, bicentric trial Authors: Antoine Gaillet, Anne-Ségolène Cottereau, Alain Cariou, David Lebeaux, Cédric Lemogne, Nicolas de Prost, Dominique Vodovar, Damien Roux Published: 2026 ABSTRACT BackgroundThe Script Concordance Test (SCT) is an effective tool for clinical reasoning assessment. However, many faculty members are still unfamiliar with this modality, and there are challenges in recruiting panelists who are mandatory for the assessment of SCT. The aim of this study was to assess the benefits of providing training in the quality of SCTs produced by junior faculties members.MethodsThis was a randomized, single-blind, two-centre trial. The intervention consisted of 2 h of interactive training. The primary outcome was a comparison of the percentage of post-intervention valid SCT between the trained and the control group. The secondary outcome was to determine the factors associated with the quality of SCT.ResultsIn two medical schools in the Paris area, 115 participants were recruited in February 2024; they produced 460 SCT. Nine panels of experts were assembled. The percentage of post-intervention valid SCTs was significantly higher in the trained group (41% versus 16%, p < 0.001). This was confirmed in a sensitivity analysis performed on all the SCTs, adjusted for unbalanced variables between the two randomization arms (“risk” of producing a valid SCT when the participants were in the trained arm: aOR 2.53 [1.57; 4.13], p < 0.001).ConclusionOur prospective randomized study suggested the significant benefits for inexperienced physicians of an initial training in SCT writing quality. Introduction Clinical reasoning is widely understood as a hypothetico-deductive process, involving the early generation of hypotheses, targeted data collection, and decision-making based on the confirmation or refutation of those hypotheses using the collected data. The Script Concordance Test (SCT), developed by Charlin [1, 2] and based on the Script Theory [3], has proven particularly effective in assessing the key stages of the medical reasoning process. In this context, scripts represent pre-stored knowledge frameworks that clinicians use to interpret clinical information, aiming to confirm or refute diagnostic hypotheses. In a SCT [4], examinees are presented with brief clinical scenarios and asked a series of questions requiring them to assess diagnostic or management strategies as new information is introduced. The core of the clinical reasoning process relies on evaluating how each new piece of information influences the working hypothesis. Responses are scored on a 5-point Likert scale, ranging from '−2' to ' + 2', reflecting the degree to which the new information supports or undermines the hypothesis. Scoring is based on the range of panelists’ responses and measures how closely an examinee's interpretation of clinical data aligns with that of an experienced panel of clinicians, known as the "expert panel." The validity, reliability, and feasibility of SCT have been demonstrated in studies conducted in various branches of the health sciences [5], as well as in health professions education at all levels – from undergraduate [6–10] to postgraduate [11, 12] – and continuing professional development [13]. Guidelines for the development of SCTs have been established [14, 15]. However, the effectiveness of SCTs as an assessment tool relies heavily on careful item development and careful selection of the expert panel, both of which can be challenging. In France, the use of SCTs has been progressively introduced in medical schools [7, 16] and at the national level for post graduate students, as exemplified by its use in self-assessment for neurosurgery residents [12]. A major development occurred with the introduction of SCTs into the national ranking exam for entering residency, called the “Épreuves Dematérialisées Nationales (EDN)”, which determine the specialty and practice location of the future residents. In fact, two new evaluation modalities have been introduced: the Structured Clinical Examination (OSCE) [17, 18] and the SCT, which have been implemented for the first time in the 2024 national exam. However, many physicians remain relatively unfamiliar with SCT and there are significant challenges in recruiting experts for SCTs, which affects the ability to generate enough high-quality SCTs, as well as in teaching expert panels to validate them, despite the expansion of selection criteria for panel members [19]. The aim of this prospective bicentric randomized study was to evaluate the effectiveness of enhanced pedagogical training in improving the quality of SCTs written by junior faculties members. Materials and methods Study design We conducted a randomized, bicentric, single blind study (intrinsic and extrinsic validation of SCTs (see below) were performed blinded to the group allocation, whereas participants themselves were not blinded to group allocation) at two medical schools in the Paris area: Université Paris-Est Créteil (UPEC), in Créteil and Université Paris Cité (UPC), in Paris. Participants The study participants were clinical fellows from all specialties, with one to four years of experience after completing their residency, at UPEC and UPC during the 2023–2024 academic year. None had previously served as members of an SCT expert panel at their institutions, nor had they been exposed to SCT methodology during their own training, as SCTs have only recently been introduced into medical education curricula in France. Participants were recruited through a single institutional email and participation was voluntary. Randomization procedure We performed a 1:1 randomization of the participants (in alphabetical order) between two types of SCT training (control versus trained), stratified by center, as illustrated in the flow-chart (Fig. 1).Fig. 1Flow-chart of the Script Concordance Test production optimization study. Abbreviations: SCT: script concordance test; UPC: Université Paris Cité; UPEC: Université Paris-Est Créteil Flow-chart of the Script Concordance Test production optimization study. Abbreviations: SCT: script concordance test; UPC: Université Paris Cité; UPEC: Université Paris-Est Créteil Study procedure At the beginning of the study, on February 2024, we first sent by e-mail to all participants a reference teaching aid writing by medical pedagogical referents, previously developed (but not published) by the SCT teaching referents of UPC. This teaching material consisted of a 63-slide document covering: (1) principles of Script Concordance Testing, (2) scoring methodology, (3) item-writing guidelines with examples, (4) expert panel composition, (5) validation of expert responses, (6) expected score distribution under random responding, (7) optimal test length and construction time, (8) use of SCT in assessing clinical reasoning and relational care, and (9) common pitfalls in SCT construction. All participants were then asked to produce, within 15 days, a first series of 2 SCTs (“SCT 1”, each SCT comprising 3 items). The main aim of this initial phase was to validate the comparability of the two groups (before training) as participants were volunteers. The trained group received a 2-h interactive training session within 30 days after producing the SCT 1. This interactive session, delivered at both sites by a medical education professor (D.R.), was structured around a 58-slide presentation. It included: (1) theoretical foundations and educational relevance of SCTs, (2) SCT structure and mechanics, (3) scoring principles, and (4) interactive analysis using a real-time audience response system (Wooclap®, Wooclap SA, Belgium) of participants previously submitted SCTs, highlighting examples of high and low-quality items (e.g., missing clinical coherence, inappropriate item formulation, syntactic errors, incorrect format and/or distribution of participants’ responses during the training) illustrated the fundamental concepts of SCT. The control group received no training other than the reference teaching aid. On April 2024, all participants were asked to produce 2 new SCTs (“SCT 2”, each comprising 3 items) within 15 days of the training date. For both groups, the reference teaching aid was sent back again at the same point. Participant engagement was ensured through two collective reminder emails followed by two individualized reminders for non-responders. For each “SCT” series, the SCT subject was free, however the specific disciplines of the available expert panels (see below) were clearly defined to ensure optimal alignment between the participants' test content and the expertise of the panels, thereby enhancing the relevance and accuracy of the evaluation process. Panelists The expert panels represented 14 specialties (made up of experts who are regularly consulted and trained), all from UPC. Each panel comprised between 10 and 25 experts. Evaluation of SCTs We conducted the evaluation of the SCTs (1 and 2) in two steps. First, we performed an “intrinsic validation” (AS.C. and A.G. independently assessed all the SCTs, and, in case of disagreement, D.R. made the final decision.) following standard recommendations [14]. Intrinsic validation assessed the internal coherence of each SCT item (clarity of clinical vignette, consistency between hypothesis and response options, and adherence to SCT format). Items were considered intrinsically invalid when the clinical scenario was incomplete, inconsistent with the proposed items, or did not respect SCT structure. Then, only SCTs considered valid during the first step were submitted to the expert panels for evaluation. Extrinsic validation was based on expert panel response distributions. Valid SCTs required a non-discontinuous distribution, a quasi-Gaussian pattern of responses, and the presence of a dominant modality, in accordance with established SCT methodology (Fig. 2) [15]. Monomodal SCTs (that is, if all the panelists gave the same answer) were accepted.Fig. 2Examples of distribution of valid TCS: A Gaussian, B unimodal; and invalid TCS: C balanced, D discontinuous Examples of distribution of valid TCS: A Gaussian, B unimodal; and invalid TCS: C balanced, D discontinuous Panel responses were made via GoogleForms® questionnaires (Google®, Mountain View, CA). Panelists had the option of responding that they were not comfortable with the item. Each specialty panel received at least three invitations to fill in the questionnaire. Endpoints The primary endpoint was the percentage of valid SCT 2 in the two groups. The secondary objective was to determine the prognostic factors of SCT quality (trained arm, SCT round, university, medical discipline, number of years post residency, specialty mismatch between SCTs and participants, SCT types). Ethical considerations This study was conducted in accordance with the ethical principles of the Declaration of Helsinki (2013 revision). The research protocol was reviewed and approved by the Ethics Evaluation Committee for Biomedical Research Projects (CEERB) Paris Nord — registered under IRB 00006477, affiliated with the university hospital group Hôpitaux Universitaires Paris Nord Val-de-Seine (HUPNVS), Université Paris Cité/Assistance Publique–Hôpitaux de Paris (AP-HP). All participants were physicians who received written and oral information regarding the study objectives, procedures, and voluntary nature of their participation. Informed consent to participate was obtained from all participants in the study. No patient data were collected or used. Statistical analysis Results were presented as median with interquartile range (IQR) for continuous variables, and as number with percentage for binary and categorical variables. Qualitative variables were compared by Chi2 or Fisher test, and quantitative variables by Student or Wilcoxon test, depending on the conditions of employment. The number of participants required for the main analysis was calculated based on 50% valid SCT 2 in the control group, with an expected gain of 20% following the intervention, taking a two-sided alpha risk of 5% and a beta risk of 10%, and considering 5% of participants lost to follow-up. Thus, the number of SCT 2 required was 240, i.e. 120 per arm, and therefore 60 participants per arm. Despite the randomized design, participants were recruited on a voluntary basis, so imbalances between the two groups were expected. A global analysis of the SCTs was therefore planned in order to study the factors predicting the quality of the SCTs (including the round and the randomization arm). Thus, prognostic factors study of SCT quality was carried out on all SCT (1 and 2) by logistic regression with random effect on participants (due to the existence of repeated measures), selecting relevant variables (independently of the degree of significance in univariate) and respecting the threshold of 10 events (valid SCT) per adjustment variable, yielding unadjusted and adjusted odds ratio (aOR) and their 95% confidence interval. A search for collinearity and interaction was also performed when appropriate. All p-values were two-tailed, with a threshold < 0.05 considered statistically significant. Data were analyzed using R 4.0.5 (R Foundation for Statistical Computing, Vienna, Austria). Results General A total of 115 participants were recruited in February 2024, 41% from UPC and 59% from UPEC (Table 1, Figs. 1 and 3). Most participants were from medical specialties (n = 90, 78%), followed by surgical (n = 14, 12%) and non-clinical specialties (n = 11, 10%). The median number of years since residency completion was 2 [1–3]. Participants in the trained group were earlier in their post-residency training than those in the control group (48%, n = 27, with 1 year of experience vs. 32%, n = 19, respectively; p < 0.001). Notice that almost half (43%, n = 49) of the participants had to produce SCT in a different discipline than their own.Table 1Characteristics of the participantsVariablesTotal(n = 115)Control(n = 59)Trained(n = 56)pUniversité Paris CitéUniversité Paris Est Créteil47 (41%)68 (59%)22 (37%)37 (63%)25 (45%)31 (55%)0.109Discipline: - Non-clinical - Surgery - Medical11 (10%)14 (12%)90 (78%)3 (5%)9 (15%)47 (80%)8 (14%)5 (9%)43 (77%) < 0.001Number of years post residency (n = 105) - 1 - 2 - ≥ 346 (40%)35 (30%)24 (20%)19 (32%)22 (37%)12 (20%)27 (48%)13 (23%)12 (21%) < 0.001Specialty mismatch between SCTs and participants49 (43%)24 (40%)25 (45%)0.443Abbreviation: SCT script concordance testsFig. 3Participants’ specialties distribution. Legend: Red zone for medical specialties, blue zone for surgical specialties and green zone for non-clinicians. Number in parentheses for the number of participants concerned Table 1Characteristics of the participantsVariablesTotal(n = 115)Control(n = 59)Trained(n = 56)pUniversité Paris CitéUniversité Paris Est Créteil47 (41%)68 (59%)22 (37%)37 (63%)25 (45%)31 (55%)0.109Discipline: - Non-clinical - Surgery - Medical11 (10%)14 (12%)90 (78%)3 (5%)9 (15%)47 (80%)8 (14%)5 (9%)43 (77%) < 0.001Number of years post residency (n = 105) - 1 - 2 - ≥ 346 (40%)35 (30%)24 (20%)19 (32%)22 (37%)12 (20%)27 (48%)13 (23%)12 (21%) < 0.001Specialty mismatch between SCTs and participants49 (43%)24 (40%)25 (45%)0.443Abbreviation: SCT script concordance tests Characteristics of the participants Université Paris Cité Université Paris Est Créteil 47 (41%) 68 (59%) 22 (37%) 37 (63%) 25 (45%) 31 (55%) - Non-clinical - Surgery - Medical 11 (10%) 14 (12%) 90 (78%) 3 (5%) 9 (15%) 47 (80%) 8 (14%) 5 (9%) 43 (77%) - 1 - 2 - ≥ 3 46 (40%) 35 (30%) 24 (20%) 19 (32%) 22 (37%) 12 (20%) 27 (48%) 13 (23%) 12 (21%) Abbreviation: SCT script concordance tests Participants’ specialties distribution. Legend: Red zone for medical specialties, blue zone for surgical specialties and green zone for non-clinicians. Number in parentheses for the number of participants concerned Of the 14 expert planned panels, only 9 could be established, ranging from 10 to 17 experts (Fig. 4). The panels for maxillofacial surgery, orthopedic surgery, oncology, geriatrics and nephrology were not selected due to a lack of experts (1 to 7 experts).Fig. 4Experts’ panels specialties distribution. Legend: Red zone for medical specialties and blue zone for surgical specialties. Number in parentheses for the number of experts concerned Experts’ panels specialties distribution. Legend: Red zone for medical specialties and blue zone for surgical specialties. Number in parentheses for the number of experts concerned A total of 460 SCTs were generated over the two rounds (230 “SCT 1” and 230 “SCT 2”; 236 SCTs in the control arm and 224 SCTs in the trained arm), the majority of which were diagnostic SCTs (62%, n = 142). SCTs were generated for all panels, with emergency, critical care, haematology, infectious diseases and nephrology being the most represented specialties (Fig. 5).Fig. 5Script Concordance Test specialties distribution. Legend: Number in parentheses for the number of Script Concordance Tests produced Script Concordance Test specialties distribution. Legend: Number in parentheses for the number of Script Concordance Tests produced Approval process Half of the SCTs (1 and 2) were intrinsically validated, balanced between the two arms for SCTs 1 (49%, n = 58/118, versus 54%, n = 60/112, respectively for the control and trained arms, p = 0.512), and with a higher percentage for the trained group for SCTs 2 (34%, n = 40/118, versus 66%, n = 74/112, respectively for the control and trained arms, p < 0.001, Table 2). The main reason for excluding SCTs at this stage was the lack of consistency of the clinical vignette with the proposed items.Table 2Comparison of SCTs by round and randomization armVariablesTotalControlTrainedp3 (SCTs 1)p4 (SCTs 2)SCT 1SCT 2SCT 1SCT 2p1SCT 1SCT 2p2SCT types: - Investigation - Treatment - Diagnostic56 (24%)27 (12%)142 (62%)42 (18%)30 (13%)150 (65%)32 (27%)11 (9%)74 (63%)21 (18%)17 (14%)76 (64%)0.16024 (21%)16 (14%)68 (61%)21 (19%)13 (12%)74 (66%)0.8670.2710.951Intrinsic validity *118 (51%)114 (49%)58 (49%)40 (34%)0.02460 (54%)74 (66%)0.0460.512 < 0.001Extrinsic validity **68 (33%)57 (28%)33 (31%)17 (16%)0.01635 (34%)40 (41%)0.5710.665 < 0.001p1: SCT 1 vs SCT 2 for control arm; p2 = SCT 1 vs SCT 2 for trained arm; p3 = SCT 1 control arm vs SCT 1 trained arm; p4 = SCT 2 control arm vs SCT 2 trained armAbbreviation: SCT script concordance tests*total of 460 SCT, 230 SCT 1 et 230 SCT 2; 118*2 SCT (1 et 2) in the control arm; 112*2 SCT (1 et 2) in the trained arm**on the whole SCTs minus those 49 without any evaluation by experts: n = 411 (206 SCT 1 and 205 SCT 2) Table 2Comparison of SCTs by round and randomization armVariablesTotalControlTrainedp3 (SCTs 1)p4 (SCTs 2)SCT 1SCT 2SCT 1SCT 2p1SCT 1SCT 2p2SCT types: - Investigation - Treatment - Diagnostic56 (24%)27 (12%)142 (62%)42 (18%)30 (13%)150 (65%)32 (27%)11 (9%)74 (63%)21 (18%)17 (14%)76 (64%)0.16024 (21%)16 (14%)68 (61%)21 (19%)13 (12%)74 (66%)0.8670.2710.951Intrinsic validity *118 (51%)114 (49%)58 (49%)40 (34%)0.02460 (54%)74 (66%)0.0460.512 < 0.001Extrinsic validity **68 (33%)57 (28%)33 (31%)17 (16%)0.01635 (34%)40 (41%)0.5710.665 < 0.001p1: SCT 1 vs SCT 2 for control arm; p2 = SCT 1 vs SCT 2 for trained arm; p3 = SCT 1 control arm vs SCT 1 trained arm; p4 = SCT 2 control arm vs SCT 2 trained armAbbreviation: SCT script concordance tests*total of 460 SCT, 230 SCT 1 et 230 SCT 2; 118*2 SCT (1 et 2) in the control arm; 112*2 SCT (1 et 2) in the trained arm**on the whole SCTs minus those 49 without any evaluation by experts: n = 411 (206 SCT 1 and 205 SCT 2) Comparison of SCTs by round and randomization arm - Investigation - Treatment - Diagnostic 56 (24%) 27 (12%) 142 (62%) 42 (18%) 30 (13%) 150 (65%) 32 (27%) 11 (9%) 74 (63%) 21 (18%) 17 (14%) 76 (64%) 24 (21%) 16 (14%) 68 (61%) 21 (19%) 13 (12%) 74 (66%) p1: SCT 1 vs SCT 2 for control arm; p2 = SCT 1 vs SCT 2 for trained arm; p3 = SCT 1 control arm vs SCT 1 trained arm; p4 = SCT 2 control arm vs SCT 2 trained arm Abbreviation: SCT script concordance tests *total of 460 SCT, 230 SCT 1 et 230 SCT 2; 118*2 SCT (1 et 2) in the control arm; 112*2 SCT (1 et 2) in the trained arm **on the whole SCTs minus those 49 without any evaluation by experts: n = 411 (206 SCT 1 and 205 SCT 2) Because five expert panels could not be formed, 47 intrinsically validated SCT (balanced between randomization phases and arms) had to be excluded (Fig. 1). Among the remaining SCTs, all response modalities were represented (Fig. 6), with the + 1 modality dominating (2.9 experts/item). On average, 2.8 modalities were answered per SCT, and only 8 SCTs (4%) had a monomodal response. Of the 60 SCTs extrinsically excluded, 70% were excluded because of non-Gaussian distribution of the responses without dominant modality, 20% because of discontinuity of responses, and 10% because all modalities had the same number of experts with no dominant modality.Fig. 6Average modalities distribution of the intrinsically validated Script Concordance Test Average modalities distribution of the intrinsically validated Script Concordance Test Thus, while the percentage of valid SCTs 1 at the end of the process (intrinsic and extrinsic validation) was similar between the two arms (31%, n = 33/105, versus 34%, n = 35/103, respectively for the control and trained arms, p = 0.665), there were significantly more valid SCTs 2 in the trained arm than in the control arm (n = 17/107 (16%) versus n = 40/98 (41%), p < 0.001, Table 2). Such difference can be explained by the drop in SCT quality in the control group between the two rounds (p = 0.016) concomitant with a non-significant improvement in that of the experimental arm (p = 0.571). SCT quality predictors The multivariate model applied to all SCTs, adjusted for all variables of interest (including those that were imbalanced between the two arms), suggested the superiority of the trained arm over the control arm (aOR 2.53 [1.57; 4.13], p < 0.001, Table 3). While the second round (all SCTs taken as a whole) was not associated with SCTs quality, there was an interaction between the round and the randomization arm (p = 0.001) meaning that round 2 was a prognostic factor for SCT quality only in the trained arm. There was also an inverse relationship between age and quality of SCT, as well as an impact of the type of SCT on its quality, with diagnostic SCT more often valid than therapeutic and investigative SCT.Table 3Association of clinical factors and validity of SCTsVariablesTotal(n = 460)SCT invalid(n = 335)SCT valid(n = 125)ORIC 95%paORIC 95%pTrained arm224 (49%)149 (44%)75 (60%)1.87 [1.24; 2.85]0.0032.53 [1.57; 4.13] < 0.001SCT round 2*230 (50%)173 (52%)57 (46%)0.78 [0.52; 1.18]0.2490.72 [0.45: 1.14]0.161Université Paris Cité188 (41%)133 (40%)55 (44%)1.19 [0.79; 1.81]0.4041.02 [0.60; 1.70]0.950Discipline: - Non-clinical - Surgery - Medical44 (9%)56 (12%)360 (78%)29 (9%)45 (13%)261 (78%)15 (12%)11 (9%)99 (79%)Ref0.47 [0.19; 1.16]0.73 [0.38; 1.46]0.252Ref0.74 [0.27; 2.02]0.85 [0.38; 1.96]0.843Number of years post residency (n = 420): - 1 - 2 - ≥ 3184 (40%)140 (30%)96 (21%)127 (38%)100 (30%)80 (24%)57 (46%)40 (32%)16 (13%)Ref0.89 [0.55; 1.44]0.44 [0.23; 0.81]0.025Ref1.06 [0.63; 1.79]0.45 [0.23; 0.85]0.023Specialty mismatch between SCTs and participants196 (43%)145 (43%)52 (42%)0.93 [0.61; 1.41]0.7450.82 [0.46; 1.45]0.501SCT types (n = 447): - Investigation - Treatment - Diagnostic98 (21%)57 (12%)292 (63%)78 (23%)47 (14%)197 (59%)20 (16%)10 (8%)95 (76%)Ref0.83 [0.35; 1.89]1.88 [1.10; 3.32] < 0.001Ref0.89 [0.34; 2.19]2.22 [1.24; 4.15] < 0.001Abbreviation: SCT script concordance tests*Interaction: Trained arm X SCT round 2 = 0.001 Table 3Association of clinical factors and validity of SCTsVariablesTotal(n = 460)SCT invalid(n = 335)SCT valid(n = 125)ORIC 95%paORIC 95%pTrained arm224 (49%)149 (44%)75 (60%)1.87 [1.24; 2.85]0.0032.53 [1.57; 4.13] < 0.001SCT round 2*230 (50%)173 (52%)57 (46%)0.78 [0.52; 1.18]0.2490.72 [0.45: 1.14]0.161Université Paris Cité188 (41%)133 (40%)55 (44%)1.19 [0.79; 1.81]0.4041.02 [0.60; 1.70]0.950Discipline: - Non-clinical - Surgery - Medical44 (9%)56 (12%)360 (78%)29 (9%)45 (13%)261 (78%)15 (12%)11 (9%)99 (79%)Ref0.47 [0.19; 1.16]0.73 [0.38; 1.46]0.252Ref0.74 [0.27; 2.02]0.85 [0.38; 1.96]0.843Number of years post residency (n = 420): - 1 - 2 - ≥ 3184 (40%)140 (30%)96 (21%)127 (38%)100 (30%)80 (24%)57 (46%)40 (32%)16 (13%)Ref0.89 [0.55; 1.44]0.44 [0.23; 0.81]0.025Ref1.06 [0.63; 1.79]0.45 [0.23; 0.85]0.023Specialty mismatch between SCTs and participants196 (43%)145 (43%)52 (42%)0.93 [0.61; 1.41]0.7450.82 [0.46; 1.45]0.501SCT types (n = 447): - Investigation - Treatment - Diagnostic98 (21%)57 (12%)292 (63%)78 (23%)47 (14%)197 (59%)20 (16%)10 (8%)95 (76%)Ref0.83 [0.35; 1.89]1.88 [1.10; 3.32] < 0.001Ref0.89 [0.34; 2.19]2.22 [1.24; 4.15] < 0.001Abbreviation: SCT script concordance tests*Interaction: Trained arm X SCT round 2 = 0.001 Association of clinical factors and validity of SCTs - Non-clinical - Surgery - Medical 44 (9%) 56 (12%) 360 (78%) 29 (9%) 45 (13%) 261 (78%) 15 (12%) 11 (9%) 99 (79%) Ref 0.47 [0.19; 1.16] 0.73 [0.38; 1.46] Ref 0.74 [0.27; 2.02] 0.85 [0.38; 1.96] - 1 - 2 - ≥ 3 184 (40%) 140 (30%) 96 (21%) 127 (38%) 100 (30%) 80 (24%) 57 (46%) 40 (32%) 16 (13%) Ref 0.89 [0.55; 1.44] 0.44 [0.23; 0.81] Ref 1.06 [0.63; 1.79] 0.45 [0.23; 0.85] - Investigation - Treatment - Diagnostic 98 (21%) 57 (12%) 292 (63%) 78 (23%) 47 (14%) 197 (59%) 20 (16%) 10 (8%) 95 (76%) Ref 0.83 [0.35; 1.89] 1.88 [1.10; 3.32] Ref 0.89 [0.34; 2.19] 2.22 [1.24; 4.15] Abbreviation: SCT script concordance tests *Interaction: Trained arm X SCT round 2 = 0.001 Discussion Our prospective randomized study suggests a beneficial effect of training in SCT writing, with a higher rate of valid SCT 2 observed in the trained group compared with the control group. This difference was supported by between-group comparisons, although within-group improvements over time did not reach statistical significance. By investing in faculty development, institutions can create a more effective teaching environment, ultimately benefiting student learning and achievement. A recent systematic review in the health field [20], including 12 studies from 14 countries, highlighted the effectiveness of teacher development programs, both faculty members and students reporting positive changes. Indeed, despite being experts in their fields, health professionals often receive limited training in “how to teach” during their medical studies and their residency [21]. SCT has been recently introduced in French medical universities but remains largely unfamiliar with most physicians, as they did not encounter it during their own exam. C. R Turker was wondering [22]: “How do you know a medical educator is ready to teach?”. In the current study, no physician had previously drafted any SCT. Among the whole SCTs produced, half were initially considered to be of poor quality in terms of construction, with this percentage increasing to approximately two-thirds after evaluation by the expert panel. A similar rate was observed in both arms for SCT 1. The main issues encountered in the SCT writing were: incomplete scenario with missing information, inconsistency between the scenario and the hypotheses proposed in the items, same option proposed in two/three consecutive questions and failure to respect the three-question format in a single SCT. Focusing on the trained arm, the percentage of valid SCTs increased from 34% before training to 41% afterward, although this change did not reach statistical significance. Interestingly, in the control arm, the SCTs produced during the second phase (with only 2 months after the first phase and despite the return of the reference teaching aid) appeared to be of significantly poorer quality than those initially created, with the rate of valid SCTs dropping from 31 to 16%. This suggests that a minimal training is necessary to enhance initial writing skills. To explain this difference, one could hypothesize that an interactive in-person training session allows for better integration — at least in the medium term — of key concepts, whereas memorization after simply reading a PDF document tends to be less effective. A limitation of our study is the limited number of experts in a subset of medical specialties. Gagnon et al. [23] advised a minimum of 15 panelists to achieve reliable score estimates, with little additional benefit from including more than 20 panel members. However, Peyrony et al. [19] showed no significant impact on student results when comparing panels of more than 10 experts versus 15 or 20 experts. By contrast, scores appeared severely compromised when panels were composed of less than 10 members [15, 23]. Based on these findings, we restricted our analyses to panels comprising at least 10 experts, which resulted in the exclusion of approximately 20% of intrinsically validated SCTs. This reflects a key practical limitation of SCT methodology, namely the difficulty in recruiting sufficiently large expert panels across all specialties despite repeated recruitment efforts. It highlights a structural constraint for the broader implementation of SCT-based assessments. Such challenges may limit the feasibility of this approach in routine educational settings or large-scale national examinations. The expected number of participants was not reached either (115 instead of 120). Nevertheless, we had planned for 5% to be lost to follow-up, which in the end corresponded to the number of patients in the study, and therefore had no impact on the theoretical power of the analyses. Another limitation of the study is the difference in the participants recruited between the two arms despite randomization. In the trained arm, there were more first-year post-residency participants, and a higher proportion of non-clinical physicians compared to surgeons. In our study the number of years post-residency was associated with the quality of the SCTs produced; as the number of years increased, the rate of valid SCTs significantly decreased. This may be because first-year participants were more likely to have encountered SCTs during university exams, while those in later years may not have had the same experience. However, in multivariable analysis, the trained group remained a significant factor for valid SCTs, with an interaction between the group and the round of the SCT. Also, the majority of SCTs produced were diagnostic, although the teaching materials and training session included examples of diagnostic, investigative, and therapeutic SCTs without prioritization; future implementations may benefit from requiring a balanced distribution across SCT types to promote greater diversity. Furthermore, the expected percentage of valid SCTs was overestimated, due to absence of reference data on the quality of the SCT production in the literature. Lastly, as the study was carried out on a voluntary basis, unfortunately it was not possible to assess the long-term impact of training on the quality of SCTs. To conclude, this prospective bi-centric study suggested the effectiveness of structured training of faculty members in enhancing the quality of SCTs submissions and raises the question of implementing it to ensure the initial quality of the SCTs and the quality of medical students’ assessment. Further studies would be needed to determine the long-term value of this training, or even its repetition over time.