Improving MCQ quality using a self-evaluation checklist: A quasi-experimental study Authors: Ayesha Sadiqa, Abid Ashar Published: 2026 ABSTRACT AbstractObjectiveTo improve the quality of multiple-choice questions created by our faculty using a standardized checklist as an intervention.MethodA pre–post quasi-experimental design was employed involving the faculty of basic medical sciences without a control group, to assess the impact of a self-evaluation checklist as an intervention tool on the quality of MCQs. Faculty members were familiarized with the predesigned checklist in an interactive training session. After the intervention, only self-evaluated MCQs were included. MCQ quality was measured through item analysis before and after the intervention. The Chi-square test was used to assess statistical significance.ResultsUsing a checklist did not significantly improve MCQs' reliability, with a shift from “low” to “acceptable” internal exam consistency. There was a mixed response to the checklist in terms of the distribution of the difficulty levels of MCQs, with a slight increase in “difficult items” for anatomy and oral biology, a moderate decrease in “easy items” and a slight rise in “good/acceptable items” for physiology, and little to no change for biochemistry. Similarly, the results indicated that using a checklist decreased the discriminative ability of exam items in most subjects. However, a statistically significant improvement was observed in MCQs, with a proportional increase in the number of MCQs featuring “well-functioning distractors,” with increases of 26.43%, 12.5%, and 46.66% in the distractor efficiency for anatomy, biochemistry, and oral biology, respectively. Moreover, using a checklist improved the academic performance of students, particularly by reducing the number of students who performed “poorly” across most subjects, and significantly increasing the “good” ratings for anatomy, biochemistry, and oral biology, while more students achieved “satisfactory” in physiology.ConclusionUsing a checklist for MCQ construction improved the quality of MCQs with well-functioning distractors, internal consistency of MCQs in exams, but had less impact on other item analysis parameters.Keywords: Checklist, Discrimination index, Distractor efficiency, Multiple-choice questions, Reliability index, Self-evaluation Introduction Assessment is an integral part of learning and teaching, with positive impacts on student learning and performance. Summative assessments of exams are crucial for grading the academic scores of students.1 Undergraduate medical and dental colleges in Pakistan use various tools for summative assessments, where each has a varying level of objectivity. In particular, the single best answer multiple choice question (MCQ) is the most widely used assessment tool due to its high efficiency, objectivity, and ease of grading high-order critical thinking.2 In medical education, the primary goal of competency-based assessment is creating a conceptual framework to provide a valid and consistent measure that accurately reflects a medical student's knowledge, skills, and attitudes.3 The National Board of Medical Examiners (NBME) provide guidelines for developing single best answer MCQs for assessing critical reasoning.4 A vast range of theoretical knowledge can be assessed quickly and efficiently using MCQs. However, to achieve the goal of practical, objective, valid, and reliable assessments of medical undergraduate students, MCQs should be high quality, with high Cronbach's alpha values and high scores for the analysis of each item component, i.e., item difficulty, item discrimination, and distractor efficiency.5 Studies support the idea that only trained faculty members can construct good-quality MCQs because the formulation of MCQs requires the interpretation, integration, synthesis, and analysis of medical knowledge and facts. Thus, the development and formal training of faculty members in the formulation of high-quality and effective MCQs is important, and considerable resources and support are required from institutes.5,6 Poorly constructed MCQs can significantly diminish the effectiveness of assessments and negatively impact the academic performance of students. MCQs with flaws or low quality can increase the probability of item bias, guessing, and cheating by students.7,8 Items that do not adhere to evidence-based guidelines typically reduce the accuracy and validity of exams, thereby considerable impacting students despite seeming harmless.8,9 Previous studies have highlighted a lack of faculty training as the primary cause of flaws in writing MCQs. Most of these studies focused on the effectiveness of workshops and training sessions for improving the quality of MCQs formulated by faculty members.10,11 Thus, the present study aimed to improve the quality of MCQs created by faculty members of the Combined Miltary Hospital Lahore Medical College and Institute of Dentistry (CMH LMC & IOD) by using a standardized checklist as an intervention. The quality of the MCQs was assessed using item analysis. Methods Study design, location, and duration A quasi-experimental quantitative study was performed from August 28, 2023 to December 15, 2024, with the faculty of basic medical sciences at the IOD, CMH LMC, i.e., anatomy, physiology, biochemistry, and oral biology, after obtaining institutional ethical approval (letter 624/ERC/CMH/LMC). The study design was selected based on its feasibility for the principal investigator (PI) and the institute. Moreover, the present study was conducted in partial fulfillment of the requirement for Membership of the College of Physicians and Surgeons in Health Professions Education, and the study design was also accepted for synopsis approval by the degree-awarding institute, i.e., the College of Physicians and Surgeons Pakistan. The pre-intervention and post-intervention phases were compared in this study without a control group. Written informed consent was obtained from each of the participants who were junior faculty members (lecturers or senior lecturers) in basic medical sciences departments. The study hypothesis was formulated as an intervention using a standardized, predesigned checklist adapted from a previously published study12 significantly enhancing the quality of MCQ items (single best answer MCQs) constructed by members of the faculty of basic medical sciences. Number of participants and items The study involved pre- and post-intervention phases. In the pre-intervention phase, 175 and 206 MCQs were tested for quality assessment. Junior faculty members from all basic sciences departments submitted all of the MCQs as study participants. In total, eight junior faculty members (lecturers and senior lecturers) and two faculty members from each department (anatomy, physiology, biochemistry, and oral biology) participated in the study by constructing MCQs and submitting them before and after the intervention. The same first-year Bachelor of Dental Surgery (BDS; batch 2022–23) class was involved in the test run for all of the MCQs in each summative assessment in both study phases. The total numbers of students who attended the summative assessments in each basic medical sciences subject were between 55 and 75, with a class total of 75. Pre-interventional sampling In the pre-intervention phase, only MCQ responses developed and submitted by the basic medical sciences faculty members were included until the end of block I summative assessments. On average, each block comprised 12 complete academic weeks, before taking the summative exam. The college's conventional approach ensured that the quality of all MCQs was checked, i.e., the departmental heads approved the MCQs after receiving the required input from subject experts. Use of intervention The intervention was a predesigned self-evaluation checklist12 with the items shown in Table 1 and it was used by the faculty members during MCQ construction. In the intervention phase, a detailed interactive session was conducted with faculty members who initially drafted the MCQs in the presence of heads of departments. A 2-h training session was conducted separately with each department's faculty members. In each session, the same PI, who was a medical educationist, emphasized the importance of high-quality MCQs and their impact on the academic performance of students, and introduced a predesigned checklist12 (Table 1) as a self-assessment tool for improving the quality of MCQs. Each of the 19 recommendations in the self-evaluated checklist12 was demonstrated with relatable examples to the involved faculty members, and any queries were addressed immediately. All of the MCQ quality parameters, i.e., reliability index (internal consistency), difficulty index, point-biserial correlation (or discrimination index), and distractor efficiency (accuracy of MCQs), were fully explained with examples. All questions from faculty members were fully addressed regarding the execution of the intervention. Table 1Standardized pre-designed12 checklist for self-evaluation of items (One-Best-Answer MCQ).Table 1Instructions For Use: Please self-evaluate each constructed MCQ (item) before submission by selecting ‘yes’ or ‘no’ for each type of recommendation in the checklist and write the MCQ in the given guided format.Sr. No.Recommendations to Construct an ItemDoes your item meet the recommended criteria?YesNo1The scenario should be related to the question.2Avoid negative questions. e.g., Which ONE of the following is NOT a characteristic …3No "EXCEPT" marked question.4Avoid usage of ambiguous (e.g., frequently, often, occasionally) or absolute terms (e.g., almost, never, frequent).5Options do not give a clue to the answer (a word in the stem repeated in the option(s)).6All options are uniform/homogenous e.g., Identification/Diagnosis/Management as a separate entity in each question.7All options must be short and of a similar length. A single long option should not be a correct answer, which gives the clue of being the true answer.8Avoid ‘all of the above' or ‘none of the above' option9Put the options in chronological order; i.e., ascending/descending if numerical, and alphabetical otherwise.10Use questions and avoid phrases.11Should have only one correct answer.12Should fulfill the cover test, i.e., if you cover the options, you can answer the questions (the single most important criteria for a good MCQ).13Options shouldn't be overlapped.14The stem should be clear.15Lead–in should be clear.16The answer should not hinge on another question.17Avoid True or False.18Scenario should be functional19Follow the guided format given below for each MCQ:Guide format for One-Best-Answer MCQSubjectTOS-related Topic or Sub-topicReference (Book, Ed., Chap., Page)Stem/QuestionOptions a. b. c. d.Keya, b, c or dDate of SubmissionDeveloper Signature Standardized pre-designed12 checklist for self-evaluation of items (One-Best-Answer MCQ). All faculty members were assured that they could contact the same PI again if there was any ambiguity or need for clarification regarding the checklist while constructing MCQs. To minimize potential bias introduced by self-evaluation, the heads of departments were involved to ensure that the self-evaluated checklist was used in accordance with the study protocol, where each MCQ submission was first received and handled by them. The trustworthiness of the intervention was addressed through a quasi-experimental study design, reflecting the implications of the intervention (the use of the checklist). In addition, the same PI ran all training sessions to help maintain consistency for all faculty members. Post-interventional sampling All faculty members and heads of departments were told that all faculty members involved in MCQ development must first self-evaluate their developed MCQs by using the standardized predesigned checklist, before submitting the MCQs to their heads of department for inclusion in the post-intervention phase, specifically in the end-block-II and pre-annual summative assessments of first-year BDS students (batch 2022–23). After the intervention phase, all of the item analysis components were again used to assess the quality of the post-intervention phase MCQs based on the responses obtained. Item analysis and scoring criteria At the end of the pre-intervention phase, the item analysis report was generated, including the following parameters that were used to assess the quality of the MCQs: test reliability index, difficulty index, point-biserial correlation for discrimination, distractor efficiency, and students' academic scores. The reference scales for the test reliability index, difficulty index, point-biserial correlation, and distractor efficiency parameters are presented Table 2.13, 14, 15, 16 Table 2Criteria for each Parameter of Item Analysis Report.Table 2Item Analysis ParametersParametersScale1Test Reliability Index13 (Cronbach alpha value)a(0.9–1.00) Very Good and Highly consistentb(0.7–0.80) Good and Acceptablec(0.6–0.70) Acceptabled(<0.60) Low Reliability (needs to be refined)2Difficulty Index14 (P value from hard to easy)a(<0.3) Too Hardb(0.3–0.79) Moderate (Acceptable)c(≥0.8) Too easy3Point–Biserial Correlation15 (For Discrimination)(-1 to 1)a(>0.4) Excellentb(0.25–0.40) Good and Acceptablec(<0.1–0.25) Poord(Negative) Defective item4Distractor Efficiency16a(>0.05) Well-functioning distractorsb(<0.05) Unwell functioned distractors5Students' Academic Scoreaa(≥70%) Excellentb(60–69%) Goodc(50–59%) Satisfactoryd(<50%) UnsatisfactoryaAs per institutional (CMH LMC & IOD) criteria based on affiliated University (National University of Medical Sciences, NUMS, Rawalpindi, Pakistan) guidelines. Criteria for each Parameter of Item Analysis Report. As per institutional (CMH LMC & IOD) criteria based on affiliated University (National University of Medical Sciences, NUMS, Rawalpindi, Pakistan) guidelines. Statistical analysis The responses of all MCQs attempted on the optical mark reader (OMR) sheets by the first year BDS students in the before and after interventional phases were first evaluated using an automated optical mark reader (OMR; Scantron Corporation, USA). The OMR scanner generated the following data analysis reports after scanning all of the OMR response sheets: student achievement report, class frequency distribution, student grades report, test statistics report, student response report for all MCQs, condensed test report, and detailed item analysis report. All of these reports were presented in Excel sheets and generated using Remark (Gravic Inc., USA). All of the quality parameters for MCQs from both phases, i.e., pre-interventional and post-interventional, were statistically compared to assess the intervention's effect. The chi-square test was used to compare the quality parameters for MCQ item analysis before and after the intervention because it is a suitable statistical test for categorical data. The confidence level was 95% and the significance level was set at 0.05 or lower. The educational significance in terms of its useful impact on the MCQ quality index was assessed based on the accuracy achieved, which was supported by the explicit statistical significance (p ≤ 0.05 from chi-square test) to genuinely reflect the pre- and post-intervention difference. Results Reliability index Overall, the findings indicated that there were no statistically significant changes (p > 0.05) in each subject's internal consistency for the assessment items, i.e., MCQs. However, a non-significant (p = 0.774) marginal improvement of 5.17% was observed in the MCQ exam reliability index in the post-intervention phase for the subject of anatomy. Similarly, a non-significant (p = 0.284) moderate increase in the Cronbach's alpha value of 15.0% was observed in physiology after the intervention. No significant change (p = 1.000) in the internal consistency of the exam items for biochemistry was observed after the intervention. In addition, a non-significant (p = 0.502) increase in the internal consistency of the exam items was observed for oral biology after the intervention, with a 12.73% increase in the Cronbach's alpha coefficient (Table 3). Table 3Analysis of the reliability index of MCQs-based assessments before and after intervention.Table 3SubjectsPhasesReliability IndexChi-squareCronbach Alpha coefficientsInterpretation in terms of reliabilitySample sizeMCQs count95% confidence intervalsX2P-valueAnatomyPre-intervention0.58Low55550.4049 0.72400.0820.774Post-intervention0.61Acceptable74600.4720 0.7267PhysiologyPre-intervention0.60Low75400.4586 0.71931.1480.284Post-intervention0.69Acceptable75560.5810 0.7823BiochemistryPre-intervention0.53Low66400.3519 0.67860.0001.000Post-intervention0.53Low70450.3579 0.6746Oral BiologyPre-intervention0.55Low74400.3898 0.68510.4500.502Post-intervention0.62Acceptable62450.4713 0.7435 Analysis of the reliability index of MCQs-based assessments before and after intervention. In particular, a shift from “low” to “acceptable” was evident in the internal consistency of the MCQ exam following intervention, but it was not statistically significant (Table 3). Difficulty factor analysis In the post-intervention phase, a significant difference (p = 0.026) was observed with respect to the distribution of the difficulty levels of assessment items. In the subject of anatomy, there were 66.65% more “easy” items, 20.7% fewer “good and acceptable” items, and 4.71% more “difficult” items. In the same phase, a significant (p = 0.005) psychometric improvement was observed, with a shift in the distribution of item difficulty levels on the physiology assessment, with 50.55% fewer “easy” items, 31.57% more “good and acceptable” items, and 7.15% more “difficult” items. However, for the subject of biochemistry, a non-significant difference (p = 0.428) was found between the pre- and post-intervention phases with respect to the distributions of the difficulty levels of assessment items. Only negligible differences were found in the post-intervention phase, with 18.53% more “easy” items, 8.48% less “good and acceptable” items, and 11.1% more “difficult” items. In addition, a significant difference (p = 0.0007) was observed in the post-intervention phase for the subject of oral biology, with a major increase of 131.12% for “easy” items, a minor decrease of 3.84% for “good and acceptable” items, and a major decrease of 106.30% for “difficult” items (Table 4). Table 4Analysis of MCQs-quality Indices before and after intervention.Table 4Difficulty Factor AnalysisChi-squareSubjectsPhasesItem CountEasy (p ≥ 0.8)Good & Acceptable (P = 0.3–0.79)Difficult (P < 0.3)X2P-valueNN (%)N (%)N (%)AnatomyPre-intervention5511 (20)37 (67.27)7 (12.73)4.9580.026Post-intervention6020 (33.33)32 (53.33)8 (13.33)PhysiologyPre-intervention4013 (32.5)19 (47.5)8 (20)7.6540.005Post-intervention569 (16.07)35 (62.5)12 (21.43)BiochemistryPre-intervention409 (22.5)27 (67.5)4 (10)0.6260.428Post-intervention4512 (26.67)28 (62.22)5 (11.11)Oral BiologyPre-intervention405 (12.5)24 (60)11 (27.5)11.450.0007Post-intervention4513 (28.89)26 (57.78)6 (13.33)Point-Biserial correlationSubjectsPhasesItem CountDefective Item (negative value)Poor (<0.1–0.25)Good & Acceptable (>0.25–0.40)Excellent (>0.40)X2P-valueNN(%)N(%)N(%)N(%)AnatomyPre-intervention553 (5.45)15 (27.27)19 (34.54)18 (32.72)31.6510.0000Post-intervention602 (3.34)37 (61.67)17 (28.34)4 (6.67)PhysiologyPre-intervention404 (10)8 (20)24 (60)4 (10)57.6360.0000Post-intervention566 (10.71)25 (44.64)18 (32.14)7 (12.5)BiochemistryPre-intervention403 (7.5)9 (22.5)12 (30)16 (40)26.4340.0000Post-intervention456 (13.34)21 (46.67)13 (28.89)5 (11.11)Oral BiologyPre-intervention401 (2.5)19 (47.5)17 (42.5)3 (7.5)13.7440.0002Post-intervention454 (8.89)17 (37.78)14 (31.11)10 (22.22)Distractor EfficiencySubjectsPhasesItem CountMCQs with Well-functioning distractors (>0.05)MCQs with Unwell-functioning distractors (<0.05)X2P-valueNN(%)N(%)AnatomyPre-intervention5530 (53.57)26 (46.4)16.7050.000Post-intervention6048 (80)12 (20)PhysiologyPre-intervention4032 (80)8 (20)0.0040.949Post-intervention5645 (80.35)11 (19.6)BiochemistryPre-intervention4027 (67.5)13 (32.5)4.0360.004Post-intervention4536 (80)9 (20)Oral BiologyPre-intervention4016 (40)24 (60)46.8910.000Post-intervention4539 (86.66)6 (13.33)SubjectsPhasesTotal StudentsItem CountStudents' Academic Results (%)Chi-squareUnsatisfactory (<50)Satisfactory (50–59)Good (60–69)Excellent (≥70)X2P-valueNN (%)N (%)N (%)N (%)AnatomyPre-intervention1275524 (18.89)26 (20.47)46 (36.22)31 (24.41)12.5920.0004Post-intervention746010 (13.51)17 (29.97)40 (54.05)7 (9.45)PhysiologyPre-intervention75409 (12)14 (18.67)37 (49.33)15 (20.0)31.1690.0000Post-intervention755621 (28)31 (41.33)19 (25.33)4 (5.33)BiochemistryPre-intervention2054048 (23.30)29 (14.07)44 (21.35)85 (41.46)28.9970.0000Post-intervention70457 (10)16 (22.85)35 (50.0)12 (17.14)Oral BiologyPre-intervention744036 (48.65)28 (37.84)8 (10.81)2 (2.70)54.4440.0000Post-intervention62456 (9.67)19 (30.64)21 (33.87)16 (25.81) Analysis of MCQs-quality Indices before and after intervention. Overall, the results indicate that in the post-intervention phase, the distribution of the difficulty levels of the assessment items improved for most subjects (Table 4). Point–biserial correlation In each subject, significant differences were found in the point-biserial discrimination distributions between the pre- and post-intervention phases. A decline in the discriminatory power was observed for the anatomy exam, with 38.72% less “defective” items, 126% more “poor discriminative” items, 17.95% less “good and acceptable discriminative” items, and 79.61% less “excellent” items in the post-intervention phase. Similarly, in the same phase, reduced discrimination of low and high achievers was observed in physiology, with 7.1% more “defective” items, 123.2% more “poor discriminative” items, 46.43% less “good and acceptable discriminative” items, and 25% more “excellent” items (Table 4). In the post-intervention phase, the negative discrimination power of the exam in biochemistry was observed, with increased percentages of “defective” items (77.86%) and “poorly discriminative” items (107.42%), and reductions in “good and acceptable” (3.7%) and “excellent” (72.23%) items. However, in the same phase, the discriminatory power of the exam in oral biology was relatively improved, with more “defective” (255.6%) items, fewer “poor” (20.46%) items, fewer “good and acceptable” (26.8%) items, and considerably more “excellent” (196.26%) items. Overall, in the post-intervention phase, the discriminative power was reduced for exam items in most subjects (Table 4). Distractor efficiency The results indicated statistically significant (p = 0.000) improvement in the quality of the MCQ exam for the subject of anatomy with respect to the distractor efficiency in the post-intervention phase, with an increase of 49.34% for MCQs with “well-functioning distractors” and decrease of 56.89% for MCQs with “non-functional distractors.” However, a non-significant statistical difference (p = 0.949) was found between the pre-intervention and post-intervention phases in terms of the quality of MCQs with respect to the distractor efficiency distribution in physiology. In the post-intervention phase, a minor increase of only 0.44% was observed for the MCQs with “well-functioning distractors” and a negligible decrease of only 2.0% for “non-functional distractors” (Table 4). A statistically significant difference (p = 0.004) was found after the intervention for biochemistry, with a great increase of 18.52% for MCQs with “well-functioning distractors” and a significant decrease of 38.46% for “non-functional distractors.” A statistically significant (p = 0.000) difference was also observed in the post-intervention phase for the subject of oral biology, with a remarkable increase of 116.65% for MCQs with “well-functioning distractors” and great reduction of 77.78% for MCQs with “non-functional distractors” (Table 4). Overall, in all subjects except physiology, statistically significant improvements were observed in the distractor efficiency distributions in the post-intervention phase, with increase in MCQs with “well-functioning distractors” and decreases in MCQs with “non-functional distractors” (Table 4). Students’ academic results Significant differences were observed between the pre-intervention and post-intervention distributions of academic grades across all subjects. In particular, in the subject of anatomy, declines were observed of 28.48% in the “unsatisfactory” category of academic results and 61% in the “excellent” category. Improvements of 46.41% and 49.22% were observed in the “satisfactory” and “good” academic results categories, respectively, in the post-intervention phase. However, in physiology, increases were found of 133.33% in the “unsatisfactory” category and 121.37% in the “satisfactory” category in the post-intervention phase. In addition, decreases of 48.65% and 73.35% were observed in the “good” and “excellent” categories of academic results, respectively, in the post-intervention phase (Table 4). More promising improvements in the academic performance of students were observed in the post-intervention phase for biochemistry. A remarkable decrease of 57.08% was observed in the “unsatisfactory” category of academic results, whereas increases of 62.40% and 134.19% were found in the “satisfactory” and “good” categories, respectively. However, a decrease of 58.65% was also observed in the “excellent” results category in the same phase. Similarly, for the subject of oral biology, improvements were found of 213.32% in the “good” category and 855% in the “excellent” category of academic performance in the post-intervention phase, as well as significant decreases of 80.12% and 19.02% in the “unsatisfactory” and “satisfactory” categories, respectively (Table 4). Overall, in the post-intervention phase, the academic performance of students improved, mainly in biochemistry, oral biology, and anatomy (Table 4). Discussion In the present study, during the post-intervention phase, significant average increases were found in the distractor efficiency for MCQ responses, and improvements in the average Cronbach's alpha scores. Similarly, a study conducted in Bhutan provided workshop training for 2 h as an educational intervention to enhance MCQ construction by faculty members of a postgraduate medical college, and an increase was found in the average difficulty factor, as well as decreases in the discrimination index and distractor efficiency.17 Similarly, a pilot study in the USA provided a 30-min PowerPoint presentation as an intervention. They compared the quality of the single best answer MCQs created by an intervention group with those developed by a non-intervention group. The findings indicated a significant increase in the MCQ quality score for the faculty intervention group compared with only a fractional improvement in the non-intervention group.18 A study in KSA found that long-term faculty development programs could improve the quality of MCQ item writing by medical faculty. The study included items from three courses (respiratory, cardiovascular, and renal) and compared the differences before and after a training workshop. Training had a positive impact, with a significant improvement in the difficulty index values, increase in the proportion of MCQs with higher discrimination indices, and decrease in the percentage of easy MCQs from 36.7% to 22.5%. Moreover, consistent with our results, reductions were found in MCQs with non-functioning distractors and item-writing flaws.12 A study conducted at Western New England University in the USA found that a 1.5-h seminar for faculty members was highly beneficial for enhancing their skills in constructing MCQs. They found a significant improvement in the MCQ quality, where the average score increased from 16.2 ± 2.4 before the seminar to 18.1 ± 1.5 after the seminar, with an improvement of 11.73% (p < 0.05). Similarly, another study observed an overall enhancement in the reliability of items in basic medical sciences from “low” to “acceptable,” although this improvement was not statistically significant.19 A longitudinal study in KSA provided a faculty development program weekly for three consecutive years. They observed a statistically significant overall improvement in MCQ quality each year, particularly an increase in the distractor efficiency percentage, where 19.52% of the MCQs had no non-functional distractors at the end of the first year, 38.24% at the end of the second year, and 116% at the end of the third year. In addition, Cronbach's alpha increased from 0.51 at the end of the first year to 0.77 at the end of the second year and 0.84 at the end of the third year. It should be noted that the study involved faculty from clinical departments at an undergraduate medical college.20 Another study with similar objectives found a major improvement in the average scores following intervention training for faculty members. In the present study, a minor increase was found in the “good and excellent” category of scores and a moderate decrease in the “unsatisfactory” category of scores across all basic medical sciences subjects after the intervention. An Indian study found a significant increase in the MCQs with “good and acceptable” difficulty after training, whereas a negligible reduction was determined in the same category in the present study. In terms of the discrimination index, the Indian study found a 10% decrease in the “acceptable” category of MCQs after training, whereas a decrease of 46.56% was determined in the present study. The Indian study found a 64% increase in MCQs with zero non-functioning distractors after training. However, in the present study, a higher increase of 85.94% was found in MCQs with zero non-functioning distractors post-intervention, and this difference was also significant.21 An American study with a similar design utilized YouTube video lectures and PowerPoint voice-over lessons as training tools to develop MCQs. Consistent with our results, they reported significantly higher mean scores in the post-intervention phase compared with the pre-intervention phase.22 A quasi-experimental study conducted at Liaquat National Hospital in Karachi assessed faculty self-evaluation before and after a workshop on creating high-quality assessment items. The results indicated a significant reduction in item writing flaws, including grammatical issues and the use of absolute terms. Improvements were also noted in MCQ development, including logical sequencing and functional distractors. Similarly, in the present study, an overall increase was found in item difficulty, as well as a better discrimination index and more functional distractors following the intervention.23 A study conducted by the College of Medicine at King Saud University in Riyadh found that longitudinal faculty training improved the quality of exam questions. Annual faculty development workshops led to significant yearly improvements in item analysis, with increases in “good and acceptable” and “difficult” items from 2012 to 2015, and a decrease in items with a low discriminating index (≤0.15). An increase was also found in items with functional distractors.24 A 3–4-h workshop was held at the National University of Medical Sciences in Rawalpindi, Pakistan, to train the medical faculty in developing high-quality test items. The workshop notably enhanced item writing skills, the ability to identify item writing flaws, and the difficulty index, as shown by differences between the pre- and post-test results. Compared with this previous study, there was an increase in the reliability index for items and a significant enhancement of the distractor efficiency in the present study. These improvements were attributed to the use of a self-assessment predesigned questionnaire as an intervention tool by medical faculty members.25 A recent (2025) systematic review and meta-analysis showed that continuing developmental strategies, such as the self-evaluated checklist used in the present study, were more consistently associated with sustainable improvement in MCQ quality. By contrast, less promising benefits were achieved through one-time faculty training sessions, such as workshops, aimed at improving the quality of items in medical education.26 Another recent study using a quasi-experimental design involving 133 medical faculty members in long-term training programs in Pakistan found that that incorporating various teaching methods, peer review techniques, individual or team-based task assignment, and providing constructive and timely feedback significantly enhanced their skills in effective MCQ writing.27 Furthermore, the present study employed a quasi-experimental design without a control group, and previous research has highlighted several factors that should be considered potential threats to the internal validity of MCQ-based exams, especially when assessing the use of an intervention. These factors include maturation, testing effects, cohort progression, and contextual changes, which can confound the interpretation of results.28,29 Maturation could have been a confounder in the present study throughout the year when sampling occurred because faculty members may have professionally developed their skills in item writing independently of the intervention (i.e., the use of a checklist). Similarly, testing effects can occur when students are repeatedly exposed to assessment cycles, which can train them to approach MCQs more effectively.29 Therefore, the improved results obtained by students in the post-intervention phase might not have been solely due to the improved quality of MCQs, thereby confounding the impact of the checklist. Similarly, cohort progression could have been another confounder because faculty cohorts differ in terms of their academic experience and educational background, which could also have weakened the efficacy of the checklist as an intervention tool. In addition, contextual changes,29,30 including routine faculty development programs and reforms implemented by the college assessment cell, may have modified and contributed to the results during the data collection phase. Several possible reasons may explain the low discrimination index obtained during the post-intervention phase in this study, including mistakes such as incorrect keying, multiple correct answers, technical issues, or even typographical errors. Heavy workloads, multitasking, tight deadlines for paper submissions by the exam cell, and long working hours might also have been underlying causes because an increased workload consistently leads to lower accuracy in tasks.31 In previous studies, various complementary strategies have been recommended in addition to using a checklist to achieve desirable discrimination, such as blueprinting, cognitive-level alignment, and structured peer review.32 A well-designed work plan for faculty can improve their ability to follow the table of specifications and provide them with suitable time for self and peer reviews,33 and most importantly, to follow the key rule of rechecking for any errors on their own before the final submission of MCQs. Limitations The present study lacked a true control group, limiting causal inference about the relationship between the intervention and MCQ quality. The findings may also not be widely applicable because the study was conducted in a single institute. The intervention was effective but the self-evaluation control had low reliability and bias. Variations in the number of MCQs and the number of students across subjects and phases may affect comparability. Relying on a single institute for the study group limits the generalizability of the findings to other contexts. Conclusion The present study found a non-significant change in the internal consistency of the MCQ exam for each subject after the intervention, with only a slight improvement from “low” to “acceptable” in the MCQ exam's reliability index. A mixed, unspecified effect of the intervention was observed on the distribution of the difficulty levels for MCQ exams. The intervention negatively affected the reliability and validity of the exams by explicitly reducing the discriminative power of exam items. However, it increased the MCQ accuracy, with a proportional rise in the number of MCQs that had well-functioning distractors. In addition, the intervention enhanced the academic performance of students, specifically by reducing the number of students who performed unsatisfactorily across most subjects. Authors contributions AS designed the research, conducted the data analysis, wrote the manuscript, and reviewed it. AA was involved in the study design, interpretation of results, critical review, and final approval of the manuscript before submission. Both authors have read and agreed to the published version of the manuscript. They have critically reviewed and approved the final draft and are responsible for the manuscript’s content and similarity index. All authors have critically reviewed and approved the final draft and are responsible for the content and similarity index of the manuscript. Source of funding This research did not receive any specific grant from funding agencies in the public, commercial, or not for-profit sectors. Ethical approval Our study is approved by the Ethical review committee of CMH Lahore Medical college and institie of Dentistry with reference number 624/ERC/CMH/LMC dated 25th August 2023. Conflict of interest The authors have no conflicts of interest to declare. Footnotes Supplementary data to this article can be found online at https://doi.org/10.1016/j.jtumed.2026.05.006. Supplementary data The following is the Supplementary data to this article: Multimedia component 1 References Associated Data Supplementary Materials Multimedia component 1