Using randomization to compare AI and expert-generated formative assessment questions in medical education

Akshat Kumar, Katie Stinson, Litao Wang, Elizabeth Nugent, Yongsu Lee, Mark Hormann, LaTanya Jones Love, Toufeeq Ahmed Syed · 2026

Read Using randomization to compare AI and expert-generated formative assessment questions in medical education by Akshat Kumar, Katie Stinson, Litao Wang, Elizabeth Nugent, Yongsu Lee, Mark Hormann, LaTanya Jones Love, Toufeeq Ahmed Syed (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/pmc-13159578.txt.

Description

Akshat Kumar and seven co-authors compare AI-made and expert-made test questions for medical students in a 2026 paper. AI tools can now write complex multiple-choice questions. Judging their quality against expert work is hard with old methods. The team used random assignment for a fair test.

Students got one AI or expert question each day for four weeks. Scores did not differ between the two question sets. Student views of the two sets also matched closely. But 53 percent of AI questions were rated easy or very easy. Only 31 percent of expert questions got that rating. The authors say new checks are needed beyond student feel.

Catalog record

CREATORAkshat Kumar, Katie Stinson, Litao Wang, Elizabeth Nugent, Yongsu Lee, Mark Hormann, LaTanya Jones Love, Toufeeq Ahmed Syed
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE13 KB
VERSION1.0
LICENSECC BY 4.0
RIGHTSCC BY
SOURCEEurope PMC

Source: https://europepmc.org/articles/PMC13159578

Browse the full library · About the Tower of Babel