Do LLMs Make Mistakes Like Students? Exploring Natural Alignment between Language Models and Human Error Patterns

Naiming Liu, Shashank Sonkar, Richard G. Baraniuk · 2025

Read Do LLMs Make Mistakes Like Students? Exploring Natural Alignment between Language Models and Human Error Patterns by Naiming Liu, Shashank Sonkar, Richard G. Baraniuk (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-15140.txt.

Description

Naiming Liu and co-authors ask whether large language models pick the same wrong answers that students pick. On multiple-choice tests, the wrong options are built to lure students who hold common misconceptions. Their 2025 study compares model likelihoods with real student response data across a set of questions.

The team finds moderate links between the distractors students choose most and those models rate most likely. When a model does err, it tends to land on the same trap answer that misleads the most students. The pattern hints that models absorb human-like error habits from their training text. Education researchers get a new lens for judging what model mistakes mean.

Catalog record

CREATORNaiming Liu, Shashank Sonkar, Richard G. Baraniuk
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-15140

Browse the full library · About the Tower of Babel