Do LLMs Give Psychometrically Plausible Responses in Educational Assessments?

Andreas Säuberli Diego Frassinelli Barbara Plank · 2025

Read Do LLMs Give Psychometrically Plausible Responses in Educational Assessments? by Andreas Säuberli Diego Frassinelli Barbara Plank (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-09796.txt.

Introduction

Andreas Säuberli, Diego Frassinelli, and Barbara Plank test whether language models answer school tests like humans do, in a 2025 paper. Real pilot studies with students are slow and costly. If models mimicked student behavior, test makers could pilot items cheaply. Eighteen instruction-tuned models answered multiple-choice items. Subjects were reading, United States history, and economics.

Two classic testing frameworks judged human-likeness. Larger models proved overconfident in their answers. Scaling their randomness made response patterns more human-like. Reading items matched students best among the subjects. Overall correlations with real students stayed weak, though. The authors warn against zero-shot piloting with today's models. Models might still help later in the design cycle. For now, human pilots remain necessary for fair test development.

Catalog record

CREATORAndreas Säuberli Diego Frassinelli Barbara Plank
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-09796

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Do LLMs Give Psychometrically Plausible Responses in Educational Assessments? Andreas Säuberli Diego Frassinelli Barbara Plank Abstract Knowing how test takers answer items in educational assessments is essential for test development, to evaluate item quality, and to improve test validity. However, this process usually requires extensive pilot studies with human participants. If large language models (LLMs) exhibit human-like response behavior to test items, this could open up the possibility of using them as pilot participants to accelerate test development. In this paper, we evaluate the human-likeness or psychometric plausibility of responses from 18 instruction-tuned LLMs with two publicly available datasets of multiple-choice test items across three subjects: reading, U.S. history, and economics. Our methodology builds on two theoretical frameworks from psychometrics which are commonly used in educational assessment, classical test theory and item response theory . The results show that while larger models are excessively confident, their response distributions can be more human-like when calibrated with temperature scaling. In addition, we find that LLMs tend to correlate better with humans in reading comprehension items compared to other subjects. However, the correlations are not very strong overall, indicating that LLMs should not be used for piloting educational assessments in a zero-shot setting. Figure 1: Example item from the NAEP dataset and illustration of our psychometric analyses of LLM responses. We use the first-token probabilities produced by LLMs and analyze how well they correspond to human test taker responses. Specifically, we look at (a) the similarity between LLM and human response distributions, (b) whether items that are difficult for humans are also difficult for LLMs, and (c) how well response probabilities in LLMs match those expected from humans. 1 Introduction Assessing students’ knowledge and skills represents an important part of education: admission to universities, scholarship awards, and even political decisions on education policy are often based on large-scale educational assessments. Developing such high-stakes tests is a long and expensive process involving experts writing and reviewing test items and repeated piloting with hundreds or thousands of participants ( Green 2020 ; Papageorgiou et al. 2021 ) . Therefore, the automation of parts of this process has been a long-standing topic in assessment research and practice ( Haladyna 2013 ; Kurdi et al. 2019 ) . Most recently, large language models (LLMs) have been explored for tasks like item generation or item difficulty prediction ( Attali et al. 2022 ; Yaneva et al. 2024 ; Owan et al. 2023 ; May et al. 2025 ) . The present work explores the possibility of using LLMs as participants of a pilot study in test development. A pilot study involves collecting and analyzing responses by human test takers to identify low-quality items and to measure item characteristics like difficulty. The statistical analysis of item responses most commonly follows one of two psychometric theories, classical test theory (CTT) or item response theory (IRT) ( Chang et al. 2021 ) . For LLMs to be useful models of human test takers, their responses must be human-like when analyzed within those theoretical frameworks – we call this psychometric plausibility . This includes, for example, that items that are difficult for humans should also be difficult for LLMs. We propose an approach to evaluate the psychometric plausibility of LLM response distributions in multiple-choice test items, which is summarized in Figure 1 . Our contributions are two-fold: First, we present methods for assessing the psychometric plausibility of LLM responses with CTT and IRT (Section 3 ). Second, we benchmark the psychometric plausibility of 18 instruction-tuned LLMs across two datasets and three test subjects, showing that none of the models are sufficiently reliable to simulate test takers for piloting (Section 4 ). 2 Related work A growing body of research has studied the use of natural language processing (NLP) for analyzing or evaluating test items. Examples of specific tasks are predicting difficulty ( Yaneva et al. 2024 ) , evaluating answerability or guessability ( Raina et al. 2023 ; Säuberli and Clematide 2024 ) , evaluating the quality of generated items ( Raina and Gales 2022 ; Gorgun and Bulut 2024 ) , or predicting correlations between items ( Hernandez and Nie 2022 ) . Some of these studies used NLP models to simulate test takers: Lalor et al. 2019 and Byrd and Srivastava 2022 used “artificial crowds”, i.e., a large number of models trained on subsampled or partially corrupted data, to simulate test takers at different ability levels. More recently, LLMs have been used. For example, Lu and Wang 2024 and Hayakawa and Saggion 2024 applied prompting techniques to simulate multiple test takers with a single LLM. Park et al. 2024 and Laverghetta Jr et al. 2022 used multiple models to represent a group of test takers, while Liusie et al. 2023 and Zotos et al. 2025 used LLM uncertainty as a proxy for predicting student’s response distributions. Simulating test takers makes it easy to generate large numbers of item responses, which in turn makes statistical item analysis feasible. For example, Liusie et al. 2023 and Hayakawa and Saggion 2024 used CTT to compare item difficulty between humans and LLMs, while Lalor et al. 2019 , Byrd and Srivastava 2022 , and Park et al. 2024 predicted IRT-based item characteristics. Laverghetta Jr et al. 2022 compared both CTT- and IRT-based item difficulty between humans and models. Apart from the application of educational assessment, the human-likeness of predicted response distributions has also been studied in the context of human label variation in tasks with inherent disagreement between annotators ( Plank 2022 ) . Techniques like temperature scaling or fine-tuning on soft labels have been employed to align predictive probabilities with human response distributions ( Baan et al. 2022 ; Chen et al. 2024 ) . Our approach combines ideas from several of these works. Our aim is to measure whether the response probabilities of a single model can be a plausible representative of a single test taker or a group of test takers. In this study, we use temperature scaling to optimize the response distributions, leaving other calibration methods as future work. We draw from both CTT and IRT for evaluation. 3 Psychometric plausibility Psychometrics is concerned with the measurement of unobserved latent variables based on observed responses to test items. Examples of possible latent variables include language proficiency, intelligence, and personality traits like introversion. In educational assessment, two theoretical frameworks are commonly applied: classical test theory (CTT) and item response theory (IRT) . These theories model the ability of test takers based on their observed test scores, but they also allow us to analyze characteristics of test items such as their difficulty or discriminating power ( Livingston 2011 ) . For this reason, CTT and/or IRT is often used in pilot studies during test development in order to identify low-quality items and improve test reliability. In our approach to evaluating psychometric plausibility, we focus on item analysis, i.e., determining item characteristics based on item responses by humans or LLMs. The key idea is that a psychometrically plausible LLM should give responses that are aligned with the characteristics of the items as measured using human responses. In the following subsections, we introduce the relevant basics of CTT and IRT. We then describe how

Read the complete text