Estimating LLM Grading Ability and Response Difficulty in Automatic Short Answer Grading via Item Response Theory
Longwei Cong, Sonja Hahn, Sebastian Gombert, Leon Camus, Hendrik Drachsler, Ulf Kroehne · 2026
Read Estimating LLM Grading Ability and Response Difficulty in Automatic Short Answer Grading via Item Response Theory by Longwei Cong, Sonja Hahn, Sebastian Gombert, Leon Camus, Hendrik Drachsler, Ulf Kroehne (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-00238.txt.
Introduction
Longwei Cong and 5 co-authors bring a tool from education research, item response theory, to the grading of short student answers by language models. Usual scores such as F1 hide where a grader struggles. Their framework instead models each grade as a contest between the grader's ability and the answer's difficulty.
Applied to seventeen open-weight models on two benchmarks, the method shows that models with alike overall scores can fade at very different rates as answers get harder. Errors on hard answers pile up on one middle label, for partly correct but incomplete work. Hard answers also tend to match the reference answer weakly and sit far from other answers in meaning.
Catalog record
| CREATOR | Longwei Cong, Sonja Hahn, Sebastian Gombert, Leon Camus, Hendrik Drachsler, Ulf Kroehne |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.00238
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Estimating LLM Grading Ability and Response Difficulty in Automatic Short Answer Grading via Item Response Theory Longwei Cong, Sonja Hahn, Sebastian Gombert, Leon Camus, Hendrik Drachsler, Ulf Kroehne Abstract Automated short answer grading (ASAG) with large language models (LLMs) is commonly evaluated with aggregate metrics such as macro-F1 and Cohen’s kappa. However, these metrics provide limited insight into how grading performance varies across student responses of differing grading difficulty. We introduce an evaluation framework for LLM-based ASAG based on item response theory (IRT), which models grading correctness as a function of latent grader ability and response grading difficulty. This formulation enables response-level analysis of where LLM graders succeed or fail and reveals robustness differences that are not visible from aggregate scores alone. We apply the framework to 17 open-weight LLMs on the SciEntsBank and Beetle benchmarks. The results show that even models with similar overall performance differ substantially in how sharply their grading accuracy declines as response difficulty increases. In addition, confusion patterns show that errors on difficult responses concentrate disproportionately on the partially_correct_incomplete label, indicating a tendency toward intermediate-label collapse under ambiguity. To characterize difficult responses, we further analyze semantic and linguistic correlates of estimated difficulty. Across both datasets, higher difficulty is associated with weaker semantic alignment to the reference answer, stronger contradiction signals, and greater semantic isolation in embedding space. Overall, these results show that item response theory offers a useful framework for evaluating LLM-based ASAG beyond aggregate performance measures. 1 Introduction Automatic short answer grading (ASAG) is the task of using computational methods to assess short free-text student responses to content questions 4 . Earlier ASAG approaches relied on handcrafted features and classical machine learning models 31 ; 25 , followed by neural architectures and encoder-based pre-trained language models (PLMs) such as BERT 14 ; 5 . More recently, generative large language models (LLMs) have attracted increasing attention for ASAG because of their strong language understanding and multi-step reasoning capabilities 9 ; 12 ; 11 . Despite this progress, evaluation in ASAG remains dominated by aggregate metrics such as accuracy, macro-F1, and agreement coefficients 3 . While useful for summarizing overall system performance, these measures provide only a coarse view of grading behavior and reveal little about how performance varies across responses of differing grading difficulty 28 ; 7 . This limitation is particularly important in ASAG, where student answers are often brief, diverse, and semantically ambiguous 4 . Consequently, two models with similar overall scores may nevertheless differ substantially in how well they handle genuinely difficult responses. This motivates a more measurement-oriented perspective on LLM-based ASAG. In particular, a useful evaluation framework should help characterize which student responses are difficult to grade and whether LLM graders with similar average performance are equally robust across the difficulty spectrum. Existing ASAG research has paid relatively little attention to these questions at the response level 17 . To address this gap, we draw on item response theory (IRT), a psychometric framework that models observed outcomes as a function of latent ability and item difficulty. Prior work has shown that IRT can support more informative evaluation in NLP when benchmark instances vary substantially in difficulty ( 21 ; 20 ) . We argue that IRT is well suited to ASAG because it separates latent grader ability from response difficulty, enabling a more fine-grained analysis of grading behavior than aggregate metrics alone. In this work, we apply an IRT-based framework to LLM-based ASAG to study grading performance at the level of individual student responses. Specifically, we make the following contributions: • We introduce an IRT-based perspective for evaluating LLM-based ASAG, estimating latent grader abilities and response grading difficulties across benchmark datasets and multiple LLM graders. • We analyze how grading performance changes across response difficulty levels and show that LLM graders differ substantially in their robustness as difficulty increases. • We examine semantic and linguistic correlates of response difficulty and find that difficult responses are consistently associated with weaker semantic alignment to the reference, stronger contradiction-related signals, and greater embedding-space isolation. • We provide error analyses showing that, on high-difficulty responses, misclassifications are not randomly distributed but increasingly concentrate in the partially_correct_incomplete category. 2 Background 2.1 Evaluation Methods in LLM-based ASAG Evaluation in ASAG has traditionally focused on benchmark performance over labeled response sets, typically reported using aggregate metrics such as accuracy, F1, Cohen’s kappa, RMSE, and correlation-based measures 17 ; 4 ; 31 . Recent LLM-based ASAG studies largely follow this paradigm, assessing LLM graders through overall label agreement or classification performance 11 ; 12 ; 9 . Yet recent research on LLM evaluation suggests that aggregate benchmark scores alone may provide only a limited view of model capability 35 ; 22 . Prior work shows that benchmark-level averages can obscure substantial variation in item properties such as difficulty and discriminability, while offering limited separability among strong models 21 ; 20 . Similarly, recent work on LLM-as-a-Judge argues that LLM-based evaluators should themselves be evaluated systematically, with reliability, bias, and robustness as central concerns 16 . Within ASAG, however, the response-level determinants of grading difficulty remain underexplored, particularly for LLM-based grading. Understanding which properties of student answers make them difficult to grade is important both for interpretability and for deployment, especially in human-in-the-loop settings where difficulty estimates could guide targeted review and improve grading efficiency. Although recent work has examined confidence and uncertainty in ASAG 13 ; 2 ; 8 , these signals are typically model-derived and thus only indirectly informative about the characteristics of the response itself. Analyzing difficulty as a property of the student response instead offers a more interpretable and educationally grounded perspective on the failure modes of LLM-based grading. 2.2 Item Response Theory Item Response Theory (IRT) provides a principled framework by disentangling latent responder ability from item properties ( 32 ) . In the simplest Rasch formulation, the probability that responder m correctly answers item n is modeled as P(y_{mn}=1\mid\theta_{m},b_{n})=\sigma(\theta_{m}-b_{n}), where \theta_{m} denotes the latent ability of responder m , b_{n} the difficulty of item n , and \sigma(\cdot) the logistic sigmoid. More flexible variants, such as 2PL models, additionally model item discrimination, allowing some items to be more informative than others 18 . Recent work has begun to apply IRT to LLM evaluation in several ways, including diagnosing the reliability of LLM-as-a-judge systems ( 7 ) and modeling latent LLM abilities together with query difficulty and discrimination for routing and performance prediction ( 6 ) . In educational technology, related work has also shown that IRT can be integrated with LLM-based modeling to predict or align item difficulty in assessment settings ( 29 ) . Collectively, these developments suggest that psychometric modeling offers a robust lens for analyzing LLM behavior through a measurement-grounded perspective. 3 Method 3.1 Datasets and LLMs We evaluate our approach on the five-way SciEntsBank and Beetle datasets 10 , two widely used benchmarks for ASAG. Released as part of the SemEval student-response analysis task, both datasets contain science-domain questions, reference answers, and student responses annotated by experts. An important property of these benchmarks is their fine-grained five-way label space, which enables analysis beyond binary correctness. Each instance consists of a question, a single reference answer, a student response, and a