MCQ Difficulty Prediction via Modeling Learner Heterogeneity Using Data-Driven Cognitive Profiling

Dhriti Krishnan, Jaromir Savelka · 2026

Read MCQ Difficulty Prediction via Modeling Learner Heterogeneity Using Data-Driven Cognitive Profiling by Dhriti Krishnan, Jaromir Savelka (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-16290.txt.

Introduction

Krishnan and Savelka predict multiple-choice question hardness by tracking learner variety instead of taking one single-peak ability spread. Using EEDI student logs, they find habit types by latent class analysis, then tune a language model to mimic response spreads per type. Those signals join topic context in a Ridge fit to guess the item response theory value. Five-fold checks beat a recent baseline, cutting mean squared error from 0.367 to 0.274 and lifting R-squared from 0.525 to 0.686, with clear types showing why items prove hard.

Catalog record

CREATORDhriti Krishnan, Jaromir Savelka
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17997
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.16290

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MCQ Difficulty Prediction via Modeling Learner Heterogeneity Using Data-Driven Cognitive Profiling Dhriti Krishnan, Jaromir Savelka Abstract Predicting the difficulty of multiple-choice questions (MCQs) is important for effective assessment, yet current methods typically assume a unimodal student ability distribution, overlooking the heterogeneous nature of student misconceptions. We propose a persona-driven framework that replaces theoretical ability sampling with data-driven cognitive profiling. Using student interactions from the EEDI dataset, we identify behavioral personas via latent class analysis (LCA), then condition a large language model (LLM) to simulate response distributions for each persona. These signals are aggregated with topic context and fed into a Ridge Regression model to predict the item response theory (IRT) difficulty parameter. With five-fold cross-validation, our method improves over a recent baseline (MSE: 0.367 → 0.274 0.367\rightarrow 0.274 ; R 2 R^{2} : 0.525 → 0.686 0.525\rightarrow 0.686 ). The discovered personas are interpretable and offer insights into why items are difficult, with potential applications to diagnostic assessment design. 1 Introduction Accurate estimates of item difficulty are fundamental to effective testing, item calibration, and curriculum design [ 23 ] . Traditional approaches based on item response theory (IRT) [ 19 , 6 ] require substantial pretesting data, meaning newly authored items, particularly multiple-choice questions (MCQs) [ 5 ] , cannot be deployed until sufficient student responses have been collected. Recent work has explored predicting difficulty without pretesting, by inferring it from the question text [ 13 , 9 ] , learned semantic representations [ 4 , 8 ] , or LLM-simulated student responses [ 10 , 18 ] . However, these approaches typically assume a unimodal student population, overlooking heterogeneity in learner behavior. Our central hypothesis is that student errors are not random but reflect systematic patterns tied to distinct cognitive profiles. Learners with comparable overall performance may differ in the types of mistakes they make and the concepts they fail to apply [ 16 ] , and student knowledge is better characterized by multiple knowledge components than a single ability parameter [ 15 ] . If this heterogeneity is ignored, simulated response distributions, and the difficulty estimates derived from them, may poorly reflect how diverse learner populations interact with assessment items. Our contributions presented in this work are as follows: 1. We propose a novel persona-driven framework that (i) uses Latent Class Analysis (LCA) on student response data to discover interpretable behavioral personas that capture how different types of learners systematically differ in their knowledge and misconceptions; (ii) conditions an LLM on these personas to simulate response distributions, replacing the standard assumption of a unimodal ability distribution. 2. We demonstrate improved prediction of IRT difficulty parameters over a recent state-of-the-art baseline [ 8 ] on the EEDI dataset using five-fold cross-validation (MSE: 0.471\rightarrow 0.274 ; R^{2} : 0.525\rightarrow 0.686 ), while producing interpretable personas with potential applications to diagnostic assessment design. 2 Related Work LLMs as Simulated Students He-Yueya et al. [ 10 ] introduced psychometric alignment as a metric for quantifying how well LLMs reflect human knowledge distributions. Park et al. [ 18 ] demonstrated that LLMs can exhibit graded proficiency levels in a zero-shot setting without fine-tuning, suggesting their potential as proxies for learners at different ability levels. Building on this, Hu and Collier [ 12 ] showed that conditioning LLMs on persona variables systematically alters response patterns, while Yuan et al. [ 24 ] argued that valid student simulation requires explicitly specifying the epistemic scope of a simulated learner’s knowledge. In our work, we extend this line of research by deriving behavioral personas directly from observed student response patterns to ensure that the simulated students reflect the actual capabilities and misconceptions found in real-world classrooms. Modeling Learner Ability LLM-based difficulty prediction has gained attention through the BEA 2024 Shared Task [ 22 ] , where participating systems showed competitive zero-shot performance but generally struggled to model distractor plausibility. Subsequent work has focused on how to represent variation in student ability within simulation pipelines. Feng et al. [ 8 ] sampled scalar knowledge levels from a standard normal distribution, while Scarlatos et al. [ 20 ] used direct preference optimization to align simulated students with IRT ability parameters. These approaches implicitly assume that student errors vary in frequency rather than in kind. Our framework relaxes this assumption by conditioning on discrete cognitive profiles that capture qualitative differences in error patterns across learner subpopulations. The Homogeneity Gap A recurring finding across literature is that LLMs tend to produce expert-biased responses, generating unrealistically accurate answers rather than reflecting the error patterns of typical learners [ 10 ] . Feng et al. [ 7 ] observed a similar pattern in distractor generation: LLMs produce mathematically valid distractors but failed to target the misconceptions that real students actually held. Taken together, these results point to a fundamental mismatch between how LLMs reason and how students err. Our work tackles this by grounding simulation in cognitive personas discovered from actual student response data. 3 Methodology Step 2 LLM Simulation Claude 3.7 Sonnet Step 3 Feature Extraction K\times 4 probs + stats Step 4 Ridge Regression predict \beta_{i} Step 1 LCA Clustering 5 personas Step 0 2PL-IRT ground truth EEDI Dataset \hat{\beta} personas prob. dists features ground truth \beta Figure 1: The Proposed Pipeline. LCA discovers 5 learner personas from student response data (Step 1). An LLM simulates per-persona response distributions (Step 2), which are aggregated into features (Step 3) and used to predict IRT difficulty via Ridge Regression (Step 4). Ground truth is estimated independently via 2PL-IRT (Step 0). We propose a four-stage framework for estimating MCQ difficulty that combines psychometric profiling with LLM-based student simulation: (1) data pre-processing and ground truth estimation, (2) psychometric profiling, (3) profile-conditioned response simulation, (4) predictive modeling of difficulty. 3.1 Data Pre-processing and Ground Truth Estimation We use tasks 3 & 4 of the EEDI Dataset [ 21 ] , which contains 948 mathematics MCQs answered by 4,918 students. Each of the 1,382,728 interaction records contains a student ID, question ID, selected answer option, and binary correctness indicator. Questions are provided as images; we extract their textual content using Tesseract OCR applied to grayscale-converted images. Questions where OCR indicated image-only content were excluded. We partition the dataset into two disjoint subsets: a profiling set for discovering student personas, and an estimation set for ground truth estimation and difficulty prediction. For the profiling set, we apply dense core filtering, retaining questions with \geq 50 responses and students with \geq 10 attempts. For the estimation set, we retain questions with \geq 20 responses, yielding 900 items. We fit a 2PL-IRT model on this subset using py-irt [ 17 ] , estimating item difficulty ( \beta ) and discrimination ( \alpha ) parameters that serve as ground truth targets. 3.2 Psychometric Profiling Figure 2: Model selection for psychometric profiling. Bayesian Information Criterion (BIC) and Akaike Information Criterion (AIC) for Latent Class models with varying numbers of latent classes. BIC reaches a global minimum at k=5 , indicating the optimal number of learner profiles. We partition students into latent performance classes using LCA with a binary measurement model (StepMix). Each student is assigned to a latent class c_{u}\in\{1,\ldots,k\} based on maximum posterior probability, and classes are ordered by mean accuracy.

Read the complete text