Response-free item difficulty modelling for multiple-choice items with fine-tuned transformers: Component-wise representation and multi-task learning

Jan Netík, Patrícia Martinková · 2026

Read Response-free item difficulty modelling for multiple-choice items with fine-tuned transformers: Component-wise representation and multi-task learning by Jan Netík, Patrícia Martinková (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-16991.txt.

Introduction

Netík and Martinková estimate multiple-choice item difficulty before any answers exist, tuning a transformer on wording instead of hand-made features. They target reading items, whose difficulty rests on inference across passage, question, options, while the base model sees one undifferentiated sequence trained on difficulty. Two extensions beat that base model at the smallest size: a component-wise variant encoding parts apart and a multi-task variant adding a question-answer task. The multi-task variant wins on every metric, and component-wise encoding wins on rank ordering.

Catalog record

CREATORJan Netík, Patrícia Martinková
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43762
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.16991

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Response-free item difficulty modelling for multiple-choice items with fine-tuned transformers: Component-wise representation and multi-task learning Jan Netík, Patrícia Martinková Abstract Response-free IDM promises to reduce reliance on response-based calibration but is intrinsically difficult on reading-comprehension multiple-choice items, where difficulty depends on inferential demands across wording components. Whereas most existing approaches extract item-text features and pass them to a separate statistical or machine-learning model, we fine-tune transformer encoders end-to-end on the item wording, eliminating the manual feature engineering and preprocessing that discards information. Moreover, two extensions to this joint-encoding approach are proposed: a component-wise variant that encodes wording components separately through a shared encoder, and a multi-task variant that retains joint encoding and adds an auxiliary MCQA objective on the shared encoder. Each method is evaluated under a Monte Carlo subsampling design at three training-set sizes on a held-out test set. We find that joint encoding is a viable end-to-end alternative to feature-engineering pipelines; while the component-wise variant shows no detectable benefit, consistent with self-attention already harvesting the cross-component signal, the multi-task variant delivers significant paired improvements in the smallest-sample regime. Transformer fine-tuning, especially if regularised by a suitable auxiliary task, recovers a substantial share of the wording-derivable signal at training-set sizes typical of applied measurement. The framework provides a customisable interface for psychometrically motivated extensions. Keywords: item difficulty modelling; transformers; fine-tuning; multiple-choice items; reading comprehension; Rasch model 1 Introduction Accurate estimation of item difficulty is a cornerstone of psychometric modelling, particularly in high-stakes educational assessment. Reliable difficulty estimates are essential for minimising measurement error at targeted ability levels, aligning the difficulty of multiple test forms during construction, and providing test-takers with optimally selected items in computerized adaptive testing. In many practical settings, however, conventional estimation from a sufficient number of response patterns is unavailable: newly written items may not yet have been administered, pre-testing can expose secure items, the target population may be too small or difficult to recruit, and automatically generated item pools may need provisional difficulty estimates before response data are available. Expert judgments can partly fill this gap, but they are not sufficiently reliable as a sole source of information ( 1 ; 25 ) . This creates a need for methods that predict item difficulty directly from the item stimulus material, that is, from the wording available before administration. We refer to this pre-administration setting as response-free IDM ( IDM ): item difficulty is inferred from the item’s wording without requiring response patterns ( 27 ) . Three lines of work approach the task (see 20 ; 1 , for recent systematic reviews) . The oldest manually defines and derives so-called item text features (readability indices, syntactic depth, lexical complexity, or, as in 25 , , averaged static word embeddings) and feeds them to a statistical or machine-learning model, typically after demanding preprocessing (stopword removal, lemmatisation; 25 ; 9 ) that discards a substantial morphosyntactic layer of the wording. The pipeline is highly interpretable but replaces the wording’s raw content with a hand-engineered feature representation, requiring substantial preprocessing and theory-based feature-design effort. The second line, rooted in explanatory IRT ( IRT ), integrates fixed item features directly into the IRT likelihood: in the linear logistic test model (LLTM; 8 ) the item text features are hand-coded as in the first approach and their “weights” are estimated, while also leveraging the response-patterns simultaneously. Some works extended the same framework using frozen contextual representations from a pre-trained transformer (specifically, the encoder-only version called “ BERT ”, BERT ; for the original architecture see 28 ) as feature vectors for cloze -style 1 1 1 The cloze task, introduced by 26 is, in effect, identical to the MLM ( MLM ) objective on which BERT ( BERT )-family encoders are pre-trained ( 6 ) : a token is hidden in context and the system must recover it, exactly as a test-taker would. items ( 17 ; 32 ) ; the LLTM ( LLTM ) specification is retained, but the hand-coded features are replaced by transformer outputs. A third line, which we pursue here, takes the natural next step: rather than freezing the encoder, we fine-tune it end-to-end against the difficulty target, so the representations are adapted to the IDM task rather than borrowed from a general-purpose pre-training objective. The fine-tuning recipe is well established in the broader NLP ( NLP ) community ( 13 ; 6 ) , and has been applied to MCQ ( MCQ ) IDM in previous work ( 18 ) . 2 reported an independent adaptation in a different setting (short factoid-style items: computer-science trivia and K-12 mathematics problems, with limited passage context). The line has not been substantially extended for reading-comprehension MCQ since. Response-free IDM for reading-comprehension MCQ is intrinsically hard. Difficulty depends on inferential demands across passage, question, and options jointly rather than on isolated lexical features, an empirical regularity that classical psychometric cognitive-component models of reading-comprehension items have characterised in detail ( 7 ; 12 ; 19 ) . Transformer fine-tuning without further modification thus leaves substantial headroom, unless one has access to large, high-quality datasets with diverse items (that are both content-diverse and combinatorially “rich”, i.e., featuring multiple questions and options per passage), along with precise difficulty estimates ( 18 ) . The recent shared task on automated prediction of item difficulty and item response time ( 33 ) for medical MCQ illustrates the regime: despite substantial model complexity across submissions, the best entries surpassed the so-called “dummy” regressor (a model that predicts the mean difficulty of the training set for every item) only marginally. The same small-sample pressure motivates recent work that integrates response-free predictions with response-based calibration as informative priors to reduce pilot-sample requirements ( 27 ) ; the response-free predictions themselves remain a load-bearing input to that pipeline. To the best of our knowledge, no published study has examined if and how different fine-tuning architectures can help in the IDM task on reading-comprehension MCQ s. Two questions remain open: (i) whether items are best encoded jointly, as a flat passage-question-options sequence, or component-wise; and (ii) whether the difficulty-regression target alone is enough, or whether adding an auxiliary task that more closely follows the process underlying test-taker behaviour (the very mechanism that predominantly generates the difficulty) changes the picture, particularly in the small-sample regime where the labelled difficulty signal is sparse. To our knowledge, this study is also the first to examine the effect of the training set size using a large, experimental dataset. To address these questions, we evaluate three transformer-based methods under a nested training-size design. The aim is not to maximise predictive performance, which would call for domain-specific pre-training and extensive hyper-parameter tuning, but to analyse whether transformer-based models are a viable approach for reading-comprehension multiple-choice items, and where further structural and supervisory inductive biases pay off when training data are scarce. A joint-encoding approach processes the concatenated item wording as a single input, collapsing the preprocessing and modelling stages of the feature-engineering tradition into one. A component-wise variant encodes the passage, question, and option set separately through a shared encoder before aggregation, making explicit at the architectural level the aforementioned functional decomposition of a MCQ . A multi-task variant retains joint encoding and adds an

Read the complete text