Measuring What Matters—or What’s Convenient?: Robustness of LLM-Based Scoring Systems to Construct-Irrelevant Factors

Cole Walsh, Rodica Ivan · 2026

Read Measuring What Matters—or What’s Convenient?: Robustness of LLM-Based Scoring Systems to Construct-Irrelevant Factors by Cole Walsh, Rodica Ivan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-25674.txt.

Introduction

Walsh and Ivan test whether an LLM-based scoring system is swayed by factors outside the target skill. The system scores short open responses on a situational judgment test. The authors pad answers with empty text, add spelling errors, and vary writing polish. The system stayed mostly robust to those changes, they report. Copying large passages into an answer lowered its score. Off-topic answers were penalized heavily. The authors say the results support careful design of LLM scoring tools for real tests.

Catalog record

CREATORCole Walsh, Rodica Ivan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34821
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.25674

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Measuring What Matters—or What’s Convenient?: Robustness of LLM-Based Scoring Systems to Construct-Irrelevant Factors Cole Walsh, Rodica Ivan Abstract Automated systems have been widely adopted across the educational testing industry for open-response assessment and essay scoring. These systems commonly achieve performance levels comparable to or superior than trained human raters, but have frequently been demonstrated to be vulnerable to the influence of construct-irrelevant factors (i.e., features of responses that are unrelated to the construct assessed) and adversarial conditions. Given the rising usage of large language models in automated scoring systems, there is a renewed focus on “hallucinations” and the robustness of these LLM-based automated scoring approaches to construct-irrelevant factors. This study investigates the effects of construct-irrelevant factors on a dual-architecture LLM-based scoring system designed to score short essay-like open-response items in a situational judgment test. It was found that the scoring system was generally robust to padding responses with meaningless text, spelling errors, and writing sophistication. Duplicating large passages of text resulted in lower scores predicted by the system, on average, contradicting results from previous studies of non-LLM-based scoring systems, while off-topic responses were heavily penalized by the scoring system. These results provide encouraging support for the robustness of future LLM-based scoring systems when designed with construct relevance in mind. 1 Introduction Automatic evaluation of open-response text, including short-answer responses and essays, is one of the earliest and most widely explored applications of natural language processing and artificial intelligence (AI) in education. Over time, methods for automatically evaluating written work have evolved from using handcrafted features (e.g., type-token ratios, part-of-speech tagging) and simple models [ 10 ] to more complex approaches including neural networks and transformer-based models [ 10 , 15 ] . With each new development automated scoring systems have consistently demonstrated strong alignment with human raters [ 10 ] . Despite their strong overall performance and successful adoptions, studies have demonstrated the susceptibility of many automated scoring systems to construct-irrelevant factors (i.e., features of responses that are unrelated to the construct assessed) regardless of underlying architecture. Chief among these construct-irrelevant factors is text length; early studies involving simple models demonstrated that repeating sentences or entire paragraphs within an essay could artificially inflate scores predicted by these models [ 14 ] , while more recent studies have demonstrated that this issue persists even in transformer-based scoring systems [ 15 ] . Other studies have demonstrated outsized impacts of injecting particular words or phrases (e.g., “from one perspective”, “on the other hand”, “in the final analysis”) [ 14 ] and including off-topic text [ 14 , 18 ] . The fragility of existing automated scoring systems under such adversarial conditions undermines trust in these systems to accurately measure the intended construct rather than exploitable proxies. In recent years, automated scoring research has begun to focus on systems employing large language models (LLMs) either directly [ 11 ] or as part of more complex scoring systems [ 1 , 17 ] . While these systems have not yet met state-of-the-art performance metrics achieved by transformer-based systems in many cases [ 1 , 7 ] , what they sacrifice in accuracy they make up for in other ways: creating human-interpretable scores and feedback, reducing the amount of required training data, and easing adoption by less technical assessment developers. With this rise in LLM-based scoring solutions, there is a renewed focus on the robustness of these systems under adversarial conditions, especially with widespread public awareness of the limitations in the underlying technology (e.g., “hallucinations” [ 3 ] ), which was not the case with prior modes of automated scoring. This study contributes towards our understanding of the effects of construct-irrelevant factors on LLM-based scoring systems by investigating how the following factors influence scores produced by one such system: 1. Meaningless Text, 2. Writing Sophistication, and 3. Off-Topic Texts Writing sophistication is often considered a construct-relevant factor by assessments of writing or language proficiency [ 1 , 13 ] . As will be discussed in greater detail in Sec. 2.1 , we constructed our LLM-based scoring system for a situational judgment test (SJT) designed to measure personal and professional skills (e.g., communication, teamwork, problem-solving, and critical thinking) [ 12 ] where elements of writing sophistication (e.g., spelling, grammar, structure, and organization) were considered construct-irrelevant. This study, then, serves a secondary purpose of investigating the feasibility of constructing automated scoring systems that measure distinctly different constructs from scoring systems employed for measuring writing or language proficiency. 2 Methods 2.1 Source Data Assessment We collected data using a 30 item open-response SJT designed to assess and provide feedback on students’ personal and professional skills. This instrument, intended primarily for low-stakes formative (as opposed to summative) usage within a program of study, assessed students’ skills along four dimensions [ 8 ] : 1. Intrapersonal skills: Understanding and regulating oneself by recognizing emotions, biases, and behaviors, staying motivated, adapting to challenges, and committing to lifelong learning for personal growth. 2. Interpersonal skills: Building meaningful connections and working effectively with others by communicating clearly, showing empathy, collaborating toward shared goals, and inspiring others towards positive outcomes. 3. Social and Ethical Responsibility : Recognizing and respecting people’s differences, upholding ethical principles, and contributing to the well-being of society through responsible actions. 4. Critical Thinking and Problem Solving: Gathering information, evaluating options, and finding effective solutions to problems while efficiently managing resources and risks. For each item, students were presented with a hypothetical scenario and asked how they would respond given the situation. Scenarios were presented in one of two formats: text-based scenarios included a short description of a situation, while video-based scenarios were depicted using AI-generated avatars enacting a situation. Below is a summary of one text-based scenario students were shown and an accompanying item (question): Scenario: Kendra and Alex, two close friends co-leading a major group project, are clashing over Kendra pushing a strict schedule to meet deadlines. Alex wants more group input on task division and worries that Kendra’s approach could undermine team trust and strain their friendship. Item: How could the co-leads, Alex and Kendra, have collaborated more effectively with the group to create a clear and realistic project plan? This item was meant to assess interpersonal skills and, more specifically, collaboration. To receive a high score on this item, students were expected to demonstrate behaviors consistent with collaboration in their responses such as suggesting approaches to managing differences and resolving conflict . Automated Scoring System We developed an automated scoring system for this assessment that used a dual-architecture LLM-as-a-Judge feature extraction component together with clear-box regression algorithms similar to those described in Refs. [ 1 , 17 ] . Aligning with the constructs assessed by the instrument, we designed the scoring system to reward higher-level features related to personal and professional skills (such as the collaboration-related behavior noted above) rather than language proficiency or content-mastery. We constructed this system using 26,571 assessment responses (spread across all 30 items) from 910 students representing six schools. These schools comprised a mixture of business, engineering, health sciences, and medicine programs. All responses were first evaluated by trained human raters using a 1–5 Likert rating scale; these ratings were used to train and evaluate the automated scoring system, which achieved performance

Read the complete text