Operationalizing Automated Essay Scoring: A Human-Aware Approach

Yenisel Plasencia-Calaña · 2025

Read Operationalizing Automated Essay Scoring: A Human-Aware Approach by Yenisel Plasencia-Calaña (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-21603.txt.

Introduction

Yenisel Plasencia-Calana studies how to run automated essay scoring in a human aware way in a 2025 paper. Scoring systems are usually judged on accuracy alone, but real use also turns on bias, robustness, and how well scores can be explained. The study compares classic machine learning scorers with large language model scorers across those human facing dimensions.

Machine learning models scored more accurately but explained their marks poorly, while language models gave richer explanations. Both families struggled with bias and with robustness on edge case scores. The analysis maps the trade offs between the two routes and argues for scoring methods that schools can trust in daily operation, not just in benchmark tests.

Catalog record

CREATORYenisel Plasencia-Calaña
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-21603

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Operationalizing Automated Essay Scoring: A Human-Aware Approach Yenisel Plasencia-Calaña Abstract This paper explores the human-centric operationalization of Automated Essay Scoring (AES) systems, addressing aspects beyond accuracy. We compare various machine learning-based approaches with Large Language Models (LLMs) approaches, identifying their strengths, similarities and differences. The study investigates key dimensions such as bias, robustness, and explainability, considered important for human-aware operationalization of AES systems. Our study shows that ML-based AES models outperform LLMs in accuracy but struggle with explainability, whereas LLMs provide richer explanations. We also found that both approaches struggle with bias and robustness to edge scores. By analyzing these dimensions, the paper aims to identify challenges and trade-offs between different methods, contributing to more reliable and trustworthy AES methods. 1 Introduction Automated Essay Scoring (AES) [ 18 , 2 , 23 ] is an interdisciplinary field at the intersection of Artificial Intelligence and Education, designed to support the essay grading process. These systems aim to overcome human limitations, including grading accuracy, human biases, and associated costs. Manual grading usually is time-consuming and slow [ 20 ] , resulting in prolonged waiting times for students to receive their grades. Besides, there are high costs associated due to the need for professional and experienced evaluators. Other challenges include the risk of errors from inherent biases, due to fatigue from evaluating large volumes of essays, and inconsistencies in grading outcomes. AES can assist educators in multiple ways, the most widely accepted application being formative assessment. In this context, AES provides students with immediate feedback on their writing, allowing them to learn about the errors they make, identify improvements, and refine their work before final submission. By offering constructive insights on grammar, structure, organization, and coherence among other dimensions, it facilitates learning and helps students to refine their writing skills over time. Another important application is summative assessment, where AES supports educators in grading tests and assignments. By automating the evaluation process, AES reduces the time and effort required for large-scale grading, while ensuring efficiency and consistency. This could be particularly useful in large scale grading such as standardized tests and online education, where manually grading essays can be labor intensive. The goal is to use AES as a tool to assist educators, where the decision making and feedback still depend on the educator, and not as replacement of human judgment. Over the years, numerous attempts have been made to develop AES systems for practical use. Notable examples include e-rater [ 1 ] , designed to support Common Core Standards in both classroom settings and large-scale state assessments, and IntelliMetric [ 21 ] , developed for scoring open-ended essays. In recent years, AES systems have evolved due to advancements in text classification methods, particularly by leveraging progress in transformer models [ 10 ] and Large Language Models (LLMs). Early AES systems, such as e-rater and IntelliMetric, relied on manually designed features, including grammar, coherence, and lexical diversity, to predict scores. More recently, deep learning models, especially transformer-based architectures like BERT, RoBERTa, and DeBERTa [ 10 ] , have revolutionized NLP by accurately representing text semantics and context, improving accuracy in different text classification tasks including AES. Current main approaches in the AES literature include models based on handcrafted features [ 23 ] , transformers-based models [ 2 ] such as DeBERTaV2 [ 10 ] and DeBERTaV3 [ 8 ] , tree-based classifiers [ 16 , 13 ] , and neural networks [ 24 ] such as Long Short-Term Memory (LSTM) [ 11 ] . More recently, we have witnessed the progress in Large Language Models (LLMs) such as GPT-4, which have introduced new paradigms for AES through zero-shot [ 18 , 26 ] and few-shot learning [ 22 ] , eliminating the need for task-specific training or fine-tuning. Most studies in the literature focus on optimizing the accuracy of AES systems. However, for AES systems to be adopted and used in a human-aware manner, accuracy is just one of several important dimensions. Other factors must be considered to develop a system that is practical and ethically sound. In this paper, we explore the challenges of operationalizing AES from a human-aware perspective for the main to groups of AES approaches in the literature. We state that there is a gap between the theoretical advances in AES and the requirements for operationalizing AES systems. We examine operationalization through key dimensions such as robustness, bias and explainability. For example, bias in AES models can disproportionately affect certain demographic groups, while lack of explainability can hinder trust and adoption in educational settings. This work explores the human-aware operationalization of AES, emphasizing key dimensions beyond accuracy. We investigate: • Robustness: How different AES approaches handle stability of output scores, confidence in predictions, computational demand, and edge cases? • Fairness: Whether AES models exhibit biases between demographic groups. • Explainability: The extent to which AES systems can explain their scores and provide human-interpretable feedback. To address these challenges, we pose the following research question: What challenges and differences do the main type of AES approaches present in relation to human-aware operationalization dimensions such as robustness, bias, and explainability? Through comparative analysis of ML-based and LLM-based AES methods, this paper highlights trade-offs between different techniques and provides insights into their suitability for practical use in educational settings. 2 Methods We selected a set of methods that represent each of the current approaches in the literature to develop our analysis. They are the following: • Methods based on handcrafted features. We combine Word2vec, term frequency and inverse document frequency (TF-IDF), readability, text complexity, and grade level features. On top of these features we use either Histogram gradient boosting [ 16 ] classification or regression, and LightGBM [ 13 ] regression. • Method based on transformer features DeBERTa, either a finetuned DeBERTaV3 [ 9 ] or DeBERTaV2 embeddings [ 10 ] + Long Short-Term Memory networks (LSTM) [ 11 ] as classifier. We chose the DeBERTa model since it has set new standards in key Natural Language Processing (NLP) domains, including language comprehension, text generation, and question answering. DeBERTa leverages a self-attention mechanism to process sequential data, which allows the model to analyze all input tokens and create contextualized representations for each token based on its surrounding context. LSTM is a type of Recurrent Neural Network which allows to capture long term dependencies in the data. We specifically use Bidirectional LSTM (BiLSTM), which processes information both forward and backward, usually improving performance in NLP tasks. • Methods based on LLMs. Recently, the progress on LLMs and the accuracy they show on different tasks, have made them attractive for different text recognition tasks, including AES [ 12 ] . The LLM can be prompted with instructions, the essay to be scored and a rubric, and it provides a grade to the essay based on the rubric. Practitioners have also introduce new approaches to LLM use based on what is called agentic design patterns, where the LLM is enhanced either with data from which it can learn, tools, or processes that can help the LLM to generate higher quality answers. In AES, agentic design patterns such as few-shot learning and chain-of-thought have been recently introduced [ 15 ] . We analyze these approaches because

Read the complete text