Automated Essay Scoring Incorporating Annotations from Automated Feedback Systems

Christopher Ormerod · 2025

Read Automated Essay Scoring Incorporating Annotations from Automated Feedback Systems by Christopher Ormerod (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22771.txt.

Introduction

Christopher Ormerod shows that the notes a feedback tool leaves on an essay can also make its automatic grade fairer. Automated essay scoring is fast but blind to much of what teachers mark. His 2025 study adds two kinds of annotation into the scoring pipeline and measures the gain.

The test bed is the PERSUADE corpus of persuasive student essays. One annotation type flags spelling and grammar slips, produced by a generative model acting as a corrector. The other marks argument parts, tagged by a classifier trained for that job. Scoring models that could see these notes graded more accurately than those given plain text. Classifiers built on encoder models gained most from the added signals.

Catalog record

CREATORChristopher Ormerod
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-22771

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Automated Essay Scoring Incorporating Annotations from Automated Feedback Systems Christopher Ormerod Abstract This study illustrates how incorporating feedback-oriented annotations into the scoring pipeline can enhance the accuracy of automated essay scoring (AES). This approach is demonstrated with the Persuasive Essays for Rating, Selecting, and Understanding Argumentative and Discourse Elements (PERSUADE) corpus. We integrate two types of feedback-driven annotations: those that identify spelling and grammatical errors, and those that highlight argumentative components. To illustrate how this method could be applied in real-world scenarios, we employ two LLMs to generate annotations – a generative language model used for spell-correction and an encoder-based token-classifier trained to identify and mark argumentative elements. By incorporating annotations into the scoring process, we demonstrate improvements in performance using encoder-based large language models fine-tuned as classifiers. 1 Introduction Automated Essay Scoring (AES) uses statistical models to assign grades to essays that approximates hand-scoring Shermis and Hamner 2013 . Automated Writing Evaluation (AWE) is the provision of automated feedback designed to help students iteratively improve their essays Huawei and Aryadoust 2023 . Initial attempts at AES and AWE were based on Bag-of-Words (BoW) models that combine frequency-based and hand-crafted features Attali and Burstein 2006 ; Page 2003 . Well-designed features can serve two purposes; to improve scoring accuracy and provide feedback to students to improve their essays. These features tend to be global features, such as the number of words, sentence length, or readability metrics, and not based on fine-grained semantics or the organizational structure of essays. Many modern AES engines employ transformer-based Large Language Models (LLM)s Rodriguez et al. 2019 , which offer improved accuracy over bag-of-words models. However, this comes at the expense of reduced interpretability due to implicit feature definition. Some researchers have sought to combine LLM-derived features with traditional hand-crafted features to enhance accuracy and provide some level of interpretability Uto and Uchida 2020 . LLMs also offer the ability to provide semantically rich feedback such as key phrases from explainable AI Boulanger and Kumar 2020 , annotation schemas for automated writing evaluation systems Crossley et al. 2022 ; Lottridge et al. 2024 , and the generation of detailed feedback through LLM prompting techniques Lee et al. 2024 . This study considers how these semantically rich features, designed primarily for providing feedback, can also enhance scoring accuracy. We demonstrate our approach using the Persuasive Essays for Rating, Selecting, and Understanding Argumentative and Discourse Elements (PERSUADE) corpus, which is a dataset of essays in which the argumentative components of the essays were annotated Crossley et al. 2022 . This dataset also contains scores assigned against an openly available holistic rubric 1 1 1 https://github.com/scrosseye/persuade_corpus_2.0 and demographic data, which allows us to test any AES system with respect to operational standards, including the addition of potential bias Williamson et al. 2012 . The two classes of features we consider are derived from Grammatical Error Correction (GEC) and Computational Argumentation. The goal of GEC is to provide a mapping from a sentence that may or may not contain errors in language, to a version with the same meaning with fewer errors in language Martynov et al. 2023 , while computational argumentation seeks to isolate and analyze the set of argumentative components of an essay Stab and Gurevych 2014a . For these features to be incorporated into an AES pipeline, we leverage the ability of language models to accurately annotate argumentative components Ormerod et al. 2023 and correct spelling and grammatical errors Rothe et al. 2021 . These spelling and grammatical corrections can then be annotated and classified to provide locally defined information on conventions Korre and Pavlopoulos 2020 . Once these annotations have been derived, we incorporate the annotations using Extensible Markup Language (XML) for easy parsing, which facilitates natural integration into an AWE system based on essays encoded in HTML. We must also be cognizant that increased automation can exacerbate biases, especially for English Language Learners Ormerod 2022a . For this reason, we also examine whether this pipeline leads to greater bias by examining the standardized mean difference for the relevant subgroups. In § 2 , we describe our methods, including the data used, the models, and the approach. §The performance of the annotation models and the scoring models are presented in § 3 . We will discuss the findings and suggest future directions in § 4 . 2 Method 2.1 Data The PERSUADE corpus is an openly available dataset of 25,996 argumentative essays between grades 6 and 12 on a range of 15 different topics Crossley et al. 2022 . The essays are responses to prompts that are either dependent on source material, or independent of source material. The set has been divided into a training set and a test set by the original authors. An outline of the composition of these two sets is presented in Table 1 . Avg. Grade Train Test Total Len. 6 688 684 1372 294.6 8 5614 4015 9629 374.9 9 1831 235 2066 426.6 10 4654 3620 8274 407.6 11 1863 1220 3083 610.9 12 243 161 404 469.0 Unk. 701 467 1168 452.5 Total 15594 10402 25996 418.1 Table 1: The grade level and length statistics for the training and testing splits for the PERSUADE corpus, and the counts of essays that are responses to prompts that are dependent (Dep.) on source material and independent (Ind.) of source material. 2.1.1 Annotations A key characteristic of the corpus that makes it useful from the standpoint of computational argumentation is the annotations. The argumentative clauses of each essay were identified and classified into one of seven classes: 1. Lead (L): An introduction that begins with a statistic, a quotation, a description, or some other device to grab the reader’s attention and point toward the thesis. 2. Position (P): An opinion or conclusion on the main question 3. Claim (C1) : A claim that supports the position 4. Counterclaim (C2) : A claim that refutes another claim or gives an opposing reason to the position 5. Rebuttal (R) : A claim that refutes a counterclaim 6. Evidence (E) : Ideas or examples that support claims, counterclaims, rebuttals, or the position 7. Concluding Statement (C3) : A concluding statement that restates the position and claims There are an average of 11.0 annotated components in each essay. Some descriptive statistics on the distribution of applied labels can be found in Table 2 . Among the labels, the most frequently applied labels are Claims and Evidence and the least frequently applied labels are Counter Claims and Rebuttals. In accordance with state standards, the development of a counter-argument in persuasive essay writing is developed at grades eight and beyond, hence, Counterclaims and Rebuttals are rarely applied at the sixth-grade level. 6 8 9 10 11 12 L 4.3 4.5 4.7 5.9 6.0 5.0 P 9.8 9.0 9.1 9.9 7.0 8.3 C1 27.2 27.7 27.6 29.8 31.2 34.3 C2 2.1 2.8 5.2 2.5 5.3 5.0 R 1.5 2.2 3.7 1.7 4.3 2.0 E 27.3 25.7 26.8 28.7 23.4 27.5 C3 8.0 7.5 7.6 8.7 6.9 7.5 Table 2: Some descriptive statistics regarding the distribution of annotations with respect to the various grade levels. We did not use the effectiveness scores for the discourse elements in this

Read the complete text