Automated Essay Scoring and Language Certification: Assessing Generalizability, Agreement and Validity for French

Rodrigo Wilkens, Rémi Cardon, Vincent Folny, Thomas François · 2026

Read Automated Essay Scoring and Language Certification: Assessing Generalizability, Agreement and Validity for French by Rodrigo Wilkens, Rémi Cardon, Vincent Folny, Thomas François (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-02009.txt.

Introduction

Wilkens and co-authors assess automated essay scoring for French language certification. Scoring benchmarks have fostered narrow evaluation habits. Broader frameworks urge multidimensional checks of such systems. High-stakes language tests need that wider view most. The team enhance an argument-based validation framework for use.

Fairness analysis joins the practical version of the framework. Correlations with linguistic features are included too. Prediction error and model agreement are also measured. Eight model architectures are compared on French essays. The main corpus holds 27,000 exam essays. Each essay carries scores from two raters. A generalization corpus adds 961 essays with more raters. The study shows what broad validation reveals about systems.

Catalog record

CREATORRodrigo Wilkens, Rémi Cardon, Vincent Folny, Thomas François
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74 KB
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2606.02009

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Automated Essay Scoring and Language Certification: Assessing Generalizability, Agreement and Validity for French Rodrigo Wilkens, Rémi Cardon, Vincent Folny, Thomas François Abstract In Automated Essay Scoring (AES), benchmarking practices have fostered minimalist evaluation practices, in contrast with the broader-view recommendations of evaluation frameworks, such as the argument-based validation framework (ABV), which argued in favor of a multidimensional assessment of systems, especially in the context of high-stakes language tests. In this paper, we introduce an enhanced and more practical version of the ABV framework, incorporating fairness analysis, correlations with linguistic features, prediction error evaluation, and model agreement compared with human raters. Applying this framework to French AES, we compare 8 model architectures on a corpus of 27k exam essays (2 raters each) and a generalization corpus of 961 essays (at least nine raters each). Our analyses illustrate the benefits of applying the ABV framework to better understand the capabilities and pitfalls of AES models, while also advancing the state-of-the-art for French AES. 1 Introduction Automated Essay Scoring (AES) 1 1 1 Automated Essay Scoring is also called Automated Essay Grading (AEG) Valenti et al. 2003 , Automated Essay Evaluation (AEE) Shermis 2015 , Automated Writing Evaluation (AWE) Beigman Klebanov and Madnani 2020 , and Analytic Writing Assessment (AWA) Rudner et al. 2006 . aims to produce systems that imitate groups of expert raters by combining principles of educational assessment and Natural Language Processing (NLP). The goal is to provide a valid, cost-efficient and reliable solution for the automatic evaluation of essays Klebanov and Madnani 2021 . In this paper, we focus on certification tests 2 2 2 For a comprehensive list of certification language tests see en.wikipedia.org/wiki/List_of_language_proficiency_tests . , which are a type of standardized language test defined as “a process by which individuals are certified as having demonstrated some level of knowledge and skill in some domain” ( AERA et al. 2014 , p. 216) . Certification tests are generally high-stake, meaning that outcomes such as “admission, promotion, placement or graduation are directly dependent on test scores” ( Shohamy et al. 1996 , p. 300) in contrast to low-stake exams (e.g., formative evaluation). In view of the social consequences of high-stake tests, their validity and reliability are critical, as already pointed out by Messick 1990 . AES systems tend to be scalable and flexible Klebanov and Madnani 2021 , and reliable and reproducible Yan et al. 2020 , which are desirable characteristics for real-world utilization. Recent AES literature, especially in NLP, has focused more on models’ improvements or new architectures than on the assessment of validity. For ease of comparison, most studies operate within a controlled setting (i.e., the ASAP dataset and a single metric) Kusuma et al. 2022 . However, this does not align with the requirements of high-stakes language tests. Commonly overlooked elements in AES evaluation include the results’ generalizability across different corpora, and the relationship between the model and the human raters’ agreement. Yet, those elements are required for standardized language certification tests, such as IELTS or TCF. In language testing, evaluating the tests’ validity and reliability has been extensively investigated, using the Argument-Based Validation (ABV) framework Chapelle et al. 2008 , best developed by Williamson et al. 2012 . This tradition has had little impact on NLP practices. Aiming to bridge this gap, we perform extensive experiments in the context of one of the major standardized language certification tests for French, the French Knowledge Test ( Test de connaissance du Français – TCF), and report them within the ABV framework. This work advances the state-of-the-art for AES, targeting the French language. We compare 8 different model architectures, including hybrid architectures 3 3 3 https://gitlab.com/rswilkens/linguistic-features-in-transformers , relying on the largest corpus made for French AES. In addition, based on the ABV framework, we report the most extensive evaluation of AES models for French to date, with a strong view on the real-world utilization of our system. We go further in our systematic comparison of various AES approaches, by leveraging a high-quality test corpus made for this specific purpose. Beyond this specific contribution for French, our main contribution consists in an enhancement of the ABV framework. As it is not easily automatable in its original form, we have identified relevant automatic metrics from NLP and systematically linked them to the framework. Thus, we offer a practical AES evaluation framework for language testing. The focus of our approach is aligned with the recommendations for adding automated scoring to an existing assessment Madnani and Cahill 2018 . We argue that, by bridging the gap between methodologies focused on model validity and more theory-grounded arguments, such a methodological approach will enable enhancements to automatic evaluation practices for AES. The outline of the paper is as follows. We first provide an overview of the literature on AES, focusing on models, their limitations for certified exams, and an introduction of the ABV framework (Section 2 ). We then describe how we approach this framework that will guide our study in Section 3 . Next, we describe the models’ architectures we compare (Section 4 ) and the corpora we work with (Section 5 ). Finally, we present the results of our experiments, structured in relation to our enhanced ABV framework (Section 6 ), discuss our findings (Section 7 ), and conclude (Section 8 ). 2 Approaches for AES 2.1 AES Models AES has a long tradition of feature engineering methods, as they facilitate studying the relation to the test construct. According to Zesch et al. 2015 , features can be classified according to their degree of task dependence, ranging from weak to strong. They showed that task-independent features transfer better between prompts, while the task-dependent ones do not generalize well but perform better. This is especially important in the context of certified exams, where prompts need to be regularly updated. Examples of feature-based works are Burstein and Chodorow 1999 with discourse features, or Zesch et al. 2015 , which considers readability features, grammar errors, and task and topical similarity. AES research then transitioned to deep learning, inspired by the results achieved in other NLP tasks. This led to the development of models independent of linguistic features. Taghipour and Ng 2016 explored CNN and LSTM architectures, replacing linguistic features with word embeddings. They observed that combining results from models trained with different samples and architectures improves performance. Building on this, subsequent studies incorporated task-specific knowledge into the models, e.g. coherence as part of the architecture Tay et al. 2018 and rating criteria Liang et al. 2018 . Yet, Jin et al. 2018 obtained good results by bringing back linguistic features into the models, opening the avenue for hybrid models. Motivated by the strong performance of large language models, Rodriguez et al. 2019 ; Mayfield and Black 2020 explored the use of BERT for AES, obtaining discouraging results. Yang et al. 2020 achieved promising results by incorporating multiple loss functions (regression and ranking) in their R2BERT model. Revisiting hybrid methods, Uto et al. 2020 proposed to concatenate 25 handcrafted essay-level features (length-based, syntactic, word-based, and readability features) to BERT distributed representations. Exploring generative models, Mizumoto and Eguchi 2023 applied GPT-3 to AES, finding a 52% exact agreement with the gold-standard, and demonstrating that the inclusion of

Read the complete text