LCES: Zero-shot Automated Essay Scoring via Pairwise Comparisons Using Large Language Models
Takumi Shibata, Yuichi Miyamura · 2025
Read LCES: Zero-shot Automated Essay Scoring via Pairwise Comparisons Using Large Language Models by Takumi Shibata, Yuichi Miyamura (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-08498.txt.
Introduction
Takumi Shibata and Yuichi Miyamura grade essays by comparison rather than by direct scores in a 2025 paper. Language models can grade essays with no training examples, which would cut marking costs. But asked for an absolute score, models drift with their biases and grade inconsistently. The authors reframe the job: show the model two essays and ask which is better.
Pairwise verdicts are steadier than absolute numbers. Collecting every possible pair grows too fast, so a RankNet model learns continuous scores from a sampled set of comparisons, much as ranking systems learn from wins and losses. On standard essay benchmarks, the comparison method beat direct zero-shot scoring while staying efficient, and it held up across different underlying language models. The authors pitch it for real marking support.
Catalog record
| CREATOR | Takumi Shibata, Yuichi Miyamura |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-08498
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LCES: Zero-shot Automated Essay Scoring via Pairwise Comparisons Using Large Language Models Takumi Shibata, Yuichi Miyamura Abstract Recent advances in large language models (LLMs) have enabled zero-shot automated essay scoring (AES), providing a promising way to reduce the cost and effort of essay scoring in comparison with manual grading. However, most existing zero-shot approaches rely on LLMs to directly generate absolute scores, which often diverge from human evaluations owing to model biases and inconsistent scoring. To address these limitations, we propose LLM-based Comparative Essay Scoring (LCES), a method that formulates AES as a pairwise comparison task. Specifically, we instruct LLMs to judge which of two essays is better, collect many such comparisons, and convert them into continuous scores. Considering that the number of possible comparisons grows quadratically with the number of essays, we improve scalability by employing RankNet to efficiently transform LLM preferences into scalar scores. Experiments using AES benchmark datasets show that LCES outperforms conventional zero-shot methods in accuracy while maintaining computational efficiency. Moreover, LCES is robust across different LLM backbones, highlighting its applicability to real-world zero-shot AES. 1 Introduction Automated essay scoring (AES) aims to assess the quality of written essays using natural language processing and machine learning techniques. AES has garnered significant attention as a means to reduce the cost relative to human grading and to ensure fairness ( Uto 2021 ; Do et al. 2023 ) . Most conventional AES methods focus on prompt-specific approaches 1 1 1 Here, we use prompt to refer to the essay topic and LLM prompt to refer to instructions given to language models. , which train machine learning models or neural networks on scored essays tailored to each essay prompt ( Alikaniotis et al. 2016 ; Dong et al. 2017 ; Yang et al. 2020 ; Xie et al. 2022 ; Shibata and Uto 2022 ; Wang and Liu 2025 ) . However, this approach requires collecting large amounts of scored essays for every prompt, resulting in substantial costs. To address this issue, recent studies have proposed cross-prompt AES methods that leverage domain adaptation or domain generalization techniques ( Ridley et al. 2021 ; Chen and Li 2023 ; Do et al. 2023 ; Chen and Li 2024 ; Li and Pan 2025 ) . In those techniques, models are trained on scored essays from source prompts and evaluated on a different, target prompt. Although these methods can maintain high score accuracy even when scored essays for the target prompt are scarce or unavailable, they still require a certain amount of scored essay data for training, leaving unsolved the fundamental challenge of satisfying data requirements. Figure 1: Comparison between (a) direct essay scoring using LLMs and (b) our proposed LCES framework. In parallel to the above, large language models (LLMs) have recently demonstrated remarkable capabilities across various natural language processing tasks in zero-shot settings ( Kojima et al. 2022 ) , motivating efforts to apply them to AES without the use of scored essays. A typical zero-shot AES approach instructs an LLM with a rubric and essay to generate a numerical score ( Mizumoto and Eguchi 2023 ) . A more advanced method first converts the original rubric defined in the dataset into trait-level rubrics using LLMs, then employs LLMs to independently predict scores for each trait, and finally aggregates these scores to estimate the overall score ( Lee et al. 2024 ) . While these approaches are promising, they still have several limitations of direct score generation. They tend to be sensitive to the phrasing of LLM prompt and susceptible to model bias, and often exhibit grading behavior inconsistent with that of human raters ( Zheng et al. 2023 ; Liu et al. 2024 ; Mansour et al. 2024 ; Li et al. 2025 ) . To address the issues above, we explore an alternative AES formulation. Instead of predicting absolute scores, we instruct the LLM to perform pairwise comparisons in which the better of two essays is determined. This approach is inspired by recent advances in LLM-based evaluation for natural language generation ( Liu et al. 2024 ; Liusie et al. 2024a ) , dialogue systems ( Park et al. 2024 ) , and information retrieval ( Qin et al. 2024 ) , where pairwise comparisons have demonstrated stronger alignment with human preferences. Despite its promise, pairwise comparisons remain largely unexplored in the AES literature. Against this backdrop, we propose L LM-based C omparative E ssay S coring (LCES), a novel framework for zero-shot AES that first collects pairwise comparisons using LLMs and then estimates continuous essay scores. As shown in Figure 1 , LCES differs from conventional LLM-based scoring by shifting from direct score generation to relative preference modeling. To scale this approach to large essay datasets, we employ RankNet ( Burges et al. 2005 ) , which allows efficient training from pairwise comparisons without exhaustively enumerating all essay pairs. This mitigates the quadratic complexity in the number of items as is typically seen in pairwise comparisons ( Liusie et al. 2024b ) . Through comprehensive experiments using standard AES benchmark datasets, we demonstrate that LCES substantially outperforms existing zero-shot scoring methods. Moreover, LCES is robust to the choice of LLM and can be applied with virtually any model, making it well suited for practical deployment. The contributions of this work are summarized as follows: (1) We introduce the first AES framework based on LLM-generated pairwise comparisons, addressing key limitations of direct score generation. (2) We leverage RankNet to convert LLM-generated preferences into continuous scores, enabling accurate and computationally efficient zero-shot AES. (3) Extensive experiments confirm that LCES outperforms conventional zero-shot AES baselines and is robust across different types of LLMs. 2 Related Work Automated Essay Scoring. Early AES systems were largely prompt-specific , beginning with handcrafted-feature-based models Yannakoudakis et al. 2011 and later adopting neural networks ( Dong et al. 2017 ; Xie et al. 2022 ) . Because it is costly to collect scored essays for every new prompt, cross-prompt methods have been proposed to train models that generalize across prompts ( Ridley et al. 2020 ; Chen and Li 2023 ; Chen and Li 2024 ) . Recently, zero-shot AES using LLMs has emerged Mizumoto and Eguchi 2023 ; Yancey et al. 2023 ; Wang et al. 2024 ; Mansour et al. 2024 ; Lee et al. 2024 , enabling score generation without the use of scored essays. Mizumoto and Eguchi 2023 used OpenAI’s text-davinci-003 to score essays based on rubric and essay content. In a zero-shot framework called Multi-Trait Specification (MTS), Lee et al. 2024 instructed an LLM to generate trait-level rubrics and then used them to evaluate essays by scoring each trait individually and aggregating the results. Mansour et al. 2024 demonstrated that LLM-generated scores are highly sensitive to the instructions given to the model, raising concerns about reliability. While zero-shot AES offers a promising direction, its scoring accuracy still lags behind supervised prompt-specific and cross-prompt methods. LLM-based Evaluation. With the growing zero-shot capabilities of LLMs, the LLM-as-a-judge paradigm ( Zheng et al. 2023 ) has gained attention as a general framework for using LLMs in evaluation tasks. Although direct score generation is common, it often suffers from LLM prompt sensitivity (