EssayBench : Evaluating Large Language Models in Multi-Genre Chinese Essay Writing

Fan Gao, Dongyuan Li, Ding Xia, Fei Mi, Yasheng Wang, Lifeng Shang, Baojun Wang · 2025

Read EssayBench : Evaluating Large Language Models in Multi-Genre Chinese Essay Writing by Fan Gao, Dongyuan Li, Ding Xia, Fei Mi, Yasheng Wang, Lifeng Shang, Baojun Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-02596.txt.

Introduction

Judging a student essay takes more than a spell-check, yet large language models are rarely tested on real essay writing. Fan Gao and co-authors built EssayBench, a benchmark for Chinese essay writing across four genres: argumentative, narrative, descriptive, and expository. They gathered and refined 728 real prompts, split into open-ended and constrained sets to mirror real classroom tasks. A fine-grained scoring framework, tuned per genre, grades the essays by combining scores level by level.

The team checked the scoring against human judges, then tested 15 large models to map their strengths and gaps by genre and instruction type. They offer the benchmark as a tool for improving machine essay writing and grading in schools.

Catalog record

CREATORFan Gao, Dongyuan Li, Ding Xia, Fei Mi, Yasheng Wang, Lifeng Shang, Baojun Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-02596

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

EssayBench : Evaluating Large Language Models in Multi-Genre Chinese Essay Writing Fan Gao, Dongyuan Li, Ding Xia, Fei Mi, Yasheng Wang, Lifeng Shang, Baojun Wang Abstract Chinese essay writing and its evaluation are critical in educational contexts, yet the capabilities of Large Language Models (LLMs) in this domain remain largely underexplored. Existing benchmarks often rely on coarse-grained text quality metrics, largely overlooking the structural and rhetorical complexities of Chinese essays, particularly across diverse genres. To address this gap, we propose EssayBench , a multi-genre benchmark specifically designed for Chinese essay writing across four major genres: Argumentative , Narrative , Descriptive , and Expository . We curate and refine a total of 728 real-world prompts to ensure authenticity and meticulously categorize them into the Open-Ended and Constrained sets to capture diverse writing scenarios. To reliably evaluate generated essays, we develop a fine-grained, genre-specific scoring framework that hierarchically aggregates scores. We further validate our evaluation protocol through a comprehensive human agreement study. Finally, we benchmark 15 large-sized LLMs, analyzing their strengths and limitations across genres and instruction types. With EssayBench 1 1 1 https://anonymous.4open.science/r/EssayBench-2B14 , we aim to advance LLM-based Chinese essay evaluation and inspire future research on improving essay generation in educational settings. Benchmark Num. Dataset Composition Evaluation Method Domain Language Constraints LLM F.G. Traits Scoring C-Eval Huang et al. 2023 13,948 General Tasks ZH ✗ ✗ ✗ - AlignBench Liu et al. 2024b 683 General Tasks ZH ✗ ✓ ✗ Direct LongBench-Write Bai et al. 2024 120 General Writing ZH&EN ✗ ✓ ✗ Direct HelloBench Que et al. 2024 647 General Tasks EN ✗ ✓ ✗ Weighted WritingBench Wu et al. 2025 1239 General Writing ZH&EN ✓ ✓ ✗ Direct EssayBench (Ours) 728 Essay Writing ZH ✓ ✓ ✓ Weighted Table 1: Comparison of EssayBench with other benchmarks in terms of size, composition, and evaluations. Figure 1: Comparison between coarse-grained evaluation methods (a) and our fine-grained and genre-oriented framework for EssayBench (b). 1 Introduction Large Language Models (LLMs) Brown et al. 2020 ; Touvron et al. 2023 ; Team. 2023 ; Team 2023 have achieved impressive results in text generation, with growing applications in education, including automated writing support and feedback Gao et al. 2024 . Among these tasks, essay writing plays a central role in language learning and assessment Venkatraman et al. 2025 ; Miura et al. 2025 ; Wen et al. 2025 . However, the lack of robust evaluation frameworks for generated essays limits the development and deployment of LLMs in real-world educational settings Kim et al. 2025 . As shown in Figure 1 , current predominant LLM-as-a-judge strategies Zheng et al. 2023 ; Li et al. 2025 for assessing texts mainly fall into two paradigms. One relies on meta-evaluation to judge response quality in terms of fluency, relevancy, coherency, readability, and hallucination Liu et al. 2023 ; Chen et al. 2023 ; Hashemi et al. 2024 ; Fu et al. 2024 , while the other employs downstream tasks (e.g., question-answering) as proxies for measuring informational richness and accuracy Tan et al. 2024 ; Que et al. 2024 ; Lee et al. 2025 . Although these methods yield valuable insights, they exhibit two fundamental weaknesses. First, the evaluation criteria remain overly coarse-grained, i.e., current LLMs consistently achieve high scores in fluency, relevancy, and coherency Gu et al. 2025 , making it difficult to reveal failure modes or specific weaknesses. Second, existing evaluation methods fail to capture the unique characteristics of essays like logographic characters, complex constructions, and rhetorical traditions, although several benchmarks like AlignBench Liu et al. 2024b and WritingBench Wu et al. 2025 have turned attention to evaluating general Chinese writing. Moreover, Chinese literary and expository practices differ markedly across genres: argumentative essays demand logical structure and persuasive rhetoric Wachsmuth et al. 2017 ; narratives require compelling plot development and character voice Somasundaran et al. 2018 ; descriptive writings emphasize vivid imagery and sensory detail McCarthy 1998 ; and expository passages call for clarity, organization and factual precision Balepur et al. 2023 . However, existing evaluation frameworks largely overlook genre-specific criteria, limiting their ability to reflect the nuanced demands of Chinese essay writing. This motivates our central research question as follows: How can we reliably assess the quality of LLM-generated Chinese essays in ways that truly reflect genre-specific conventions? In this paper, we introduce EssayBench , a fine-grained and multi-genre benchmark tailored for Chinese essay writing. EssayBench covers four widely recognized genres in Chinese education: Argumentative , Narrative , Descriptive , and Expository writing. To ensure alignment with real-world educational scenarios, we collect and manually refine a total of 728 essay prompts. These prompts are further categorized into two types based on their instruction style: Open-Ended and Constrained , allowing us to examine LLMs’ behavior under different writing conditions, as introduced in Section 2 . Additionally, to overcome the limitations of existing evaluation methods for Chinese essay writing, we propose a fine-grained and genre-oriented evaluation framework, as shown in Figure 1 . We define multiple evaluation traits with hierarchical dependencies based on their complexity, ranging from basic to advanced requirements for each essay genre. For each trait, we design targeted sub-questions that reflect genre-specific writing expectations at different levels. To account for the hierarchical nature of these traits, we further introduce a dependency-weighted score aggregation strategy to better capture the writing quality, as introduced in Section 3 . We conduct two key experiments to validate the proposed framework. First, to assess its effectiveness and robustness, we perform a comprehensive human agreement study and a quality sensitivity analysis. The results demonstrate that our evaluation protocol aligns closely with human judgments, especially when applied to more advanced LLMs. More importantly, it significantly improves the ability to distinguish essay quality across high-, medium-, and low-level responses (See Section 4 ). Second, we benchmark 15 large-scale LLMs on the Chinese essay writing using our framework, offering detailed comparisons of their capabilities in writing Chinese essays (See Section 5 ). In Table 1 , we highlight the key differences between our work and existing approaches. In summary, our main contributions are as follows: • We present EssayBench , a multi-genre benchmark tailored for Chinese essay writing, covering Argumentative , Narrative , Descriptive , and Expository genres. The benchmark is curated from real-world scenarios and is suitable for practical use in educational applications. • We propose an effective and robust evaluation protocol for Chinese essays that aligns closely with human judgments and greatly improves the ability to distinguish essays of varying quality. • We benchmark 15 widely used large-scale LLMs to evaluate their strengths and weaknesses in Chinese essay writing, and highlight areas for future improvement. Figure 2: Overview of the EssayBench dataset curation, representative prompts, and the evaluation framework. 2 EssayBench Dataset EssayBench originally contributes to developing the datasets specifically tailored for Chinese Essay Writing. While prior benchmarks Wu et al. 2025 have largely provided queries on creative writing tasks in general domains, they do not adequately capture the structure, purpose, and constraints of formal essays, particularly within educational and academic contexts. To effectively benchmark the essay generation abilities, EssayBench introduces a comprehensive set of essay prompts that span four major and widely recognized genres in Chinese writing instruction Chadbourne 1983 :

Read the complete text