Specialists or Generalists? Multi-Agent and Single-Agent LLMs for Essay Grading
Jamiu Adekunle Idowu, Ahmed Almasoud · 2026
Read Specialists or Generalists? Multi-Agent and Single-Agent LLMs for Essay Grading by Jamiu Adekunle Idowu, Ahmed Almasoud (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22386.txt.
Introduction
Jamiu Adekunle Idowu and Ahmed Almasoud ask whether a team of AI graders beats a lone one on school essays. Their 2026 preprint tests single-agent and multi-agent setups on the ASAP 2.0 set of student essays. Auto grading is now common, yet how the setup shapes scores is little known.
They check scores across essays of high, mid, and low quality. A setup that shines on strong essays may sag on weak ones. Knowing that split helps schools pick a fair grader. It also warns where human review must stay in the loop. The work guides real classroom use.
Catalog record
| CREATOR | Jamiu Adekunle Idowu, Ahmed Almasoud |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-22386
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Specialists or Generalists? Multi-Agent and Single-Agent LLMs for Essay Grading Jamiu Adekunle Idowu, Ahmed Almasoud Abstract Automated essay scoring (AES) systems increasingly rely on large language models, yet little is known about how architectural choices shape their performance across different essay quality levels. This paper evaluates single-agent and multi-agent LLM architectures for essay grading using the ASAP 2.0 corpus. Our multi-agent system decomposes grading into three specialist agents (Content, Structure, Language) coordinated by a Chairman Agent that implements rubric-aligned logic including veto rules and score capping. We test both architectures in zero-shot and few-shot conditions using GPT-5.1. Results show that the multi-agent system is significantly better at identifying weak essays while the single-agent system performs better on mid-range essays. Both architectures struggle with high-quality essays. Critically, few-shot calibration emerges as the dominant factor in system performance – providing just two examples per score level improves QWK by approximately 26% for both architectures. These findings suggest architectural choice should align with specific deployment priorities, with multi-agent AI particularly suited for diagnostic screening of at-risk students, while single-agent models provide a cost-effective solution for general assessment. 1 Introduction The automated assessment of student writing is a long-standing challenge in educational technology, driven by the need to reduce the substantial workload teachers face in grading ( Seßler et al., 2025 ; Figueras et al., 2025 ). While traditional methods have existed for decades, the emergence of Large Language Models (LLMs) has fundamentally altered the landscape of Automated Essay Scoring (AES). Unlike previous generations of automated graders, LLMs possess linguistic knowledge that allows them to analyze essays holistically and provide feedback that potentially benefits both teachers and students ( Mansour et al., 2024 ). Recent studies have demonstrated that fine-tuning models like GPT-3.5 can significantly outperform earlier architectures such as BERT in scoring accuracy ( Latif & Zhai, 2024 ), and that closed-source models continue to show advantages over open-source alternatives in alignment with human ratings ( Seßler et al., 2025 ). However, despite these advances, the optimal architectural deployment of LLMs for essay grading remains an open question. Proponents of multi-agent architectures argue that decomposing the grading task, separating rubric extraction, scoring, and feedback into distinct roles, can enhance performance, interpretability, and human-machine agreement ( Wang, Ding, et al., 2025 ; Su et al., 2025 ). Conversely, other research indicates that increasing complexity does not guarantee better results. For instance, recent comparisons in student reflection assessment found that single-agent strategies utilizing few-shot prompting actually achieved higher match rates with human evaluators than multi-agent alternatives ( Li et al., 2025 ). Similarly, in collaborative problem-solving contexts, multi-agent workflows failed to improve accuracy relative to single-agent models ( Wang, Gopalakrishnan, & Bergner, 2025 ). This divergence in findings highlights a critical gap in the literature, especially since little is known about how architectural choices interact with essay quality at a granular level. Does a multi-agent system justify its increased computational cost across the entire score distribution? This study addresses this gap by conducting a rigorous comparison of Single-Agent and Multi-Agent architectures using the ASAP 2.0 corpus ( Crossley et al., 2025 ). 2 Related Work 2.1 LLM Capabilities in Automated Scoring The application of LLMs to automated grading is gaining traction, with results suggesting they can effectively reduce teacher workload ( Seßler et al., 2025 ). Comparisons between models, such as ChatGPT and Llama, indicate that performance is highly dependent on prompt engineering, yet these models generally exhibit comparable average performance to state-of-the-art methods while offering the added benefit of qualitative feedback ( Mansour et al., 2024 ). However, challenges regarding consistency and bias remain. Errica et al. (2025) highlighted the sensitivity of LLMs to minor prompt variations, proposing metrics to measure consistency across rephrasings. Furthermore, Guo et al. (2025) investigated scoring disparities toward English Language Learners (ELLs), finding that while large training datasets can mitigate bias, limited samples may lead to scoring distortions. These findings underscore the need for robust evaluation frameworks when deploying LLMs for assessment. 2.2 Multi-Agent Architectures in Education The integration of multi-agent systems represents a significant trend in educational AI. These systems typically decompose complex assessment tasks into specialized roles. Some frameworks introduced by recent studies include: • AutoSCORE ( Wang, Ding et al., 2025 ) used two agents, one for component extraction and one for scoring, to ensure reasoning follows a structured process, reporting improvements over single-agent baselines. • CAFES ( Su et al., 2025 ) implemented a collaborative framework with a "Reflective Scorer" that iteratively refines scores based on feedback, achieving high agreement with ground truth in multimodal contexts. • CAELF ( Hong et al., 2025 ) and EvaAI ( Lagakis & Demetriadis, 2024 ) focused on the interactive and human-mimicking aspects of grading, using multiple agents to generate contestable feedback or simulate tutor-student interactions. • Decomposition for Specificity: Kang and Kong (2026) demonstrated the value of decomposition by breaking down "Topic Relevance" into sub-characteristics, allowing for more precise assessment in Chinese composition. Similarly, Chu et al. (2024) introduced Rationale-based Multiple Trait Scoring (RMTS), where separate agents generate qualitative rationales to assist the quantitative scoring model. 2.3 The Architectural Trade-off: Single vs. Multi-Agent Despite the proliferation of multi-agent frameworks, recent empirical evidence suggests that complexity does not always guarantee superior performance. While Wang, Ding, et al. (2025) found multi-agent designs superior for complex rubrics, Li et al. (2025) reported that a single-agent strategy with few-shot prompting outperformed multi-agent strategies in assessing student reflections. Furthermore, Wang, Gopalakrishnan, and Bergner (2025) noted that while multi-agent systems could generate transparent rubrics, they did not improve classification accuracy for collaborative problem-solving items compared to single-agent models. 3 Methodology The overall experimental workflow, comparing the Single-Agent and Multi-Agent pathways across different prompting conditions, is illustrated in Figure 1. Figure 1: High-level overview of the methodology 3.1 Dataset This study uses data extracted from the ASAP 2.0 corpus – designed to support automated essay scoring (AES) research ( Crossley et al., 2025 ). The full ASAP 2.0 comprises 24,278 source-based argumentative essays written by U.S. secondary students (grades 6 - 10) across seven distinct prompts. Crossley et al. (2025) also provided an official rubric used by expert raters. The rubric, adapted from Scholastic Aptitude Test (SAT), used a 1–6 scale that had interval levels with a score of 6 indicating consistent mastery of writing. From this corpus, we employed stratified sampling to extract a representative test set of 450 essays. Additionally, for few-shot prompting, we curated a separate set of "calibration examples" (2 examples per score level, totaling 12 essays), ensuring no overlap between the calibration and test data. We also used the official full rubric in our prompts. 3.2 Experiment We compared two distinct architectural approaches to automated grading: a Monolithic (Single-Agent) approach and a Hierarchical (Multi-Agent) approach. Both architectures were tested in Zero-Shot (rubric only) and Few-Shot (rubric + calibration examples) environments. All experiments were conducted using the GPT-5.1 model via the OpenAI API. 3.2.1 Baseline: Single-Agent Architecture In the single-agent configuration, the model acted as the sole expert examiner. It was provided with the full rubric text and asked to output a holistic score (1–6) and a rationale. • Zero-Shot: The model relied solely on the full rubric. • Few-Shot: The