Dean of LLM Tutors: Exploring Comprehensive and Automated Evaluation of LLM-generated Educational Feedback via LLM Feedback Evaluators
Keyang Qian, Yixin Cheng, Rui Guan, Wei Dai, Flora Jin, Kaixun Yang, Sadia Nawaz, Zachari Swiecki, Guanliang Chen, Lixiang Yan, Dragan Gašević · 2025
Read Dean of LLM Tutors: Exploring Comprehensive and Automated Evaluation of LLM-generated Educational Feedback via LLM Feedback Evaluators by Keyang Qian, Yixin Cheng, Rui Guan, Wei Dai, Flora Jin, Kaixun Yang, Sadia Nawaz, Zachari Swiecki, Guanliang Chen, Lixiang Yan, Dragan Gašević (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-05952.txt.
Introduction
Keyang Qian and co-authors build, in a 2025 paper whose head titles the system DeanLLM, an automated reviewer for feedback written by AI tutors. Language model tutors now draft comments on student work at scale. Most research checks how feedback is made, not whether it is any good.
Weak feedback teaches little, and invented claims can mislead students outright. The framework grades feedback on sixteen dimensions spanning content, teaching value, and hallucination risk. Human experts scored sample feedback on practice computer science assignments to validate the grades. The team then tested whether language models can apply the rubric themselves. A working dean for AI tutors could screen comments before students ever see them.
Catalog record
| CREATOR | Keyang Qian, Yixin Cheng, Rui Guan, Wei Dai, Flora Jin, Kaixun Yang, Sadia Nawaz, Zachari Swiecki, Guanliang Chen, Lixiang Yan, Dragan Gašević |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-05952
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Dean of LLM Tutors: Exploring Comprehensive and Automated Evaluation of LLM-generated Educational Feedback via LLM Feedback Evaluators Keyang Qian, Yixin Cheng, Rui Guan, Wei Dai, Flora Jin, Kaixun Yang, Sadia Nawaz, Zachari Swiecki, Guanliang Chen, Lixiang Yan, Dragan Gašević Abstract The use of Large Language Model (LLM) based tutors to provide automated educational feedback to students on student assignment submissions has received much attention in the AI in Education (AIED) field. However, the stochastic nature and tendency for hallucinations in LLMs can undermine both quality of learning experience and adherence to ethical standards. To address this concern, we propose a method that uses LLM feedback evaluators ( DeanLLMs ) to automatically and comprehensively evaluate feedback generated by LLM tutor for submissions on university assignments before it is delivered to students. This allows low-quality feedback to be rejected and enables LLM tutors to improve the feedback they generated based on the evaluation results. We first proposed a comprehensive evaluation framework for LLM-generated educational feedback, comprising six dimensions for feedback content, seven for feedback effectiveness, and three for hallucination types. Next, we generated a virtual assignment submission dataset (n=200 assignment submissions) covering 85 university assignments from 43 computer science courses using eight commonly used commercial LLMs. We labelled and open-sourced the assignment dataset to support the fine-tuning and evaluation of LLM feedback evaluators. Our findings show that o3-pro demonstrated the best performance in zero-shot labelling of feedback (Accuracy 74.3%, F1-score 74.5%) while o4-mini demonstrated the best performance in few-shot labelling of feedback (Accuracy 74.9%, F1-score 75.2%). Moreover, GPT-4.1 achieved human expert level performance after fine-tuning (Accuracy 79.8%, F1-score 79.4%; human average Accuracy 78.3%, F1-score 82.6%). Finally, we used our best-performance model to evaluate 2,000 assignment feedback instances generated by 10 common commercial LLMs, 200 each, to compare the quality of feedback generated by different LLMs. Gemini 2.5 Pro demonstrated the highest feedback quality and 0-detection of hallucination issues. Small LLMs like GPT-4.1 nano and Gemini 2.0 Flash-Lite underperformed in our evaluation. Our LLM feedback evaluator method advances our ability to automatically provide high-quality and reliable educational feedback to students. Introduction Since feedback plays a crucial role in student success, delivering it in a timely and high-quality manner poses a substantial logistical problem ( Middleton et al. 2023 ) , particularly in large computing classes facing strict resource constraints ( Nguyen et al. 2014 ; of Sciences et al. 2018 ) . Consequently, the application of Large Language Models (LLMs) to generate automated feedback has emerged as a key focus areas in educational technology research ( Kim et al. 2025 ; Rudolph, Tan, and Tan 2023 ; Dai et al. 2024a ) , especially in the field of computer science education ( Denny et al. 2024 ; Balse et al. 2023 ; Hellas et al. 2023 ; Liffiton et al. 2023 ; Leinonen et al. 2023 ; Pankiewicz and Baker 2023 ; Phung et al. 2023a ; Phung et al. 2023b ; Prather et al. 2023 ) . While the development of LLMs shows great promise for generating educational feedback, researchers have raised concerns about hallucinations often found in LLM-generated content ( Yan et al. 2024 ) . Hallucinations refer to the phenomenon where LLMs generate content that is irrelevant, fabricated, or inconsistent ( Lakera AI 2024 ) . A heated debate is ongoing within the GenAI research community whether hallucinations are an inherent aspect of LLM generation ( Sangaré 2023 ) . One prevalent perspective ( Xu, Jain, and Kankanhalli 2024 ; Banerjee, Agarwal, and Singla 2024 ) suggest that hallucinations are unavoidable in LLMs. However, limited studies have focused on mitigating this issue in LLM-generated educational feedback. Furthermore, LLM generation may produce content that lacks relevance or meaning even in the absence of hallucinations ( Zhao et al. 2023 ) . Therefore, it is necessary to develop robust methodological approaches to automatically assess the quality of feedback generated by an LLM tutor (i.e., a LLM who serves like a human tutor to give educational feedback to students) before presenting it to students. Such methods can be utilised to establish a Dean system for LLM tutors. The Dean screens out low-quality and hallucinated feedback, prompts regeneration by the LLM tutor, and selects high-quality feedback from multiple generated options, to ensure that only high-quality feedback is delivered to students. Despite extensive research on feedback generation by LLM-based tutors ( Kim et al. 2025 ; Dai et al. 2024a ; Denny et al. 2024 ) , limited research has focused on automated evaluation of feedback generated by LLM tutors based on the feedback text itself. Lin et al. 2024 and Scarlatos et al. 2024 utilised LLMs to evaluate LLM-generated feedback for single dimensions, specifically praise components and correctness. Osakwe et al. 2022 employed four binary classifiers to assess feedback effectiveness across task, process, self-regulation, and self levels. Dai et al. 2024b and Dai et al. 2025 expanded evaluation frameworks to include readability and content-related dimensions, leveraging LLMs for assessment. However, they overlooked hallucination dimensions ( Zhang et al. 2023b ; Rawte, Sheth, and Das 2023 ) , crucial for reliability, safety, and trustworthiness in educational feedback ( Alfredo et al. 2024 ) . Detecting hallucinations is feasible for LLMs ( Zhang et al. 2023a ; Rawte, Sheth, and Das 2023 ; Kaddour et al. 2023 ; Dhuliawala et al. 2023 ) . The current study extends previous work on feedback quality evaluation principles by proposing a comprehensive evaluation framework for LLM tutor educational feedback, and labelled a feedback dataset according to the framework, which results will become open-source. Based on that, we explored different methods of letting LLMs serve as the Dean of LLM tutors to evaluate LLM tutor feedback. Our contributions can be summarised as follows: · We proposed a comprehensive evaluation framework for LLM tutor educational feedback with 16 dimensions, comprising six dimensions for feedback content, seven for feedback effectiveness, and three for hallucination types. · According to this evaluation framework, we labelled feedback generated by 8 common commercial LLMs for 200 assignment submissions spanning 85 university assignments from 43 computer science courses. Since the AAAI26 anonymous submission does not allow insertion of hyperlinks, the dataset and codes are put in supplementary materials, and will be made publicly available upon acceptance. · We found that a fine-tuned few-shot prompting GPT-4.1 model can achieve the same accuracy and F1 performance as human experts in evaluating LLM tutor feedback. · Using the model, we comprehensively compared feedback quality of LLM tutors with difference common commercial LLMs. · The ’Dean of LLM Tutors’ framework represents a significant advance in our ability to deliver reliable and high-quality educational feedback to students automatically in a scalable way. Figure 1: Overview of experiment process and data flow. Method Comprehensive LLM Feedback Evaluation Framework We propose our framework that aims to comprehensively evaluate the generated feedback, with the following dimensions that are widely recognised evaluation dimensions in the feedback and LLM literature. The supporting literature are included for each dimension as follows. · Feedback content dimensions. This aspect evaluate features of feedback content that are proved to be good feedback features for students in the literature. These dimensions include: 1). Alignment with goals. This evaluates to what extent most