Ensemble ToT of LLMs and Its Application to Automatic Grading System for Supporting Self-Learning

Yuki Ito, Qiang Ma · 2025

Read Ensemble ToT of LLMs and Its Application to Automatic Grading System for Supporting Self-Learning by Yuki Ito, Qiang Ma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-16399.txt.

Introduction

Yuki Ito and Qiang Ma present a 2025 paper on an automatic grading system that helps students learn on their own. Fast, detailed feedback is vital for self-study, yet grading by hand is slow. Systems that grade with a single language model are promising but limited by that one model's habits.

Their framework joins several models in three steps: check each model's grading leanings, gather candidate grades, then refine them into one result through a simulated debate. The grading system built on it gives scores that are both accurate and explainable. Tests show the team approach beats single-model grading.

Catalog record

CREATORYuki Ito, Qiang Ma
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-16399

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Ensemble ToT of LLMs and Its Application to Automatic Grading System for Supporting Self-Learning Yuki Ito, Qiang Ma Abstract Providing students with detailed and timely grading feedback is essential for self-learning. While existing LLM-based grading systems are promising, most of them rely on one single model, which limits their performance. To address this, we propose Ensemble Tree-of-Thought (ToT) , a framework that enhances LLM outputs by integrating multiple models. Using this framework, we develop a grading system. Ensemble ToT follows three steps: (1) analyzing LLM performance, (2) generating candidate answers, and (3) refining them into a final result. Based on this, our grading system first evaluates the grading tendencies of LLMs, then generates multiple results, and finally integrates them via a simulated debate. Experimental results demonstrate our approach’s ability to provide accurate and explainable grading by effectively coordinating multiple LLMs. A Preprint Keywords Large Language Models \cdot EdTech \cdot Ensemble \cdot ToT. 1 Introduction Since the release of ChatGPT in 2022, generative AI has gained significant popularity, with many students increasingly adopting it as a learning aid for various purposes. A survey on students’ use of AI for learning identified several common applications, such as brainstorming, generating ideas, summarizing readings, and analyzing large datasets ( Chan and Hu, 2023 ) . Among these, AI’s ability to provide personalized and immediate learning support is particularly valued. Notably, some students in the survey reported using AI to obtain grading and its explanations on their completed homework. They found these grading comments helpful for enhancing their depth of thinking and understanding. This highlights the importance of providing students with detailed, point-by-point feedback on their homework to facilitate self-learning. However, generative AI responses are not always reliable, raising concerns about the potential for misleading feedback. While feedback from human teachers is ideal for guiding student self-learning, it is often impractical for teachers to provide individualized and immediate support to every student. In this context, automatic grading systems emerge as a practical solution. These systems evaluate student answers and deliver timely grading comments, bridging the gap without placing additional burdens on teachers. Before this work, several studies have proposed automatic grading methods. For example, Gobbo et al. introduced GradeAid, a framework that evaluates student answers by extracting both lexical and semantic features to determine grading scores ( del Gobbo et al., 2023 ) . Similarly, Huang et al. developed a system to grade reading comprehension answers ( Huang et al., 2023 ) . Their system not only evaluates student answers but also provides auto-generated hints to guide students toward correct answers. Despite these advances, existing automated grading systems often fall short of effectively supporting self-learning. Many systems lack the ability to provide high-quality, detailed feedback necessary for guiding students’ learning processes. For instance, GradeAid does not explain why an answer is incorrect or incomplete. This leaves students without clear guidance on how to improve. While some methods attempt to generate more detailed explanations about their grading results, the quality remains inadequate. For example, Huang et al. say about their system that their auto-generated feedback quality is “still far from satisfactory." These deficiencies hinder the effectiveness of the systems for self-learning. To overcome these challenges, integrating multiple LLMs into grading systems is a promising approach. Existing methods typically rely on a single LLM, which may struggle to provide consistently accurate and reliable grading comments. In contrast, a previous study demonstrated that using ensembles of pre-trained BERT models for scoring tasks significantly improves prediction accuracy ( Ormerod, 2022 ) . This highlights the advantages of combining multiple models. Inspired by this insight, we advocate for leveraging multi-LLM systems to address the limitations of single-model approaches. However, an effective way to combine multiple LLMs, especially language generation models, has not yet been well studied. In this paper, we propose a novel framework to utilize multiple LLMs called Ensemble ToT and its application, a grading system named Graders by Ensemble ToT (GET) . Ensemble ToT combines the strengths of multiple language generation models by integrating ensemble learning techniques with the Tree-of-Thought (ToT) approach. Fig. 1 shows the overview of this framework. Figure 1: Overview of Ensemble ToT Framework : The framework integrates ensemble learning techniques with the Tree-of-Thought (ToT) approach. It identifies individual LLM performance tendencies and synthesizes multiple candidate solutions generated by LLMs into a single refined result. It operates in three key steps: 1. Identification of LLM performance tendencies : Inspired by ensemble learning principles, we evaluate the performance of individual LLMs on tasks similar to the target problem. This helps us identify the strengths and tendencies of each LLM for the given task. 2. Generation of multiple candidate solutions : Drawing on the ToT framework, multiple LLMs independently generate candidate solutions for the task. 3. Integration of the solutions : These candidate solutions are synthesized into a single, refined result, leveraging insights from the performance tendencies. The GET system leverages the Ensemble ToT framework to deliver highly accurate grading and its detailed explanations — both of which are difficult to achieve using one single LLM. Fig. 2 provides the overview of the GET system process flow. Figure 2: Process Diagram of GET : The system consists of three stages: pseudo-learning , multi-LLM grading , and debate Integration . This enables accurate grading by taking advantage of the characteristics of multiple models. Following the Ensemble ToT structure, it operates in three main steps: pseudo-learning , multi-LLM grading , debate integration . In pseudo-learning , three LLMs separately grade past student answers and determine their grading labels. By analyzing the results of this task, grading capabilities and tendencies of LLMs are identified. Next, in the multi-LLM grading step, the three LLMs separately generate grading candidates of student answers. They get questions, reference answers, and student answers as the inputs and output grading labels along with their reasons. Finally, in the debate integration step, an LLM integrates multi-LLM grading results into the single, refined grading conclusion. This integration is accomplished by simulating a debate script between the grader LLMs. Pseudo-learning ’s outcome is utilized in this integration phase to guide debate outputs. By providing students with grading reasons accurately and speedily, the GET system enhances the explainability of grading outcomes, helping students understand and correct their mistakes. Furthermore, by combining the output results from multiple LLMs, the system compensates for the weaknesses of individual models. This advancement enables more accurate and balanced grading, accompanied by clear and valid reasoning for student answers. It is expected to enhance student self-learning while minimizing the workload for educators. The key contributions of this work are as follows: • Propose Ensemble ToT, a novel framework to collaborate multiple LLMs effectively. It contains three main steps. – Analysis of LLMs performance tendencies inspired by ensemble learning. – Generation of several candidate solutions with multiple LLMs based on the Tree-of-Thought framework. – Integration of candidate solutions considering the performance tendencies. • Propose an automated grading system, GET, which is constructed based on the Ensemble ToT framework. • Evaluate the grading performance of the GET system on existing datasets. By comparing it to several baselines, we guarantee its effectiveness. 2 Related Work 2.1 Methods to Collaborate Multiple LLMs Recent advancements in large language models (LLMs) have

Read the complete text