Conversational Education at Scale: A Multi-LLM Agent Workflow for Procedural Learning and Pedagogic Quality Assessment

Jiahuan Pei, Fanghua Ye, Xin Sun, Wentao Deng, Koen Hindriks, Junxiao Wang · 2025

Read Conversational Education at Scale: A Multi-LLM Agent Workflow for Procedural Learning and Pedagogic Quality Assessment by Jiahuan Pei, Fanghua Ye, Xin Sun, Wentao Deng, Koen Hindriks, Junxiao Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-05528.txt.

Introduction

Jiahuan Pei and co-authors build a multi-agent workflow that stages teaching chats between language models and grades their quality, in a paper from 2025. Virtual teachers and learners have grown fast, but work so far rarely spans large, varied course content or checks teaching quality with care.

Their system, WikiHowAgent, casts models as teacher and learner, with a manager running the exchange and an evaluator scoring it. The focus is step-by-step skills drawn from how-to guides. The team built a dataset of over one hundred thousand chats grounded in thousands of tutorials across many domains. Scoring mixes automatic metrics, rubric judgments, and human checks. Results across setups show the workflow works and reveal where model teaching is strong or weak.

Catalog record

CREATORJiahuan Pei, Fanghua Ye, Xin Sun, Wentao Deng, Koen Hindriks, Junxiao Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-05528

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Conversational Education at Scale: A Multi-LLM Agent Workflow for Procedural Learning and Pedagogic Quality Assessment Jiahuan Pei, Fanghua Ye, Xin Sun, Wentao Deng, Koen Hindriks, Junxiao Wang Abstract Large language models (LLMs) have advanced virtual educators and learners, bridging NLP with AI4Education. Existing work often lacks scalability and fails to leverage diverse, large-scale course content, with limited frameworks for assessing pedagogic quality. To this end, we propose WikiHowAgent , a multi-agent workflow leveraging LLMs to simulate interactive teaching-learning conversations. It integrates teacher and learner agents, an interaction manager, and an evaluator to facilitate procedural learning and assess pedagogic quality. We introduce a dataset of 114,296 teacher-learner conversations grounded in 14,287 tutorials across 17 domains and 727 topics. Our evaluation protocol combines computational and rubric-based metrics with human judgment alignment. Results demonstrate the workflow’s effectiveness in diverse setups, offering insights into LLM capabilities across domains. Our datasets and implementations are fully open-sourced. 1 Introduction \Acp LLM have sparked smart teaching and learning, reshaping education and bridging NLP with AI4Education Chu et al. 2025 . \Acp LLM have been employed as single agents in education, serving as teachers Chen et al. 2024b ; Lan and Chen 2024 , learners Xu et al. 2025a , or evaluators Hu et al. 2025 to enhance personalized learning Park et al. 2024 . Specifically, multi-LLMs have shown promise in simulating teacher-learner interactions, assisting competitive problem solving Islam et al. 2024 , providing personalized feedback Park et al. 2024 , immersive multimodal training Pei et al. 2024 , and evaluating educational content Lagakis and Demetriadis 2024 ; Jinxin et al. 2023 . However, none of the existing approaches use diverse and large-scale course content, which is vital for capturing the variability in instructional materials. And they lack a comprehensive evaluation for assessing pedagogic quality, limiting insights into the effectiveness of teaching-learning interactions Xu et al. 2025a . Figure 1: The proposed multi-LLM agent workflow for procedural learning and pedagogic quality assessment. To this end, we introduce a multi-agent workflow, namely WikiHowAgent (See Figure 1 ), leveraging LLM to simulate an interactive teaching-learning conversation graph. It utilizes LLM-powered teacher and learner agents, processes instructional tutorials, and generates guided conversations in an interactive mode. The workflow is designed to facilitate a seamless interaction between the teacher and learner agents, allowing for a natural flow of conversation. The workflow consists of four main components: (1) Teacher Agent , which provides instructions, answers clarification questions, and guides tutorial progression. (2) Learner Agent , which simulates understanding and generates responses, either feedback or a question if anything is unclear. (3) Interaction Manager , which monitors the conversation state, tracks tutorial progress, and determines the next node (i.e., teacher , learner , end ) in the graph, ensuring seamless transitions. And (4) Evaluator , which assesses the generated conversation using diverse evaluation metrics, including computational metrics and rubric-based metrics generated by an LLM-based evaluator agent, offering insights into pedagogic quality and model performance. We seek the answers to the following research questions: ( RQ1 ) How effective is the proposed multi-agent workflow with diverse LLM ? How does it perform in homogeneous learning (the same LLM for teacher, learner, and evaluator) versus heterogeneous learning (different LLMs for learner given the same teacher)? ( RQ2 ) How do LLMs perform across various domains, and what insights can be drawn about their strengths and limitations in specific areas of procedural learning? ( RQ3 ) How well do the diverse automatic evaluation metrics align with human judgment in assessing pedagogic quality and conversational performance? We list our key contributions as follows: • A large-scale dataset consisting of 114,296 teacher-learner conversations grounded in 14,287 tutorials. It spans 17 domains and 727 topics, organized in a hierarchical knowledge graph. • A multi-LLM agent workflow for procedural learning and pedagogic quality assessment. A benchmark of 8 prevailing LLMs from diverse providers, including open-source and closed-source models. • A comprehensive evaluation protocol, including diverse metrics and a human judgment alignment assessment method. 2 Multi-LLM Agent Procedural Learning Algorithm 1 Multi-LLM Agent Workflow for Procedural Learning 1: Tutorial \mathcal{T}=[t_{1},t_{2},\dots,t_{n}] , max interactions M 2: Initialize State: 3: \mathcal{S}\leftarrow\{\text{messages: }[],\text{tutorial: }\mathcal{T},\text{current\_step: }0, 4: \text{needs\_clarification: }\text{False},\text{finished: }\text{False}\} 5: Define Nodes: 6: \text{TeacherNode: }\mathcal{S}\leftarrow\text{teacher\_agent}(\mathcal{S}) 7: \text{LearnerNode: }\mathcal{S}\leftarrow\text{learner\_agent}(\mathcal{S}) 8: Simulate a conversation by iterating through the nodes in the graph: 9: while \neg\mathcal{S}.\text{finished} and |\mathcal{S}.\text{messages}|<M do 10: if \mathcal{S}.\text{messages}=[] then 11: \mathcal{S}\leftarrow\text{TeacherNode}(\mathcal{S}) \triangleright Provide first instruction 12: else if \mathcal{S}.\text{messages}[-1]\in Teacher then 13: \mathcal{S}\leftarrow\text{LearnerNode}(\mathcal{S}) \triangleright Generate learner response 14: if Learner asks a question and not mention “next step” then 15: \mathcal{S}.\text{needs\_clarification}\leftarrow\text{True} \triangleright Set flag for Teacher 16: else 17: \mathcal{S}.\text{current\_step}\leftarrow\mathcal{S}.\text{current\_step}+1 \triangleright Next learning step 18: end if 19: else 20: \mathcal{S}\leftarrow\text{TeacherNode}(\mathcal{S}) \triangleright Provide teacher instruction 21: if \mathcal{S}.\text{needs\_clarification} then 22: \mathcal{S}.\text{needs\_clarification}\leftarrow\text{False} \triangleright Reset flag after clarification 23: end if 24: end if 25: if \mathcal{S}.\text{current\_step}\geq|\mathcal{T}| then 26: \mathcal{S}.\text{finished}\leftarrow\text{True} \triangleright End conversation if tutorial is complete 27: end if 28: end while 29: Evaluate conversation: 30: \mathcal{E}\leftarrow\text{Evaluator}(\mathcal{S}.\text{messages}) 31: return \mathcal{S}.\text{messages},\mathcal{E} As shown in Figure 1 , we propose a multi-LLM agent workflow designed for procedural learning through interactive dialogue. Algorithm 1 outlines the implementation of the workflow, comprising a teacher agent, a learner agent, and an evaluator agent, to dynamically simulate guided tutorial sessions using a conversational graph and assess pedagogic quality. First, we initialize the learning state and conversation graph, where each node represents a conversational turn, and edges represent interactions between the teacher and learner agents. Then, we define the nodes, invoking either the teacher agent or the learner agent: The teacher agent is responsible for providing instructions and answering clarification questions. The learner agent simulates a learner’s understanding and generates follow-up questions or responses. Next, we simulate the conversation by iterating through the nodes in the graph: (1) It begins with the teacher agent providing instructions based on the first-step’s tutorial content. (2) The interaction manager monitors the conversation state, tracks tutorial progress, handles clarification needs, and determines the next node in the graph. (3) It continues iteratively, with the teacher agent providing instructions and the learner agent generating responses until the tutorial is completed or a maximum interaction limit is reached. Last, the evaluator assesses the generated conversation using a range of evaluation metrics, including computational metrics and rubric-based metrics facilitated by an LLM-based evaluator agent. 3 Evaluation Protocol 3.1 Evaluation Metrics 3.1.1 Computational Metrics We utilize a set of automatic metrics, including: (1) Question : Computes the proportion of learner utterances that contain questions. (2) Completion : Computes the proportion of generated conversation reached completion by generating the special token finished in the end. (3) Diversity : Computes a 2-gram diversity score for linguistic variation in responses. (4) BLEU ( Papineni et al. 2002 ) : Measures the overlap of 4-grams between the generated and reference tutorial, focusing on precision. (5) METEOR ( Banerjee and Lavie 2005 ) : Evaluates 4-gram similarity using a harmonic mean of precision and recall, along with alignment of words, including synonyms and stemming. (6) ROUGE ( Lin 2004 ) : Assesses the quality of generated text by comparing overlapping 4-grams, word sequences, and word pairs with reference tutorial, focusing on recall. (7) BERTScore Reimers and Gurevych 2019 : Measures context-aware semantic similarity between the

Read the complete text