Training LLM-based Tutors to Improve Student Learning Outcomes in Dialogues

Alexander Scarlatos, Naiming Liu, Jaewook Lee, Richard Baraniuk, Andrew Lan · 2025

Read Training LLM-based Tutors to Improve Student Learning Outcomes in Dialogues by Alexander Scarlatos, Naiming Liu, Jaewook Lee, Richard Baraniuk, Andrew Lan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06424.txt.

Introduction

An AI tutor can sound wise while the student learns little, since it was trained to sound like a teacher, not to cause learning. Alexander Scarlatos and four co-authors retrain the tutor on outcomes in 2025. For each turn, the system drafts several replies and grades them two ways.

One grade comes from a student model that forecasts the chance of a right answer next. The other is a teaching rubric scored by a large model. Wins and losses among the drafts then teach an open model by direct preference tuning. The result speaks to lift the odds of a right reply while keeping sound teaching form. Trials in live chats test the claim.

Catalog record

CREATORAlexander Scarlatos, Naiming Liu, Jaewook Lee, Richard Baraniuk, Andrew Lan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-06424

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Training LLM-based Tutors to Improve Student Learning Outcomes in Dialogues Alexander Scarlatos, Naiming Liu, Jaewook Lee, Richard Baraniuk, Andrew Lan Abstract Generative artificial intelligence (AI) has the potential to scale up personalized tutoring through large language models (LLMs). Recent AI tutors are adapted for the tutoring task by training or prompting LLMs to follow effective pedagogical principles, though they are not trained to maximize student learning throughout the course of a dialogue. Therefore, they may engage with students in a suboptimal way. We address this limitation by introducing an approach to train LLMs to generate tutor utterances that maximize the likelihood of student correctness, while still encouraging the model to follow good pedagogical practice. Specifically, we generate a set of candidate tutor utterances and score them using (1) an LLM-based student model to predict the chance of correct student responses and (2) a pedagogical rubric evaluated by GPT-4o. We then use the resulting data to train an open-source LLM, Llama 3.1 8B, using direct preference optimization. We show that tutor utterances generated by our model lead to significantly higher chances of correct student responses while maintaining the pedagogical quality of GPT-4o. We also conduct qualitative analyses and a human evaluation to demonstrate that our model generates high quality tutor utterances. 1 1 1 Our code is available at https://github.com/umass-ml4ed/tutorbot-dpo Published in AIED 2025: The 26th International Conference on Artificial Intelligence in Education Keywords Large Language Models \cdot Math Education \cdot Reinforcement Learning \cdot Tutor-Student Dialogues. 1 Introduction Recent advances in generative artificial intelligence (AI), including large language models (LLMs), have opened new possibilities in education and in particular on scaling up personalization. One form of personalization that generative AI powers is interactive learning via tutoring dialogues between AI-powered tutors and students. These interactions have the potential to tailor instruction to each student’s needs and progress, while offering personalized feedback, all in real time, in a scalable way. Given the widespread success of human tutors for improving student outcomes [ 30 ] , many recent works have developed LLM-based tutors, showing promise across various educational domains [ 15 , 26 , 31 , 33 , 34 , 40 , 43 , 51 ] . Many LLM-based tutors are even deployed in practice, such as Khan Academy’s Khanmigo [ 22 ] and Carnegie Learning’s LiveHint [ 4 ] . Several preliminary studies have shown that interacting with LLMs can increase student learning [ 53 ] , although some have shown that students can develop an over-reliance on LLMs which negatively impacts their learning [ 24 ] . Many prior works have focused on improving LLMs’ ability to follow effective tutoring principles, adapting them for the tutoring task that they are not pre-trained for. One approach, explored in [ 47 ] , analyzes the decision-making process underlying human tutor utterances, showing that integrating expert decisions enhances LLM-based tutoring. Another study, [ 29 ] , examines tutor moves in interactions with an LLM-powered simulated student agent, demonstrating that move annotation data contributes to better tutoring performance. Similarly, [ 45 ] investigates the role of AI roleplay in generating synthetic tutoring data and finds that fine-tuning LLMs on this data, along with human tutor-student interactions, significantly improves their pedagogical effectiveness. Moreover, [ 9 ] introduces the concept of tutor uptake—acknowledging student responses—as a valuable strategy for LLMs to adopt. While these efforts offer valuable insights into how LLMs can emulate effective human tutoring strategies, the question remains whether such approaches truly maximize student learning outcomes. Rather, a data-driven approach, where student outcomes form a reward signal, could potentially lead to AI tutors that are more aligned with educational goals. Additionally, current approaches often rely on large proprietary LLMs, coming with many downsides; they are not fully controllable and cannot be easily customized, can be costly to use, and relinquish control of private student data. Therefore, developing effective AI tutors with smaller, open-source LLMs remains an important goal. 1.1 Contributions In this paper, we propose a novel approach to train a small, open-source LLM to not only follow effective pedagogical principles, but directly optimize for student learning outcomes. Our method performs three key steps. First, at each dialogue turn, we gather multiple candidate tutor utterances from a variety of sources, including human tutors and LLMs with varying styles and sizes. Second, we evaluate each candidate utterance on two aspects: (1) whether it elicits a correct response in the next student turn, using a trained student model for dialogues to predict student behavior, and (2) whether it adheres to a set of effective pedagogical principles, using GPT-4o in a LLM-as-a-judge evaluation setup. Third, we contrast good candidate utterances with poor ones, and fine-tune Llama 3.1 8B [ 10 ] with offline Reinforcement Learning (RL), specifically Direct Preference Optimization (DPO) [ 36 ] . We demonstrate that our optimized LLM tutor significantly increases the likelihood of next turn student response correctness, while reaching comparable pedagogical quality to that of a much larger, proprietary LLM, GPT-4o [ 32 ] . Through qualitative analysis and human evaluation, we confirm that our approach produces high-quality tutor utterances and reveal emergent tutoring strategies that arise from our training approach. We acknowledge up front that the most significant limitation of our work is that we do not experiment with real students. Since access to students at the scale necessary for our work is beyond our capability, we use a simulated student model instead. While we believe that our work is a reasonable starting point to train LLM-based tutors to maximize student outcomes, future work with real students in the loop is highly important. Therefore, to facilitate further research, we publicly release our code and encourage researchers and practitioners with access to real-world tutoring dialogue settings to build on our work. 2 Related Work AI Tutors in Dialogues There is a long history of AI-based tutors in education that interact with students through dialogues. Early systems, such as Cognitive Tutors, construct cognitive models of students to provide targeted feedback [ 1 ] . AutoTutor engages students by asking targeted questions, and assesses student correctness using latent semantic analysis [ 14 ] . Why2-Atlas converts a student response to a proof, which it uses to identify misconceptions and guide a dialogue [ 46 ] . While these systems were often effective for improving student learning, they required significant engineering and had limited flexibility. In contrast, recent LLM tutors can more easily adapt to new contexts, interpret student responses, and cater personalized content towards the student. Several LLM tutors are implemented through refined prompt engineering [ 22 ] , with some taking on specialized roles such as teachable agents [ 40 ] or “co-pilots” for human tutors [ 48 ] . Other works fine-tune LLM tutors to enhance the pedagogical capabilities over the base models. A common approach is to generate simulated dialogues, where the tutor utterances are constructed to follow good pedagogical practice, and fine-tune on those [ 41 , 45 ] . Several works also generate examples of low quality tutor utterances and use them as negative samples in DPO training to improve over fine-tuning [ 2 , 42 ] . Student Outcome Modeling While there are many ways of measuring student outcomes, in

Read the complete text