From Problem-Solving to Teaching Problem-Solving: Aligning LLMs with Pedagogy using Reinforcement Learning

David Dinucu-Jianu, Jakub Macina, Nico Daheim, Ido Hakimi Iryna Gurevych Mrinmaya Sachan · 2025

Read From Problem-Solving to Teaching Problem-Solving: Aligning LLMs with Pedagogy using Reinforcement Learning by David Dinucu-Jianu, Jakub Macina, Nico Daheim, Ido Hakimi Iryna Gurevych Mrinmaya Sachan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15607.txt.

Introduction

David Dinucu-Jianu, Mrinmaya Sachan, and four co-authors train language models to teach, not just solve. Their paper, from twenty twenty-five, starts from a clash. Models tuned to answer at once make poor tutors, since good teaching holds back the answer. Their method uses trial chats with a fake student to shape the tutor by reward.

A mid-size tutor trained this way matched big paid models on teaching tests. No human labels were needed for the training. A dial trades teaching support against student success, tracing a clear frontier. The tutor keeps its raw reasoning skill better than plain fine-tuned rivals. Optional thinking tags expose its plan, which aids trust.

Catalog record

CREATORDavid Dinucu-Jianu, Jakub Macina, Nico Daheim, Ido Hakimi Iryna Gurevych Mrinmaya Sachan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-15607

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Problem-Solving to Teaching Problem-Solving: Aligning LLMs with Pedagogy using Reinforcement Learning David Dinucu-Jianu, Jakub Macina, Nico Daheim, Ido Hakimi Iryna Gurevych Mrinmaya Sachan Abstract Large language models (LLMs) can transform education, but their optimization for direct question-answering often undermines effective pedagogy which requires strategically withholding answers. To mitigate this, we propose an online reinforcement learning (RL)-based alignment framework that can quickly adapt LLMs into effective tutors using simulated student-tutor interactions by emphasizing pedagogical quality and guided problem-solving over simply giving away answers. We use our method to train a 7B parameter tutor model without human annotations which reaches similar performance to larger proprietary models like LearnLM. We introduce a controllable reward weighting to balance pedagogical support and student solving accuracy, allowing us to trace the Pareto frontier between these two objectives. Our models better preserve reasoning capabilities than single-turn SFT baselines and can optionally enhance interpretability through thinking tags that expose the model’s instructional planning. * * footnotetext: Equal contribution. https://github.com/eth-lre/PedagogicalRL Figure 1: LLM tutoring forms a multi-objective scenario in which LLM tutors should increase the student’s solve rate (y-axis) while minimizing solution leakage (x-axis). Here, the \Delta solve rate measures how often a student can solve a problem before and after the dialog with a tutor and leaked solutions measures how often the tutor tells the solution to the student. Our RL-trained Qwen-2.5-7B models with varying penalty \lambda are on the Pareto-front and match the performance of specialized closed-source models when tutoring on Big-Math. 1 Introduction Large Language Models (LLMs) hold significant promise in education, particularly as personalized tutors capable of guiding students individually through problems. Recent advances have demonstrated remarkable LLM performance in math and science Chervonyi et al. (2025) ; Saab et al. (2024) . However, deploying LLMs effectively as educational tutors involves more than excelling on benchmarks Tack and Piech (2022) ; Gupta et al. (2025) . To be truly effective, a tutor must facilitate learning by guiding students toward independently constructing correct solutions rather than simply revealing the answers. We refer to this shift from assistant to tutor as pedagogical alignment. Achieving robust pedagogical alignment remains an open challenge Macina et al. (2025) ; Maurya et al. (2025) . Approaches that rely on supervised fine-tuning (SFT) Daheim et al. (2024) ; Kwon et al. (2024) can suffer from generalization issues while existing RL-based techniques typically depend on costly, and often proprietary, preference annotations Team et al. (2024) or require a much larger model as a source of training data of tutor responses Sonkar et al. (2024) ; Scarlatos et al. (2025) . Due to these limitations, these prior works have largely focused on single-turn feedback, which fails to capture the multi-turn dynamics that are essential for effective tutoring. Figure 2: Workflow of our RL framework. First, we perform multiple complete student-tutor conversation simulations. After each conversation ends, the reward is computed: 1) post-dialog student solve rate (success) conditioned on the dialog, and 2) the pedagogical quality of the tutor guidance throughout the conversation. This setup uses data from the current tutor model (is on-policy) and does not use offline static dialog data (is online). To address these gaps, we propose a multi-turn reinforcement learning (RL) method that enables the model to learn directly from its own dialogs with a student to find optimal teaching strategies. Grounded in mastery learning and active teaching principles Chi and Wylie (2014) ; Freeman et al. (2014) , our system simulates multi-turn interactions on challenging problems from Big-Math Albalak et al. (2025) , with the tutor LLM using Socratic questioning Shridhar et al. (2022) and targeted hints instead of handing out solutions. We design reward functions that mirror authentic long-term learning outcomes, namely, how often a student can solve a problem after a dialog with the tutor and how much the tutor follows sound pedagogical principles throughout the full conversation. Our key contributions are the following: • Cost-efficient training via synthetic student–tutor interactions: Our online RL method replaces the need for expensive human-annotated data with a synthetic data pipeline, enabling a 7B Tutor Model to almost match the performance of LearnLM. • Controllable pedagogy–accuracy trade-off: Our method enables explicit control over the balance between pedagogical support and student answer correctness by adjusting a penalty weight to navigate a Pareto frontier. • Preservation of reasoning capabilities: Our approach maintains performance across standard reasoning benchmarks, unlike prior methods such as SocraticLM Liu et al. (2024) . Evaluations on MMLU, GSM8K, and MATH demonstrate that pedagogical alignment does not come at the cost of reasoning ability. 2 Related Work 2.1 LLMs for Dialog Tutoring While effective human tutors not only provide answers but more importantly scaffold the learning of students, LLMs are predominantly trained for providing answers which limits their tutoring capabilities Tack and Piech (2022) ; Macina et al. (2023b) . Hence, various approaches have been proposed to improve their pedagogical skills. Arguably the simplest is prompt engineering, where pedagogical criteria are encoded in the prompt, for example, for asking questions ( Sonkar et al., 2023 ; Puech et al., 2025 ) or detecting mistakes ( Wang et al., 2024b ) but it is tedious and sensitive to changes ( Jurenka et al., 2024 ) . A more robust alternative is to use gradient-based updating, for example, SFT on teacher-student dialogs. However, this is challenging because only a few high-quality tutoring datasets exist publicly, for example, MathDial which is semi-synthetically created by pairing LLM students with real teachers for solving math problems ( Macina et al., 2023a ) . Hence, many works resort to synthetic data ( Wang et al., 2024a ) . For example, SocraticLM ( Liu et al., 2024 ) is trained on 35k math tutoring dialogs created using a multi-agent setting and TutorChat ( Chevalier et al., 2024 ) is trained using 80k synthetic teacher–student conversations grounded in textbooks. Larger scale approaches in industry, such as, LearnLM Jurenka et al. (2024) use a mixture of synthetic and human-collected data but this requires substantial resources. Finally, recent works use Reinforcement Learning from Human Feedback (RLHF) ( Ouyang et al., 2022 ) , for example, to improve next tutor dialog act prediction ( Sonkar et al., 2024 ) or to improve math tutors by turn-level rewards using GPT-4-generated preference data ( Scarlatos et al., 2025 ) . However, it is unclear how single-turn synthetic data translates to tutoring more complex multi-turn conversations. Prior works treat tutoring as an offline off-policy problem by relying on large-scale synthetic or proprietary data which introduces exposure bias ( Ross and Bagnell, 2010 ; Ranzato et al., 2016 ) as the tutor does not learn from its own interactions during training. In contrast, our work adopts an online on-policy setup where the model is trained on its own interactions throughout training. 2.2 Dialog as RL Task & Verifiable Rewards Previous work has commonly framed educational dialog as a next teacher utterance generation task, where the teacher’s last turn serves as a ground truth response and the dialog history serves as context Macina et al. (2023a) . However, a dialog is inherently a multi-turn interaction towards a goal (e.g. student learns to solve a problem) and single-turn methods limit the model’s ability to plan across multiple

Read the complete text