Large Language Models Approach Expert Pedagogical Quality in Math Tutoring but Differ in Instructional and Linguistic Profiles

Ramatu Oiza Abdulsalam, Segun Aroyehun · 2025

Read Large Language Models Approach Expert Pedagogical Quality in Math Tutoring but Differ in Instructional and Linguistic Profiles by Ramatu Oiza Abdulsalam, Segun Aroyehun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-20780.txt.

Introduction

Ramatu Oiza Abdulsalam and Segun Aroyehun compare how humans and chatbots tutor mathematics. Expert tutors, novice tutors, and seven large language models all answered the same student mistakes in remedial math dialogues. The team scored teaching moves such as pressing for reasoning and restating student ideas, plus features of word choice and tone.

Experts beat novices, and larger models scored near the experts on average quality. Still, the models leaned less on the discussion moves experts favor, while writing longer and more polite replies. Extra politeness and strong self-directed language actually tracked with lower ratings. The study argues that judging tutoring takes more than checking final answers.

Catalog record

CREATORRamatu Oiza Abdulsalam, Segun Aroyehun
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-20780

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Large Language Models Approach Expert Pedagogical Quality in Math Tutoring but Differ in Instructional and Linguistic Profiles Ramatu Oiza Abdulsalam, Segun Aroyehun Abstract Recent work has explored the use of large language models for generating tutoring responses in mathematics, yet it remains unclear how closely their instructional behavior aligns with expert human practice. We examine this question using a controlled, turn-level comparison in which expert human tutors, novice human tutors, and multiple large language models respond to the same set of math remediation conversation turns. We examine both instructional strategies and linguistic characteristics of tutoring responses, including restating and revoicing, pressing for accuracy, lexical diversity, readability, politeness, and agency. We find that large language models approach expert levels of perceived pedagogical quality on average but exhibit systematic differences in their instructional and linguistic profiles. In particular, large language models tend to underuse restating and revoicing strategies characteristic of expert human tutors, while producing longer, more lexically diverse, and more polite responses. Statistical analyses show that restating and revoicing, lexical diversity, and pressing for accuracy are positively associated with perceived pedagogical quality, whereas higher levels of agentic and polite language are negatively associated. Overall, recent large language models exhibit levels of perceived pedagogical quality comparable to expert human tutors, while relying on different instructional and linguistic strategies. These findings underscore the value of analyzing instructional strategies and linguistic characteristics when evaluating tutoring responses across human tutors and intelligent tutoring systems. 1 Introduction Effective feedback plays a central role in supporting student learning by acknowledging effort, identifying errors, and providing clear guidance for self-correction 12 . In instructional settings, the quality of feedback depends not only on whether mistakes are addressed, but also on how explanations are framed linguistically and pedagogically. As large language models (LLMs) are increasingly used to generate tutoring responses, an important open question is how closely their instructional behavior aligns with that of human tutors when responding to student errors. Prior work 28 ; 22 ; 18 has shown that LLM-generated text exhibits systematic linguistic regularities relative to human writing across domains, including characteristic patterns of lexical choice and reduced stylistic variability. While this work provides valuable descriptive insights into how model-generated language differs from human language, it has largely examined language in settings without reference to task-specific evaluation criteria. As a result, it remains unclear whether such linguistic regularities shape instructional behavior or pedagogical evaluation in tutoring contexts. This paper addresses this gap by comparing expert human tutors, novice human tutors, and multiple large language models responding to the same set of mathematics remediation conversation turns. Specifically, we examine differences in both instructional strategies and linguistic characteristics, including restating and revoicing, pressing for accuracy, lexical diversity, readability, politeness, and agency. We evaluate tutor responses in terms of pedagogical quality, operationalized through structured annotations of error handling and guidance, and examine how pedagogical quality relates to both instructional strategies and linguistic characteristics. This research is guided by three research questions. RQ1: How do instructional strategies and linguistic characteristics differ across expert human tutors, novice human tutors, and large language models in responses to the same mathematics remediation prompts? RQ2: How does perceived pedagogical quality vary across expert human tutors, novice human tutors, and large language models? RQ3: Which instructional strategies and linguistic characteristics are associated with perceived pedagogical quality in tutoring responses? 2 Related Work Recent research has examined the use of large language models (LLMs) as mathematical tutors, with mixed evidence regarding their pedagogical quality. While LLMs can generate fluent and well-structured feedback, human tutors continue to outperform them on core pedagogical functions, particularly the accurate identification and correction of student errors involving conceptual misunderstandings or multi-step reasoning 27 . Related findings show that LLMs may produce feedback that appears appropriate while failing to correctly determine whether a student response is incorrect 8 . Comparative work further indicates differences in how human and LLM tutors structure feedback. Both tend to rely on hint-based guidance rather than direct solutions, but LLMs more often provide compound feedback, whereas human tutors typically deliver focused, single-action interventions 10 . These studies document differences in feedback structure but do not examine whether such differences correspond to systematic linguistic properties or how they relate to pedagogical quality. Recent research comparing human and LLM-generated text report stable linguistic differences in evaluative and explanatory writing, including differences in lexical choice, syntactic patterns, and discourse organisation 28 ; 22 ; 18 . This literature does not examine instructional feedback and does not evaluate pedagogical quality. Consequently, it remains unknown whether similar linguistic regularities appear in pedagogical feedback or how such regularities might align with a tutor’s ability to identify and correct student errors. Evaluations of tutoring interactions often focus on traits such as engagement, empathy, scaffolding, and conciseness 4 . While relevant to the tutoring experience, these traits are typically assessed independently of correctness and error identification and therefore capture only a subset of pedagogical quality. The foundations of pedagogically effective feedback are grounded in human tutoring practices, particularly those of expert tutors who adapt their responses to the specific nature of a student’s mistake 26 . Prior work suggests that LLMs can produce feedback that is more extensive and readable than that of human tutors 21 , and pedagogical research consistently emphasises the role of feedback in supporting learning 6 . At the same time, the tendency of LLMs to generate fluent but incorrect responses raises concerns in educational settings, where learners may rely on linguistic fluency as a cue for correctness. Recent evaluations of LLMs on mathematical reasoning tasks address these concerns by assessing models’ ability to detect, classify, and correct student errors 13 . Although LLMs can perform some of these tasks in isolation, attempts to prompt them to emulate expert tutors reveal limitations in adapting feedback to student misconceptions compared to human experts 14 . Educational research has long established that accurately identifying what is correct or incorrect in student work is a necessary condition for pedagogical feedback that supports learning 19 ; 3 . 3 Methodology 3.1 Data The dataset contains 300 teacher-student dialogues at middle school level 15 and at elementary level 27 . The dataset includes 2476 individual conversations where a student presents a mathematical mistake and receives responses from multiple tutors. Each response includes annotations that identify pedagogically relevant features such as mistake identification, mistake location, providing guidance and actionability of the tutor’s response 16 . Each conversation features eight to nine responses generated from two human tutors (an Expert and a Novice) and the remainder generated by various large language models, including GPT-4 2 , Gemini 25 , Sonnet(Anthropic), Mistral 7B 7 , Llama-3.1-405B and two light weight models Llama-3.1-8B 5 and Phi-3 1 . The structure of the dataset makes it well-suited for studying how human and LLM tutors differ in both linguistic expression (e.g., lexical complexity and politeness) and pedagogical effectiveness, thus enabling systematic comparison. Dataset Annotations The dataset consists of human annotation of responses using a set of pedagogical dimensions described in prior work (Maurya et al., 2024). The annotated dimensions are as follows: • Mistake Identification: The tutor’s response should identify the student’s mistake or confusion. • Mistake Location: Tutor’s response should clearly

Read the complete text