Educators’ Perceptions of Large Language Models as Tutors: Comparing Human and AI Tutors in a Blind Text-only Setting

Sankalan Pal Chowdhury, Terry Jingchen Zhang, Donya Rooein, Dirk Hovy, Tanja Käser, Mrinmaya Sachan · 2025

Read Educators’ Perceptions of Large Language Models as Tutors: Comparing Human and AI Tutors in a Blind Text-only Setting by Sankalan Pal Chowdhury, Terry Jingchen Zhang, Donya Rooein, Dirk Hovy, Tanja Käser, Mrinmaya Sachan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-08702.txt.

Introduction

Sankalan Pal Chowdhury and co-authors ask teachers to judge AI tutors against human ones, blind, in a study from 2025. Language models now staff homework helpers everywhere. Quality claims usually come from students or from scores. This study turns to the professionals instead.

Grade-school math word problems were taught in text-only chats. Some chats hid a human tutor, others a model. Educators rated both without knowing which was which, on engagement, empathy, scaffolding, and conciseness. The model won all four in their eyes. Empathy showed the widest gap: four of five raters preferred the machine's manner. Text-only tutoring may already favor AI's patience and polish. Classrooms will still decide what that is worth.

Catalog record

CREATORSankalan Pal Chowdhury, Terry Jingchen Zhang, Donya Rooein, Dirk Hovy, Tanja Käser, Mrinmaya Sachan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-08702

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Educators’ Perceptions of Large Language Models as Tutors: Comparing Human and AI Tutors in a Blind Text-only Setting Sankalan Pal Chowdhury, Terry Jingchen Zhang, Donya Rooein, Dirk Hovy, Tanja Käser, Mrinmaya Sachan Abstract The rapid development of Large Language Models (LLMs) opens up the possibility of using them as personal tutors. This has led to the development of several intelligent tutoring systems and learning assistants that use LLMs as back-ends with various degrees of engineering. In this study, we seek to compare human tutors with LLM tutors in terms of engagement, empathy, scaffolding, and conciseness. We ask human tutors to annotate and compare the performance of an LLM tutor with that of a human tutor in teaching grade-school math word problems on these qualities. We find that annotators with teaching experience perceive LLMs as showing higher performance than human tutors in all 4 metrics. The biggest advantage is in empathy, where 80% of our annotators prefer the LLM tutor more often than the human tutors. Our study paints a positive picture of LLMs as tutors and indicates that these models can be used to reduce the load on human teachers in the future. 1 Introduction Recent improvements in Large Language Models (LLMs) have opened up the possibility of using them in multiple new domains, including as personal tutors. This possibility has led to the development of several Intelligent Tutoring Systems (ITSs) and learning assistants Schmucker et al. 2023 ; Liffiton et al. 2023 ; Lieb and Goel 2024 ; Vanzo et al. 2024 that use LLMs as backends with various degrees of engineering. Surveys by Intelligent.com Int 2023 and DEC Singapore DEC 2024 indicate that a large number of students are already using LLMs like ChatGPT in educational roles such as tutoring. Despite their popularity, a clear understanding of the pedagogical effectiveness of educational chatbots, especially compared to humans, is lacking. The common way of using LLMs as tutor is to interact with them via a chat interface, where the LLM roleplays a tutor. It is known that the full benefit of a human tutor goes well beyond verbal or textual communication Bambaeeroo and Shokrpour 2017 , giving human tutors an advantage over LLM-based tutors. However, it remains unclear how LLM-based tutors compare with their human counterparts, in this chat setting. A good tutor keeps students engaged , empathises with their struggles, scaffolds them to correct answers, all while keeping the conversation to the point and concise . Is an LLM-based tutor capable of doing the same? In this study, we compare human tutors with LLM-based tutors, through the dialogs generated via chat interfaces. Our main research question is: How do LLM-based tutors compare to human tutors in terms of engagement, empathy, scaffolding, and conciseness? Although there have been some recent attempts to compare learning gains from LLM-based tutors and human tutors (see Sec 2 ), these studies focus on the observable outcomes of learning gains. Our study seeks to complement these studies by instead focusing only on the latent factors (we will provide a more detailed definition and justification in Sec. 3.2 ), and run comparisons on these directly. We believe that knowing how LLMs stand on these would allow researchers to better focus on what to improve in these models. Our contributions are: 1. We create a setup to ask human annotators to compare tutoring dialog snippets in a blind pairwise preference selection setting. 2. We use this setup to have teachers compare a human tutor with an LLM tutor on a dataset of MWPs to identify how they compare the 4 latent factors involved in student learning. 3. We publicly release the annotation data consisting of 210 annotated dialog pairs to help future research better align LLM outputs to human judgments. Our experiments find that annotators with teaching experience perceive the LLM tutor to be more engaging and empathetic while also being concise and better at scaffolding the student. This also aligns with LLMs self-judgments, though fine-grained tendencies are quite different. 2 Related Work 2.1 Designing and Evaluating LLM Based Tutors With the recent progress in LLMs, there have been several efforts to develop and evaluate LLM-based tutors. A large number of these have focused on computer science and programming education Yang et al. 2024 ; Qi et al. 2024 ; Liffiton et al. 2023 ; Kazemitabaar et al. 2024 ; Jacobs and Jaschke 2024 ; Liu et al. 2024 ; Lyu et al. 2024 ; Li et al. 2024 ; Choudhuri et al. 2023 ; Pankiewicz and Baker 2024 , but there have also been developments in domains like mathematics Chowdhury et al. 2024 ; Butgereit et al. 2023 ; Pardos and Bhandari 2024 , language learning Polakova and Klimova 2024 ; Park et al. 2024 ; Vanzo et al. 2024 , health sciences Kavadella et al. 2024 ; Chheang et al. 2024 ; Wang et al. 2024 and other domains Thway et al. 2024 ; Chen and Chang 2024 . However, most of these works focus on the engineering behind developing the tutor, and if any evaluation is done, it is either in terms of learning gains, or in the terms of student self-reports of efficacy and motivation. Moreover, the comparison in these studies is always between having an LLM-based tutor and not having anything, and not with human tutors. Finally, we also lack an understanding of the factors contributing to a good quality tutor. 2.2 Comparing AI Tutors with Human Tutors Tutoring was established as one of the best ways to improve learning outcomes by Bloom in 1984 Bloom 1984 , and matching the learning gains of a human tutor has been one of the main targets of computer-based tutors ever since Sleeman and Brown 1982 . Several studies have compared the learning gains from different types of computer-based tutors with humans Kulik and Kulik 1991 ; Anderson et al. 1995 ; VanLehn 2011 and with the development of LLM-based tutors, the same has also been extended to LLM based tutors Schmucker et al. 2023 ; Zhang et al. 2024 . However, these works focus only on the final learning gain, not on the latent qualities that could cause it. Since several computer-based tutors communicate in natural language, another line of work follows from Alan Turing’s Imitation Game Turing 1950 , which was later adapted into the ‘Bystander Turing Test’ Person and Graesser 2002 . While our work is similar to this in terms of the text-only setup and blind selection, we differ in that instead of asking the annotator to determine which party is human, we ask them to determine which one is better on a set of metrics. 3 Method 3.1 Datasets To compare human and LLM tutors, we need parallel data sets of student-tutor text interactions, both for human and LLM tutors. Among the limited one-on-one tutoring datasets available, we cannot use data sets such as TSCC Caines et al. 2020 or CIMA Stasaski et al. 2020 because they use human students, and a fair comparison would require us to repeat the LLM side of the experiment with the same humans. To avoid doing this, we draw our conversations from MathDial Macina et al. 2023 for the human side because the

Read the complete text