Toward Automated Qualitative Analysis: Leveraging Large Language Models for Tutoring Dialogue Evaluation

Megan Gu, Chloe Qianhui Zhao, Claire Liu, Nikhil Patel, Jahnvi Shah, Jionghao Lin, Kenneth R. Koedinger · 2025

Read Toward Automated Qualitative Analysis: Leveraging Large Language Models for Tutoring Dialogue Evaluation by Megan Gu, Chloe Qianhui Zhao, Claire Liu, Nikhil Patel, Jahnvi Shah, Jionghao Lin, Kenneth R. Koedinger (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-13882.txt.

Introduction

Judging a tutor's craft usually means a human reading chat logs line by line, and Megan Gu and co-authors test whether a language model can do the grading, in a 2025 paper. Their system reads dialogues from a public teacher-student chatroom corpus and scores five strategies: praising well, reacting to errors, probing what students know, handling inequity, and answering negative self-talk.

GPT-3.5, given a few examples, sorted each strategy's use as desired or undesired. The model was good at ruling things out, with true negative rates around two thirds to three quarters, but poor at spotting the good moves, catching only a third to two fifths. Handling inequity scored best on both measures. The authors call the approach promising and point to stronger models and richer feedback as the next step.

Catalog record

CREATORMegan Gu, Chloe Qianhui Zhao, Claire Liu, Nikhil Patel, Jahnvi Shah, Jionghao Lin, Kenneth R. Koedinger
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE8 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-13882

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Toward Automated Qualitative Analysis: Leveraging Large Language Models for Tutoring Dialogue Evaluation
Megan Gu, Chloe Qianhui Zhao, Claire Liu, Nikhil Patel, Jahnvi Shah, Jionghao Lin, Kenneth R. Koedinger

Abstract
Our study introduces an automated system leveraging large language models (LLMs) to assess the effectiveness of five key tutoring strategies: 1. giving effective praise , 2. reacting to errors , 3. determining what students know , 4. helping students manage inequity , and 5. responding to negative self-talk . Using a public dataset from the Teacher-Student Chatroom Corpus, our system classifies each tutoring strategy as either being employed as desired or undesired. Our study utilizes GPT-3.5 with few-shot prompting to assess the use of these strategies and analyze tutoring dialogues. The results show that for the five tutoring strategies, True Negative Rates (TNR) range from 0.655 to 0.738, and Recall ranges from 0.327 to 0.432, indicating that the model is effective at excluding incorrect classifications but struggles to consistently identify the correct strategy. The strategy helping students manage inequity showed the highest performance with a TNR of 0.738 and Recall of 0.432. The study highlights the potential of LLMs in tutoring strategy analysis and outlines directions for future improvements, including incorporating more advanced models for more nuanced feedback.



[
[email protected]
]

[
[email protected]
]

[
[email protected]
]

[
[email protected]
]

[
[email protected]
]

[
[email protected]
]

*1

[
[email protected]
]

keywords

Qualitative Analysis ,Large Language Models ,Dialogue Analysis ,Feedback

†
†
copyrightyear:
2025
†
†
copyright:
Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0
International (CC BY 4.0).
†
†
venue:
LAK’25 Workshop: LLMs for Qualitative Analysis in Education
March 04, 2025, Dublin, Ireland
†
†
address:
Carnegie Mellon University,
5000 Forbes Ave, Pittsburgh, PA, 15213, United States
†
†
address:
The University of Hong Kong,
Pokfulam Rd, Hong Kong, China
†
†
address:
Monash University,
Wellington Rd, Clayton VIC 3800, Australia
†
†
corresp:
Corresponding author.

1
Background

Tutoring is widely recognized as one of the most effective forms of personalized learning support
Nickow et al. 2020
;
Han et al. 2024
. Within tutoring sessions, strategies such as praising student effort and providing feedback play a critical role in enhancing student learning outcomes
Royer et al. 2019
;
Lin et al. 2022
. When effectively employed, these strategies can support students’ cognitive development, meet their emotional needs, and foster a positive learning environment. For example, a well-placed praise such as “
You are making great progress on this problem
” (rather than generic praise like “
Good job
”) can emphasize the importance of the learning process, building student resilience and motivation
Royer et al. 2019
. Understanding how these tutoring strategies are employed during sessions is crucial, as it highlights whether they align with desired practices and are delivered in a manner that promotes student growth
Han et al. 2024
. However, the ability to automate this analysis has been constrained by the limitations of earlier natural language processing (NLP) tools, leaving room for significant improvements. Recent advancements in large language models (LLMs) offer a promising opportunity to develop automated systems for analyzing tutoring dialogues. These models (e.g., ChatGPT and Llama), with their ability to process and understand complex language patterns, provide a promising avenue for evaluating tutoring strategies in a nuanced and context-aware manner. To analyze the dialogue transcripts, our study leverages LLMs to develop an automated system (Figure
1
), accessible via
https://tutor-dialogue.vercel.app/dashboard/transcripts
.

Figure 1:
The Tutor Dialogue Classification system interface. On the left is (1) the navigation panel, which includes options for (a) accessing dialogue records for each transcription file, (b) viewing a comprehensive table of classification results for all available dialogue data, (c) evaluating classification patterns, and (d) explaining response categories. On the right is (2) the dialogue table, which features: (a) classification results displayed within the conversation between a tutor and a student, (b) filtering options for classification analysis, and (c) export functionalities for data analysis.

The system is designed to detect the use of tutoring strategies and assess whether they are employed in their desired form. It allows users to upload a spreadsheet containing dialogue transcripts, with each line of dialogue and its corresponding speaker specified. As shown in Figure 1, for each strategy detected, the system determines whether it was used effectively (good) or ineffectively (bad), and this information is presented in a color-coded format for easy interpretation: blue indicates effective use (
good example
), while red indicates ineffective use (
bad example
).

2
Method

2.1
Data

Our study used the dataset provided from the Teacher-Student Chatroom Corpus
Caines et al. 2022
. The dataset contains one-on-one English lessons in an online chatroom. It was released in 2022, and contains a total of 262 transcriptions. Then, we hired 4 annotators to annotate a total of 9 transcriptions for usage of 5 different tutoring strategies. In our annotation scheme, we assigned the following labels to each instance: ¡-1¿ when the tutoring strategy was not applicable, ¡0¿ when the tutoring strategy was used undesirably, and ¡1¿ when the tutoring strategy was used by the tutor in a desired manner.

2.2
Prompt Engineering

We used few-shot chain-of-thought prompting for each of the five tutoring strategies: (1)
Giving Effective Praise
, (2)
Reacting to Errors
, (3)
Determining what students know
, (4)
Helping Students Manage Inequity
, and (5)
Responding to Negative Self-Talk
. These tutoring strategies generally encourage students to persevere and increase their engagement, which are drawn from the PLUS Tutors Platform,
https://www.tutors.plus/en/solution/training
.

3
Results

Our study used the GPT-3.5 model to detect and classify tutoring strategies through few-shot prompting. Table
1
presents the accuracy of GPT-3.5 in identifying and classifying five tutoring strategies, measured by True Negative Rate (TNR) and Recall. GPT-3.5 achieves moderate TNR (0.655-0.738) but lower Recall (0.327-0.432). This suggests that the model performs somewhat effectively at excluding incorrect classification, but still struggles with identifying the correct one from the remaining two options. “
Helping Students Manage Inequity
” strategy achieves the highest performance with TNR of 0.738 and Recall of 0.432, though overall performance remains limited.

Table 1:
The performance of prompting GPT-3.5 model on identifying desired or undesired tutoring strategies

Tutoring Strategies

True Negative Rate

Recall

Giving Effective Praise

0.655

0.327

Reacting to Errors

0.683

0.376

Determining What Students Know

0.694

0.413

Helping Students Manage Inequity

0.738

0.432

Responding to Negative Self-Talk

0.665

0.331

Further enhancements to our transcription analysis system will focus on incorporating more advanced LLMs, providing detailed statistics and feedback based on the classification results, reporting the frequency with which each strategy was used effectively or ineffectively and generating overall feedback from the model. This feedback will evaluate the tutor’s effectiveness in employing each strategy and offer suggestions for improvement.

Acknowledgements.

This research was funded by the Richard King Mellon Foundation (Grant #10851) and the Learning Engineering Virtual Institute (
https://learning-engineering-virtual-institute.org/
). The opinions, findings, and conclusions expressed in this paper are those of the authors alone.

Declaration on Generative AI

Read the complete text