How well do Large Language Models Recognize Instructional Moves? Establishing Baselines for Foundation Models in Educational Discourse

Kirk Vanacore, René F. Kizilcec · 2025

Read How well do Large Language Models Recognize Instructional Moves? Establishing Baselines for Foundation Models in Educational Discourse by Kirk Vanacore, René F. Kizilcec (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-19903.txt.

Introduction

Kirk Vanacore and Rene Kizilcec benchmark how well today's language models can read a classroom. Their task is classifying instructional moves, the recognizable teaching actions inside authentic lesson transcripts, a simple-sounding job that matters for education technology built on top of these models. Six models were tested out of the box, under zero-shot, one-shot, and few-shot prompting, against expert-coded annotations.

Zero-shot performance was moderate. Comprehensive few-shot examples lifted state-of-the-art models substantially, the best configuration reaching a Cohen's kappa of 0.58 against expert coding, a moderate agreement level. Gains were uneven across move types, and better recall often arrived with more false positives. The authors conclude that foundation models show meaningful but limited capacity for interpreting educational discourse, with prompt design surfacing capability rather than creating reliability.

Catalog record

CREATORKirk Vanacore, René F. Kizilcec
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-19903

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

How well do Large Language Models Recognize Instructional Moves? Establishing Baselines for Foundation Models in Educational Discourse Kirk Vanacore, René F. Kizilcec Abstract Large language models (LLMs) are increasingly adopted in educational technologies for a variety of tasks, from generating instructional materials and assisting with assessment design to tutoring. While prior work has investigated how models can be adapted or optimized for specific tasks, far less is known about how well LLMs perform at interpreting authentic educational scenarios without significant customization. As LLM-based systems become widely adopted by learners and educators in everyday academic contexts, understanding their out-of-the-box capabilities is increasingly important for setting expectations and benchmarking. We compared six LLMs to estimate their baseline performance on a simple but important task: classifying instructional moves in authentic classroom transcripts. We evaluated typical prompting methods: zero-shot, one-shot, and few-shot prompting. We found that while zero-shot performance was moderate, providing comprehensive examples (few-shot prompting) significantly improved performance for state-of-the-art models, with the strongest configuration reaching κ = 0.58 \kappa=0.58 against expert-coded annotations. At the same time, improvements were neither uniform nor complete: performance varied considerably by instructional move, and higher recall frequently came at the cost of increased false positives. Overall, these findings indicate that foundation models demonstrate meaningful yet limited capacity to interpret instructional discourse, with prompt design helping to surface capability but not eliminating fundamental reliability constraints. 1 Introduction Research on generative artificial intelligence (GenAI) in education has advanced our understanding of how large language models (LLMs) can be harnessed to improve or extend existing educational technologies. For instance, extensive research has examined whether LLMs can be tailored for specific instructional tasks [ 1 ] , such as generating feedback or producing learning materials [ 2 , 3 ] . Yet an equally pressing question concerns how well these general-purpose foundation models perform when used without task-specific tuning in authentic educational contexts. The widespread availability of LLMs is already reshaping how students and educators seek academic support: students report using AI for assignment completion [ 4 ] and obtaining help while learning [ 5 ] . One recent study found that a large majority of university students now use ChatGPT for academic purposes, including concept explanation, problem-solving, and clarifying assignment expectations [ 6 ] . Furthermore, there is growing evidence that teachers rely on AI to create educational content [ 7 , 8 ] , while many districts are not providing training on the use of AI in the classroom [ 9 ] . Collectively, these patterns suggest that openly available foundation models already function as de facto educational tools regardless of whether they have been explicitly adapted or optimized for such tasks. Researchers have developed benchmarks to evaluate the baseline performance of foundation models, including LLMs released by OpenAI, Anthropic, and Alphabet. These benchmarks elucidate how models perform out-of-the-box on education-related tasks, rather than how they perform when used by researchers with carefully engineered prompts, curated examples, downstream verification methods, or even fine-tuning. These idiosyncratic customizations make it difficult to separate genuine model capability from scaffolded performance. Researchers have examined how AI models perform on educational assessments, understand problems, and identify students’ misconceptions [ 10 , 11 , 12 , 13 ] . We add to the growing body of work on educational benchmarking of AI models by examining how current foundation models behave under minimal guidance when classifying instructional moves in real instructional transcripts, establishing an empirical baseline for future methodological and applied work. We find that leading foundation models (e.g., Claude 4.5 Opus) can achieve moderate reliability when provided with clear definitions and extensive examples (few-shot). However, the observed patterns of performance also suggest potential weaknesses in LLMs’ current ability to evaluate the complex blend of social and cognitive discourse that underlies effective instruction without such support. 1.1 Current Benchmarks AI Performance in Educational Contexts Recent efforts in educational AI benchmarking span a range of tasks, from pedagogical test performance to diagnosing specific algebraic misconceptions. For example, the Pedagogy Benchmark leaderboard 1 1 1 https://benchmarks.ai-for-education.org/ focuses on broader pedagogical alignment, particularly regarding cross-domain pedagogical knowledge assessed through tests on pedagogical theory and practice [ 13 ] . Many foundation models perform well, answering 80%–90% of questions correctly. Other benchmarks show that foundation model performance varies on more complex tasks. Otero et al. [ 10 ] provide a benchmark that focuses on diagnostic capability, testing whether models can identify 55 distinct algebraic misconceptions (such as variable cancellation errors) rather than simply generating a correct solution. They found that current AI models exhibit varying performance in correctly identifying misconceptions, depending on the type of math problem (e.g., lower performance on ratio problems and higher performance on number operations). Yang et al. [ 11 ] extended this work by introducing MMTutorBench, which evaluates multimodal tutoring capabilities. Their findings reveal a critical gap: while models may correctly recognize text, they struggle to integrate visual student work (like handwritten equations) into a coherent tutoring strategy, often failing to distinguish between a student’s insight error and a mere execution error. In the domain of open-ended dialogue, Macina et al. [ 12 ] highlight an "expertise-pedagogy trade-off." Their work on MathTutorBench shows that models optimized for high-level mathematical problem-solving often exhibit lower pedagogical quality, struggling to perform "Socratic" moves such as scaffolding a hint without revealing the answer. 1.2 AI Identification of Instructional Moves Annotation is a common benchmarking task for AI models, which may be particularly important for assessing whether these models can operate in educational contexts similarly to human experts. Recent research examines how LLMs can streamline qualitative analysis of educational data, including workflows specifically designed for student and teacher discourse. For human-in-the-loop workflows, Henkel and Hills [ 15 ] demonstrate that AI-assisted comparative judgment improves reliability when scoring open-ended student responses. Similarly, Barany et al. and Zambrano et al. find that LLMs function as "critical partners" in quantitative ethnography, helping researchers generate, refine, and consolidate codebooks for analyzing complex learning interactions [ 17 , 16 ] . When evaluating LLMs as an independent labeler of instructional content, performance is promising but highly context-dependent. Liu et al. [ 19 , 20 ] report that GPT-4 can match human reliability in coding virtual tutoring transcripts, though it struggles with underrepresented constructs. In the context of whole-class discussion, Long et al. [ 22 ] find that while LLMs achieve moderate agreement with experts, they exhibit systematic weaknesses in identifying nuanced instructional moves. Methodological challenges remain a core focus: Nahum et al. warn that errors in human "gold standard" data can distort model evaluation [ 23 ] , while He et al. [ 27 ] note that without high-quality gold labels, even experts struggle to improve model performance through iterative prompting. Finally, while few-shot prompting (providing examples within the prompt) generally improves performance in educational coding tasks [ 20 ] , it is not a panacea [ 21 ] . Studies on classroom dialogue indicate that adding examples can sometimes degrade performance if the examples are not perfectly aligned with the specific pedagogical context [ 22 ] , suggesting that rigorous, construct-specific evaluation is essential for deploying AI in educational research. 2 Current Study Despite rapid advances in AI-assisted educational analytics, less is known about

Read the complete text