Do Large Language Models Mentalize When They Teach?

Sevan K. Harootonian, Mark K. Ho, Thomas L. Griffiths, Yael Niv, Ilia Sucholutsky · 2026

Read Do Large Language Models Mentalize When They Teach? by Sevan K. Harootonian, Mark K. Ho, Thomas L. Griffiths, Yael Niv, Ilia Sucholutsky (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-01594.txt.

Introduction

Harootonian and colleagues test whether large language models decide what to teach by reasoning about a learner's knowledge or by simple rules. In a task used to study human teaching, a teacher model sees a learner's path through a reward-annotated directed graph. It must reveal one edge so the learner picks a better path if they replan. They fit models' trial-by-trial choices with human cognitive models, including a Bayes-Optimal teacher inferring missing transitions via inverse planning. Comparison shows Bayes-Optimal teaching best explains most models' choices.

Catalog record

CREATORSevan K. Harootonian, Mark K. Ho, Thomas L. Griffiths, Yael Niv, Ilia Sucholutsky
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40711
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.01594

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Do Large Language Models Mentalize When They Teach? Sevan K. Harootonian, Mark K. Ho, Thomas L. Griffiths, Yael Niv, Ilia Sucholutsky Abstract How do LLMs decide what to teach next: by reasoning about a learner’s knowledge, or by using simpler rules of thumb? We test this in a controlled task previously used to study human teaching strategies. On each trial, a teacher LLM sees a hypothetical learner’s trajectory through a reward-annotated directed graph and must reveal a single edge so the learner would choose a better path if they replanned. We run a range of LLMs as simulated teachers and fit their trial-by-trial choices with the same cognitive models used for humans: a Bayes-Optimal teacher that infers which transitions the learner is missing (inverse planning), weaker Bayesian variants, heuristic baselines (e.g., reward based), and non-mentalizing utility models. In a baseline experiment matched to the stimuli presented to human subjects, most LLMs perform well, show little change in strategy over trials, and their graph-by-graph performance is similar to that of humans. Model comparison (BIC) shows that Bayes-Optimal teaching best explains most models’ choices. When given a scaffolding intervention, models follow auxiliary inference- or reward-focused prompts, but these scaffolds do not reliably improve later teaching on heuristic-incongruent test graphs and can sometimes reduce performance. Overall, cognitive model fits provide insight into LLM tutoring policies and show that prompt compliance does not guarantee better teaching decisions. 1 Introduction Large language models (LLMs) are rapidly being integrated into education as tutors, feedback generators, and teaching assistants. A recent systematic review of 88 empirical studies since ChatGPT’s release reports that intelligent tutoring systems are the most common educational application of LLMs, with overall evidence for improved performance and engagement alongside concerns about over-reliance and reliability ( 13 ) . Complementing these broad surveys, controlled evaluations have begun to test instructional efficacy directly—for example, comparing LLM-generated hints to human-authored help in mathematics ( 9 ) , and evaluating AI tutoring systems in randomized trials ( 7 ) . Parallel work has focused on eliciting better tutoring behavior via design and prompting. Recent approaches propose pedagogical steering methods that prompt an LLM to follow predefined multi-turn tutoring plans ( 11 , e.g., Productive Failure;) , and tutor-facing copilots that use LLMs to support human tutors in real time and encourage more effective pedagogical moves ( 14 ) . While these lines of research show that LLM-mediated instruction can improve learning outcomes, they primarily benchmark what the model produces (and its downstream effects) rather than how it chooses instructional actions. Consequently, it remains unclear what teaching strategy an LLM uses when deciding what to teach next. Work on human teaching provides a useful framework for posing this question. In controlled teaching tasks, people often rely on either (i) a model-based strategy that mentalizes a learner’s knowledge and chooses information to maximally improve it, or (ii) model-free heuristics that ignore a learner model and rely on environmental cues ( 5 ) . Related contrasts appear in work on adaptive versus routine expertise, and learners penalize instruction that appears untailored ( 6 ; 2 ) . If LLMs default to heuristics, deployment as tutors may require stronger external scaffolds and monitoring; if they behave in a way that is more model-based, we can better leverage and test that capacity. Here, we probe LLM teaching strategies by adapting the Graph Teaching task of 5 . On each trial, an LLM teacher observes a hypothetical learner’s trajectory in a reward-annotated graph and reveals a single edge that could change the learner’s future path. We evaluate multiple LLMs as simulated teachers and fit their trial-by-trial decisions with the same family of cognitive models used in the human study—Bayesian teacher models, heuristic baselines, and non-mentalizing utility models. Finally, we test whether lightweight scaffolding prompts shift LLM teaching behavior in ways analogous to interventions known to affect strategy use in humans. 2 Methods 2.1 Task: Graph Teaching We used the Graph Teaching task introduced in 5 . On each trial, the environment is a directed deterministic acyclic graph with a single start node at the top and multiple terminal nodes at the bottom. Each node carries a non-negative reward, and a learner traverses the graph from top to bottom, moving only straight down or diagonally down, earning the sum of rewards along its path. The learner is assumed to know only an unknown subset of the true edges and to choose a path that is optimal given its current (partial) transition knowledge. The teacher, by contrast, observes the full transition structure and node rewards, as well as a single trajectory taken by the learner on that trial (Figure 1 A). The teacher knows that the learner is doing the best they can, and must select a single edge to reveal given only the learner’s trajectory, so that if the learner were to replan optimally given this new information, its expected future reward would be as high as possible (Figure 1 B). As in experiments on human participants, unbeknownst to the teacher, the learner is an RL agent that solves for the optimal policy on each trial, given the reward function and subset of the transition matrix. A Full derivation of the learner and task structure can be found in 5 . In addition, a demo of the human task with full instructions can be found at https://sharootonian.github.io/CognitiveStrategiesInTeaching/GraphTeachingTask_demo/ Figure 1: Graph Teaching Task. The numeric value in each node indicates the reward obtained when the learner visits that node. The learner starts at the top node and moves down or diagonally down until reaching a terminal node. A) Learner computes and executes the best available trajectory. B) Teacher observes the learner’s trajectory and selects an edge to teach. C) Learner incorporates the taught edge and computes and executes a new, better trajectory. Note: participants and models only see part B and make assumptions for parts A and C. 2.2 Cognitive model fitting and comparisons To characterize the teaching strategies expressed by each LLM, we fit the same family of cognitive models used in the human study: Bayesian Teacher models (which infer a learner knowledge state and maximize expected teaching utility), heuristic models (which rely on task features rather than a learner model), and non-mentalizing utility models (which treat the task as a puzzle-solving problem). Full derivations are provided in 5 . Across models, each candidate edge e is assigned a model-specific utility U_{\text{model}}(e) and converted to a choice probability via a softmax rule, P(e)=\frac{\exp\!\big(\beta\,U_{\text{model}}(e)\big)}{\sum_{e^{\prime}}\exp\!\big(\beta\,U_{\text{model}}(e^{\prime})\big)}, (1) where \beta is an inverse-temperature parameter fit to each simulated subject. Bayesian Teacher models. In these models, we model the learner as an agent navigating a directed deterministic graph with transition knowledge T_{L}\subseteq T and reward function R . Given an observed learner trajectory \zeta , the Bayes Optimal Teacher (BOT) performs inverse planning by inverting a forward planning model ( 1 ) to infer a posterior over T_{L} given the observed trajectory P_{\mathrm{BOT}}(T_{L}\mid\zeta) , and evaluates the expected teaching utility of revealing an edge x as U_{\mathrm{BOT}}(x;\zeta)=\sum_{T_{L}}P_{\mathrm{BOT}}(T_{L}\mid\zeta)\Big(V_{T_{L}\cup\{x\}}(s_{0})-V_{T_{L}}(s_{0})\Big), (2) where V_{T_{L}}(s_{0}) is the learner’s optimal value at the start state under knowledge T_{L} . We also fit two inference variants that weaken the mentalizing step: the No Inverse Planning Bayesian Teacher

Read the complete text