What Do You Think I Think? Accounting for Human Beliefs Using Second-Order Theory of Mind

Patrick Callaghan ([email protected]), Reid Simmons, Henny Admoni · 2026

Read What Do You Think I Think? Accounting for Human Beliefs Using Second-Order Theory of Mind by Patrick Callaghan ([email protected]), Reid Simmons, Henny Admoni (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12745.txt.

Introduction

Callaghan, Simmons and Admoni give agents a second-order theory of mind for handling what people wrongly think agents know. Gaps between an agent's actual knowledge and a person's beliefs about that knowledge can hinder interaction. Using an interactive partially observable Markov decision process framework, the agent models how a person's erroneous beliefs evolve and the cognitive biases and heuristics behind them. It can then detect when such biases are at play and generate feedback that accounts for them, improving teacher informativeness in an in-person study.

Catalog record

CREATORPatrick Callaghan ([email protected]), Reid Simmons, Henny Admoni
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30985
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.12745

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

What Do You Think I Think? Accounting for Human Beliefs Using Second-Order Theory of Mind Patrick Callaghan ([email protected]), Reid Simmons, Henny Admoni Abstract Discrepancies between an agent’s actual knowledge and what a person thinks the agent knows can hinder interactions. If an agent could detect such discrepancies, it could provide feedback to account for them and improve current and future interactions. Using the I-POMDP as a framework for a second-order Theory of Mind (ToM-2), this work endows an agent with the ability to model the evolution of a person’s erroneous beliefs about an agent and the cognitive biases and heuristics (CBH) from which they arise. In doing so, the agent can detect when CBH might be at play during an interaction and adaptively generate feedback that accounts for them. An in-person user study shows how a ToM-2 learner can account for the effects of a teacher’s CBH to significantly improve the informativeness of teacher actions, and subjective results suggest people find the ToM-2 learner’s feedback more useful. Keywords: Cognitive Biases; Cognitive Heuristics; Theory of Mind; Human-Agent Interaction; Human Modeling Introduction Intelligent agents can support people in a variety of interactive scenarios in the home or the workplace, but for the best chance at having successful interactions, people should have an accurate model of what the agent is capable of or likely to do. However, people employ cognitive biases and heuristics (CBH) that may (incorrectly) influence their expectations for, and understandings of, the agent ( 25 ; 24 ; 12 ) . In this work, we aim to improve human-agent interactions by enabling an agent to model the influence CBH can have on a person’s beliefs about the agent so it can take actions to account for CBH’s impact on said beliefs. Whereas Theory of Mind (ToM) refers to one’s ability to infer another’s beliefs from their actions ( 19 ) , second-order Theory of Mind (ToM-2) is the ability to infer an agent’s beliefs about another agent’s beliefs—a skill people recruit to facilitate interactions with one another ( 3 ; 18 ; 8 ) . Prior work in human-AI interaction investigates the utility of endowing agents with a ToM or ToM-2 ( 1 ; 10 ; 15 ; 26 ; 27 ; 5 ) , and while valuable contributions, we wish to learn an approximation of the human’s model of the agent to generate agent actions that positively influence the human’s true model and their downstream behaviors. Additionally, we specifically aim to do so by modeling and accounting for ways CBH can influence what people think an agent thinks. Our key insight is that a ToM-2 enables the agent to explicitly model a person’s beliefs of the agent’s beliefs during an interaction such that the agent can adapt its behavior to account for erroneous beliefs and their possible causes. To investigate this idea, we leverage the nested modeling afforded by an Interactive POMDP (I-POMDP) to maintain explicit models of a person and the effects that CBH could have on their perceptions of the agent ( 13 ) . In doing so, the agent can approximate the impact that different CBH might have upon a person’s inferences of agent beliefs. As a result, the agent can identify discrepancies between those approximations and its actual knowledge, as well as the candidate causes of those discrepancies. In turn, the agent can adaptively generate feedback which aims to aid the person’s reasoning and inference processes. Figure 1: An AI learner with a zeroth-order Theory of Mind (blue) maintains state-based beliefs informed by a human teacher’s demonstrations but does not model human mental states. It provides feedback only with respect to state. ToM-2 augments the learner’s belief with a model of what its teacher thinks the learner knows (orange) to provide additional feedback with respect to approximated teacher beliefs. We contrive a teacher-learner domain to model how the Representativeness Heuristic and Confirmation Bias might affect a teacher’s perceptions of an AI learner and their pedagogical reasoning. Furthermore, we design agent feedback that adaptively optimizes the information it communicates based on its beliefs and its approximation of the teacher’s beliefs. With this formulation, we conduct an in-person user study and showcase: • A novel I-POMDP-based framework that enables an agent to approximate when and how CBH affects a human’s behavior and beliefs; • This model enables an agent to generate feedback that accounts for a person’s CBH upon being detected; and • Empirical evidence that this feedback results in improved interactions between humans and agents. Preliminaries Figure 2: Proof-of-Concept Domain : Multi-featured cards are sorted into bins according to a rule. While our work is intended to generalize beyond the teacher-learner context, work in pedagogical reasoning is especially relevant given our domain of study ( 6 ) . Shafto et al. formalize teaching as a recursive reasoning process and assume a teacher selects its data with helpful intent ( 23 ) . Our work compliments this formal representation using an alternative method for recursive reasoning from the learner’s perspective while aiming to investigate how models of human CBH might affect their ability to select helpful data. Additionally, Rafferty et al. showcase how POMDPs enable the planning of teaching actions when faced with different cognitive models ( 20 ) . While important work and similar to ours in multiple ways, our focus is on how a ToM-2 enables an AI learner to model and account for a teacher’s potential CBH when selecting learner actions. Domain We designed a turn-based teaching task to study how a ToM-2 can enable an agent to model, detect, and account for a person’s CBH during an interaction (Figure 2 ). In this task, a “rule” governs how multi-featured cards are sorted into two bins. There are four Classes of card features (Color, Fill, Number, Shape), each with three possible Feature Values (e.g., Blue, Green, Red). A rule consists of one Class from which two Feature Values are assigned to bins one and two respectively. Crucially, the third Feature Value can go in both bins, thereby introducing ambiguity which the teacher must also resolve. For example, one rule could be: (Class: Color. Bin 1 Value: Red. Bin 2 Value: Blue). Here, Green cards can be placed in either bin. Going forward, this rule will serve as a running example to provide intuition where helpful. In a single teaching round, the teacher sorts a card into a bin and the agent responds with a statement pertaining to its current belief over rule features (detailed in the Approach section). This process continues until the teacher determines that the agent knows the true rule with absolute certainty and attempts to end the session. If the learner knows the rule, the session ends; otherwise, the session recommences until the teacher’s next attempt to end it. Model An Interactive Partially Observable Markov Decision Process (I-POMDP) for an agent with a level- L Theory of Mind (ToM- L ) comprises the tuple: \langle IS_{L},A,T,O,R,\Omega\rangle with [A] ction and [O] bservation spaces, [T] ransition and [R] eward functions, and observation function \Omega ( 13 ) . Unlike a traditional POMDP, the I-POMDP models an interactive state space IS_{L} in which each is_{L}\in IS_{L} comprises a state s and a model

Read the complete text