Can you see how I learn? Human observers’ inferences about Reinforcement Learning agents’ learning processes Conference: Proc. of the Adaptive and Learning Agents Workshop (ALA 2025); May 19 – 20, 2025; Detroit, Michigan, USA, ala-workshop.github.io 897

Bernhard Hilpert, Muhan Hou, Kim Baraka, Joost Broekens · 2025

Read Can you see how I learn? Human observers’ inferences about Reinforcement Learning agents’ learning processes Conference: Proc. of the Adaptive and Learning Agents Workshop (ALA 2025); May 19 – 20, 2025; Detroit, Michigan, USA, ala-workshop.github.io 897 by Bernhard Hilpert, Muhan Hou, Kim Baraka, Joost Broekens (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-13583.txt.

Introduction

People watching a learning agent form theories about how it learns, Bernhard Hilpert and co-authors show in a 2025 paper. Reinforcement learning agents often behave in ways that are not intuitively interpretable. In collaborative teaching settings, that opacity can lead to suboptimal feedback, since a human teacher may reward or correct the wrong behavior at the wrong time.

The team built an observation-based paradigm to assess human inferences about agent learning directly. An exploratory interview study with nine participants identified four core themes: agent goals, knowledge, decision making, and learning mechanisms. A confirmatory study with thirty-four participants applied an expanded paradigm across two tasks, including navigation. Readers get a data-driven picture of how humans read machine learning behavior.

Catalog record

CREATORBernhard Hilpert, Muhan Hou, Kim Baraka, Joost Broekens
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-13583

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Can you see how I learn? Human observers’ inferences about Reinforcement Learning agents’ learning processes Conference: Proc. of the Adaptive and Learning Agents Workshop (ALA 2025); May 19 – 20, 2025; Detroit, Michigan, USA, ala-workshop.github.io 897 Bernhard Hilpert, Muhan Hou, Kim Baraka, Joost Broekens Abstract . Reinforcement Learning (RL) agents often exhibit learning behaviors that are not intuitively interpretable by human observers, which can result in suboptimal feedback in collaborative teaching settings. Yet, how humans perceive and interpret RL agent’s learning behavior is largely unknown. In a bottom-up approach with two experiments, this work provides a data-driven understanding of the factors of human observers’ understanding of the agent’s learning process. A novel, observation-based paradigm to directly assess human inferences about agent learning was developed. In an exploratory interview study ( N =9), we identify four core themes in human interpretations: Agent Goals, Knowledge, Decision Making, and Learning Mechanisms. A second confirmatory study ( N =34) applied an expanded version of the paradigm across two tasks (navigation/manipulation) and two RL algorithms (tabular/function approximation). Analyses of 816 responses confirmed the reliability of the paradigm and refined the thematic framework, revealing how these themes evolve over time and interrelate. Our findings provide a human-centered understanding of how people make sense of agent learning, offering actionable insights for designing interpretable RL systems and improving transparency in Human-Robot Interaction. Avalos, Aydeniz, Müller, Mohammedalamen (eds.) 1. Introduction Hybrid Intelligence (HI) refers to collaborative human–machine systems that combine strengths to accomplish tasks neither could perform alone ( Akata et al. 2020 ) . A key example is Human-in-the-loop Reinforcement Learning (RL), where human teachers guide learning agents using signals such as feedback or demonstrations ( Chetouani 2021 ; Knox and Stone 2009 ; Abel et al. 2017 ) . While humans can provide insight, expert knowledge, or common sense ( Celemin et al. 2022 ) , feedback is often suboptimal: delayed ( Arzate Cruz and Igarashi 2020 ) , inconsistent ( Thomaz and Breazeal 2007 ) , misaligned with the agent’s needs ( Carroll et al. 2019 ) or based on misinterpretation of the agent’s behavior ( MacGlashan et al. 2017 ) . Recent surveys emphasize explainability and human involvement across all stages of the RL lifecycle, from design to deployment ( Retzlaff et al. 2024 ) . However, relatively little attention has been paid to human side of teaching: how humans infer learning from watching agents, and how such inferences influence their teaching decisions. Prior work has only examined such inferences indirectly, via interaction patterns (see e.g. ( MacGlashan et al. 2017 ; Ho et al. 2019 ; Carroll et al. 2019 ) ) or simulated settings ( Grislain et al. 2023 ) . Moreover, existing paradigms confound observation with interaction: when humans actively teach, their expectations shape not only the feedback but also the agent’s behavior, making it hard to isolate what they actually understand ( Ho et al. 2019 ; Sarin et al. 2021 ) . This may obstruct an unbiased examination of the human’s perspective of the learning progress, especially at later learning stages. To the best of our knowledge, no study has so far examined human inferences about the RL agent learning process from behavior observation alone. Open questions include the correct setting, time-chunking of observations, how to even query inferences about an RL agent’s learning process from human observers, and which aspects of learning do humans attend to when making sense of agent behavior. The main Research Question for this study therefore is: What do human teachers infer about RL agents’ learning processes from observing the agents’ learning behavior? This work takes a human-centered, bottom-up approach by introducing a novel observational paradigm specifically designed to elicit human inferences about RL agent behavior. Across two experiments — spanning different tasks and RL algorithms— a set of four recurring inference themes are identified and validated: Agent Goals, Knowledge, Decision Making, and Learning Mechanisms. A detailed analysis further examines how these themes evolve over time and contribute to structuring observers’ understanding of agent learning. 2. Background and Related Work The mechanisms of Reinforcement Learning (RL) are inherently difficult for human observers to interpret from behavior alone, both due to algorithmic complexity ( Habibian et al. 2023 ) and the challenge of intuitively interpreting concepts like reward functions ( Hejna III and Sadigh 2023 ) . In interactive settings, a teacher’s perception of an agent’s behavior can recursively shape their interpretation of the learning process and thereby influence the course of the interaction ( MacGlashan et al. 2017 ) . For instance, Ho and colleagues demonstrate that teachers are strongly inclined to use rewards in the teaching process intuitively as a communicative signals rather than a reinforcement signal ( Ho et al. 2015 ; Ho [n.d.] ; Ho et al. 2019 ) , indicating a persistent misalignment between users’ assumptions and actual RL mechanisms. Although RL agents are designed to optimize cumulative rewards, the observed interaction pattern suggests that human teachers seem to interpret their actions as if the agent were responding to the communicative intent rather than reinforcement signals. Specifically, the researchers showed that the pattern persisted, even when it explicitly hindered agent learning progress ( Ho et al. 2019 ) . The authors explained this through assumed differences in how teachers interpret the agent’s state space, and that teachers seem to evaluate the agent by comparing its inferred policy to the teacher’s own desired policy ( Ho et al. 2019 ) . While such judgments may reflect a reasonable teaching strategy from the human perspective, the formation of positive reward cycles in the teaching process based on the resulting feedback ( Ho et al. 2019 ) highlights a disconnect between human interpretations of learning and the underlying mechanisms of RL. Further work indicates that these communicative assumptions about feedback generalize to Human-Human Interaction, where punishment and reward are also interpreted through inference ( Sarin et al. 2021 ) . More broadly, the Inferential Social Learning framework describes behavior interpretation as a process of "probabilistic inferences, guided by an intuitive understanding about how people plan and act" ( Gweon 2021 ) . This suggests that individuals acquire knowledge about their interaction partners by inferring insights from behavior observation. Understanding how this process applies in the context of Human Agent-Interaction settings, is therefore crucial in order to design effective Human-in-the-loop RL frameworks. However, research in Human-Agent Interaction has primarily studied teacher inferences indirectly, by analyzing interaction behaviors. The inferences (what teachers actually think about how an agent learns) have rarely been studied directly. The present work addresses this gap by directly examining how human observers infer and interpret the learning processes of RL agents from behavior observation alone. 3. Experiment 1: Exploratory Interview Study In this first exploratory study, an observational experimental paradigm is developed and its exploratory results presented. 3.1. Methodology 3.1.1. Participants: A total of n =9 ( n =3 female, n =6 male) were recruited. Five were bachelor students with RL exposure, four had no RL background. 3.1.2. Set-up & Procedure: In the absence of a validated paradigm to directly examine observer inferences about agent learning processes, an adapted version of the Grounded Theory framework was used to develop a reliable experimental

Read the complete text