Agent-ToM: Learning to Monitor Autonomous LLM Agents via Theory-of-Mind Reasoning

Nesreen K. Ahmed, Nima Nafisi · 2026

Read Agent-ToM: Learning to Monitor Autonomous LLM Agents via Theory-of-Mind Reasoning by Nesreen K. Ahmed, Nima Nafisi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24216.txt.

Introduction

Ahmed and Nafisi propose Agent-ToM, a learning-to-monitor frame for catching covert harmful conduct in self-run agents. Hidden goals can hide behind benign conduct over long attack spans, and prior watchers treat each trace alone without learning from past watching. Agent-ToM infers beliefs, intent guesses, likely actions, and drifts from norms, then runs a Reason-Verify-Refine pipeline. Training distills critique signals into a lasting meaning guardrail memory. On hostile benchmarks it beats rival watchers and teams with a two-call pipeline.

Catalog record

CREATORNesreen K. Ahmed, Nima Nafisi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70843
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.24216

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Agent-ToM: Learning to Monitor Autonomous LLM Agents via Theory-of-Mind Reasoning Nesreen K. Ahmed, Nima Nafisi Abstract Monitoring autonomous large language model (LLM) agents for covert malicious behavior (e.g., covertly pursuing a hidden malicious objective) is challenging due to delayed, context-dependent, and long-horizon attack patterns. In adversarial settings such as sabotage, agents may pursue hidden objectives while maintaining superficially benign behavior, making detection difficult even with full trajectory access. Prior monitoring approaches primarily improve monitor scaffolding or ensemble aggregation, but treat each trajectory independently and do not improve from prior monitoring experience. Moreover, standard reasoning methods explain observed behavior but do not explicitly reason about agent beliefs, intentions, and goal alignment required to distinguish benign task execution from covert deviation. We propose Agent-ToM , a learning-to-monitor framework grounded in Theory-of-Mind (ToM) reasoning for security analysis of autonomous agents. Agent-ToM performs structured full-trajectory analysis by inferring beliefs, step-level intent hypotheses with calibrated confidence, expected actions, and deviations from task-consistent behavioral baselines. At inference time, it employs a Reason–Verify–Refine pipeline to construct and validate monitoring decisions. At training time, Agent-ToM learns from prior monitoring episodes by distilling critique signals into a persistent semantic guardrail memory that accumulates monitoring strategies, enabling reusable belief- and intent-conditioned constraints to be applied across episodes. We evaluate Agent-ToM on adversarial agent monitoring benchmarks (SHADE-Arena and CUA-SHADE-Arena). Agent-ToM achieves strong precision–recall balance and outperforms state-of-the-art monitoring baselines, including ensemble methods, while using a single coherent reasoning pipeline. These results demonstrate that learning at the monitoring layer , combined with structured ToM reasoning and verification, provides an effective and deployable foundation for securing autonomous LLM agents. 1 Introduction Autonomous large language model (LLM) agents are increasingly deployed to perform multi-step tasks involving tool use, code execution, and interaction with external systems ( 18 ; 8 ) . While such agents enable powerful automation, they also introduce new security risks: they may exhibit covert misbehavior, violating safety policies or deviating from intended user goals. Moreover, malicious behavior may be delayed, context-dependent, and distributed across long-horizon interaction traces, making it difficult to detect using short-horizon or action-local monitors. Effective monitoring therefore requires reasoning over full trajectories, inferring intent beyond surface actions, and distinguishing legitimate task execution from covert deviation. Recent work on agent monitoring has primarily focused on improving detection through scaffolding strategies such as chunking, hierarchical decomposition, sequential or sliding-window analysis, and ensemble aggregation ( 5 ; 4 ; 10 ; 6 ) . While these approaches improve coverage over long trajectories, they treat each execution trace independently and rely on fixed detection heuristics or aggregation over model outputs. As a result, monitoring systems do not improve from prior failures and lack mechanisms for accumulating reusable knowledge about adversarial behavior. Moreover, they do not explicitly model the beliefs, intentions, and goal alignment that underlie agent behavior, limiting their ability to distinguish benign deviations from covert malicious intent. In practice, this leads to brittle performance under evolving attack strategies and shifts the burden toward increasingly complex monitoring scaffolding, ensemble strategies, and threshold tuning. However, in real-world deployment settings, monitoring agents must learn at the monitoring layer . Rather than repeatedly evaluating each trace in isolation, monitors should adapt over time by incorporating lessons from past incidents ( 11 ; 3 ) . Moreover, monitoring requires reasoning abstractions that differ fundamentally from task-solving reasoning. While generic chain-of-thought explanations can describe what an agent did, they do not capture why it acted that way, whether its beliefs were consistent with the stated task, or whether observed deviations reflect benign exploration or malicious intent ( 12 ) . We therefore view monitoring not as a stateless classification problem over trajectories, but as a learning-based inference process that evolves through accumulating and refining reusable behavioral constraints from agent trajectories. We propose Agent-ToM , a learning-to-monitor framework grounded in Theory-of-Mind (ToM) reasoning for security analysis of autonomous LLM agents. Agent-ToM performs structured full-trajectory analysis by inferring agent behavior in terms of beliefs, inferred intent, expected actions, and deviations from task-consistent behavioral baselines. To make this reasoning robust and adaptive, Agent-ToM integrates a Reason–Verify–Refine inference paradigm, in which hypotheses about agent behavior are first constructed and an initial classification is produced through structured ToM reasoning, then subsequently verified and optionally refined based on evidential sufficiency and consistency. In addition, Agent-ToM leverages persistent semantic guardrail memory that accumulates monitoring strategies by distilling reusable belief- and intent-conditioned constraints from prior monitoring episodes, enabling cross-episode learning without retraining the underlying models or modifying the monitor. This design allows monitoring decisions to be explicitly constructed, critically evaluated, reused, and improved iteratively. We evaluate Agent-ToM on adversarial monitoring benchmarks derived from SHADE-Arena ( 5 ) , which provides agents with a benign main task coupled with a hidden malicious task and covers 17 task pairs (i.e., agent applications). In addition, we evaluate out-of-distribution performance on the recent CUA-SHADE-Arena benchmark ( 4 ) , which extends TheAgentCompany ( 17 ) to general computer-use agents and provides 15 main–hidden task pairs. We focus on challenging traces that span various adversarial strategies, including deception, obfuscation, and monitor-targeted attacks. Across multiple configurations, Agent-ToM achieves strong precision–recall balance and competitive AUC, matching or exceeding strong ensemble-based monitoring baselines with complex scaffolding involving many LLM calls, while using a single coherent reasoning pipeline with only two LLM calls during inference. These results demonstrate that belief- and intent-based monitoring, when combined with verification and experience-driven monitor-strategy learning, provides an effective, efficient, and deployable foundation for securing autonomous LLM agents. Contributions. We summarize the contributions of this paper as follows: 1. Learning-based agent monitoring via Theory-of-Mind reasoning. We introduce Agent-ToM , a unified learning-based cognitive monitoring framework grounded in Theory-of-Mind reasoning that performs structured full-trajectory analysis of autonomous agent behavior for security analysis. 2. Structured behavioral analysis via belief and intent inference. Agent-ToM infers beliefs and step-level intent hypotheses with confidence, along with expected actions and task-specific behavioral baselines, enabling evidence-based discrimination between benign and malicious behavior beyond surface trajectory patterns. 3. Reason–Verify–Refine inference paradigm for monitoring. Agent-ToM leverages a structured inference paradigm in which an initial hypothesis and classification are produced via ToM reasoning, and subsequently verified and optionally refined based on evidential sufficiency and consistency, improving robustness and calibration of monitoring decisions. 4. Critique-driven learning of monitoring strategies. At training time, Agent-ToM learns monitoring strategies by distilling critique signals from episodic traces into a persistent semantic guardrail memory, capturing reusable behavioral constraints that generalize across episodes. 2 Proposed Framework We present Agent-ToM, a learning-based monitoring framework for autonomous LLM agents. Instead of treating monitoring as a stateless classification problem over agent trajectories, Agent-ToM formulates monitoring as a learning-based inference problem that evolves by accumulating and refining reusable monitoring strategies from prior experience. Agent-ToM decomposes monitoring into structured inference over latent agent states and critique-driven learning of reusable behavioral constraints. At inference time, the framework reasons over full execution trajectories to infer beliefs, intent, expected actions, and deviations from task-consistent behavior. At training time, it converts critique signals from prior episodes into a semantic guardrail memory that guides future monitoring decisions. 2.1 Problem Definition & Threat Model We consider the problem of monitoring autonomous or semi-autonomous agents operating over extended episodes. Each agent is assigned a main task , a

Read the complete text