Detection of adversarial intent in Human-AI teams using LLMs

Abed K. Musaffar, Ambuj Singh, Francesco Bullo · 2026

Read Detection of adversarial intent in Human-AI teams using LLMs by Abed K. Musaffar, Ambuj Singh, Francesco Bullo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20976.txt.

Introduction

Musaffar and co-authors test whether language models can spot malicious intent inside human and AI teams. Support agents help with search, coding, and decisions, but attackers can poison data or inject prompts to mislead them. Past work cast such models as targets or attackers, not defenders. Using real team conversations over twenty five rounds, they frame detection as a task on interaction traces. The models flag malicious behavior in real time, without task specific detail. Simple heuristics miss the same behavior.

Catalog record

CREATORAbed K. Musaffar, Ambuj Singh, Francesco Bullo
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25704
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.20976

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Detection of adversarial intent in Human-AI teams using LLMs Abed K. Musaffar, Ambuj Singh, Francesco Bullo Abstract Large language models (LLMs) are increasingly deployed in human-AI teams as support agents for complex tasks such as information retrieval, programming, and decision-making assistance. While these agents’ autonomy and contextual knowledge enables them to be useful, it also exposes them to a broad range of attacks, including data poisoning, prompt injection, and even prompt engineering. Through these attack vectors, malicious actors can manipulate an LLM agent to provide harmful information, potentially manipulating human agents to make harmful decisions. While prior work has focused on LLMs as attack targets or adversarial actors, this paper studies their potential role as defensive supervisors within mixed human-AI teams. Using a dataset consisting of multi-party conversations and decisions for a real human-AI team over a 25 round horizon, we formulate the problem of malicious behavior detection from interaction traces. We find that LLMs are capable of identifying malicious behavior in real-time, and without task-specific information, indicating the potential for task-agnostic defense. Moreover, we find that the malicious behavior of interest is not easily identified using simple heuristics, further suggesting the introduction of LLM defenders could render human teams more robust to certain classes of attack. 1 Introduction Over the past few years, the agentic framework has rapidly transitioned AI from static tools to active collaborators in hybrid human-AI teams ( 3 ) . Despite growing interest in deploying these teams in high-stakes environments ( 16 ; 15 ; 12 ) , widespread adoption is often hindered by the fact that team success depends on the calibration of trust between agents. As established in 13 , human interaction with automation is characterized by use, misuse, or disuse, depending on whether the human agent accurately perceives the system’s underlying capabilities and intent. Unfortunately, maintaining calibrated trust is non-trivial. This is especially true as AI agents become more sophisticated, and potentially more deceptive. For example, modern LLM agents are prone to sycophancy, often prioritizing human agreement over factual correctness and failing to intervene in harmful behavioral patterns ( 14 ) . Importantly, prior work on deceptive behavior in LLMs hypothesizes that maliciously compromised models may strategically farm trust through helpful actions in order to later achieve greater harm ( 7 ) . Moreover, research on AI adversaries has already demonstrated that such manipulation can occur purely through behavioral channels, leaving no linguistic trace for traditional safety filters to detect ( 1 ) . To address this vulnerability in human-AI teams, we investigate the capability of LLM agents to serve as overseers that detect adversarial intent through action patterns alone. This approach is distinguished from prior research on LLM-based adversary detection (e.g., 11 ) in that the signal is strictly non-verbal. To accomplish this task, we leverage a dataset of human-AI interactions in the context of a 25-round trivia game between three human agents and a malicious AI manipulator. We demonstrate that off-the-shelf models are capable of identifying adversarial behavior given a sufficient signal length, even with zero prior information regarding the specific task. Furthermore, we find that in even in minimal information settings where the overseer has no knowledge of the task or the adversary’s strategy, lightweight commercial models can be fine-tuned to reliably flag adversarial behavior. Our results suggest that LLMs possess the ability to recognize patterns of strategic deception from behavioral traces alone. By evaluating LLMs’ capability to detect adversarial behavior, our work connects human cognitive modeling with AI oversight and highlights a promising path toward safer human–AI collaboration. 2 Related work Strategic deception in multi-agent reinforcement learning (MARL) Recent work shows that LLMs can execute sophisticated deception strategies in pursuit of misaligned goals ( 4 ) . This ability raises concerns given human automation bias and users’ tendency to over-rely on AI suggestions ( 13 ) . Beyond language models, cooperative multi-agent reinforcement learning has shown that even simple, non-linguistic agents are vulnerable to adversarial exploitation ( 6 ; 5 ) . In such settings, attackers can conceal malicious intent and manipulate team dynamics from within, allowing even a single black-box adversary to substantially degrade team performance ( 10 ; 9 ) . LLMs as overseers and human-AI trust calibration LLMs have attracted interest as automated judges ( 8 ) , but two key obstacles remain. First, LLMs exhibit systematic biases that can lead to incorrect assumptions about agent intent ( 18 ) . Second, prior work has largely evaluated deception detection in structured social games such as Mafia ( 19 ) , which rely on explicit game rules and access to ground-truth dialogue. In contrast, we evaluate whether an LLM can act as a task-agnostic overseer, detecting malicious intent from heterogeneous behavioral signals without access to task-specific information. Online anomaly detection Traditional anomaly detection focuses on unsupervised identification of outliers in structured time-series data ( 2 ) . While LLMs have been benchmarked for zero-shot anomaly detection in static NLP settings ( 17 ) , their use in monitoring dynamic behavioral streams remains underexplored. We evaluate LLMs as online, task-agnostic anomaly detectors that identify adversarial intent across sequential interactions. 3 Methods Figure 1: Overview of single-round behavioral detection pipeline. Round data from a human-AI team interaction is serialized and analyzed by an LLM-based observer. In the 1-round setting, individual classifications are aggregated via an anomaly detection algorithm to identify malicious intent over n detections. The gray feedback loop represents the planned integration of real-time intervention. Problem formalization: We use the dataset introduced in 1 , which consists of interactions between three humans and one AI assistant during a 25-round trivia game. In that setting, humans allocate fractional trust to teammates based on perceived reliability, while the AI assistant follows a model-based reinforcement learning (MBRL) policy to strategically manipulate trust and degrade team performance. Our subsequent analysis relies on the adversarial trajectories recorded in this dataset as a rich source of examples where the AI strategically leverages helpful behavior to gain trust before exploiting the team. Behavioral detection pipeline. Figure 1 depicts our pipeline. We convert each interaction round into a structured, JSON-serialized object containing human chat logs, agent selections, the round state, post-discussion trust allocations, and the ground-truth answer. Multiple rounds can optionally be combined into a single JSON object to form a sample. The samples contain no information about the trivia question or answer options, making the task fully task-agnostic. An LLM-based observer f then performs binary classification on each sample to determine whether the AI behavior is adversarial, relying solely on behavioral signals. For detailed prompt information, see Appendix A . Trajectory windowing. We evaluate LLM detection capabilities over three observation windows: 10-round, 5-round, and 1-round trajectories. The 10-round window provides the maximum available behavioral context, while the 5-round window evaluates whether shorter histories support earlier detection. The 1-round window captures a real-time, streaming setting in which decisions must be made from minimal context. Together, these windows allow us to study how detection performance varies with the amount of available behavioral information. Observability constraints and task-agnostic detection. We study the problem of detecting adversarial intent from behavioral traces alone, without access to task semantics, privileged signals, or the adversary’s policy. Our LLM agent receives only

Read the complete text