Insider Attacks in Multi-Agent LLM Consensus Systems

Xiaolin Sun, Zixuan Liu, Yibin Hu, Zizhan Zheng · 2026

Read Insider Attacks in Multi-Agent LLM Consensus Systems by Xiaolin Sun, Zixuan Liu, Yibin Hu, Zizhan Zheng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08268.txt.

Introduction

Xiaolin Sun, Zixuan Liu, Yibin Hu, and Zizhan Zheng study a saboteur inside a team of language model agents. Groups of agents increasingly reach decisions by exchanging messages until they agree, and most frameworks assume every member shares the group's goal. A malicious insider can instead pose as a normal member while secretly working to delay or block consensus.

The authors formalize the attack as a sequential decision problem. To optimize it, they learn a world model that mimics the latent behavior of the honest agents, then train the attacker against that surrogate. The study maps how fragile multi-agent agreement can be from the inside.

Catalog record

CREATORXiaolin Sun, Zixuan Liu, Yibin Hu, Zizhan Zheng
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.08268

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Insider Attacks in Multi-Agent LLM Consensus Systems Xiaolin Sun, Zixuan Liu, Yibin Hu, Zizhan Zheng Abstract Large language models (LLMs) are increasingly deployed in multi-agent systems where agents communicate in natural language to solve tasks jointly. A key capability in such systems is consensus formation, where agents iteratively exchange messages and update decisions to reach a shared outcome. However, most existing multi-agent LLM frameworks assume that all participating agents are aligned with the system objective. In practice, a malicious insider may participate as a legitimate member of the group while pursuing a hidden adversarial goal. In this work, we study insider manipulation in multi-agent LLM consensus systems. We formalize the problem as a sequential decision-making task in which a malicious agent seeks to delay or prevent agreement among benign agents. To make attack optimization tractable, we propose a world-model-based framework that learns surrogate dynamics over the latent behavioral states of benign agents and then trains an attacker using reinforcement learning based on this learned model. Preliminary results show that the trained attacker reduces the benign consensus rate and prolongs disagreement more effectively than the direct malicious-prompt baseline. These results suggest that combining latent world models with reinforcement learning is a promising direction for adaptive insider attacks in language-based multi-agent systems. 1 Introduction Multi-agent LLM systems have recently emerged as a promising paradigm for collaborative AI, where multiple language-model-based agents interact through natural language to perform reasoning, planning, negotiation, and collective decision-making ( Li et al. 2023 ; Wu et al. 2023 ; Tran et al. 2025 ) . Compared with single-agent pipelines, such systems can benefit from diversity of perspectives, iterative refinement, and cross-agent verification ( Wu et al. 2023 ; Tran et al. 2025 ) . A central mechanism in many of these systems is consensus formation : agents exchange messages over multiple rounds and eventually produce a shared decision ( Chen et al. 2023 ) . Despite their promise, existing multi-agent LLM frameworks often assume that all agents are aligned with the same task objective ( Li et al. 2023 ; Wu et al. 2023 ) . This assumption is fragile in practice. A malicious participant may act as an insider attacker , participating through the same interface as benign agents while pursuing a hidden goal that conflicts with the collective objective. Unlike external attacks that perturb prompts or infrastructure from outside the system, insider attacks operate from within the deliberation process itself. In language-based systems, this threat is especially difficult to study because the attacker can influence others through subtle natural-language strategies such as persuasion, framing, selective agreement, or ambiguity ( Ashery et al. 2025 ; Ren et al. 2024 ) . Recent work on agentic misalignment and multi-agent attack settings further suggests that seemingly cooperative LLM agents can exhibit strategically harmful behavior while remaining superficially consistent with their assigned roles ( Anthropic 2025 ; Yu et al. 2025 ) . More broadly, related work in opinion dynamics and social learning highlights that collective decision-making systems are often fragile under malicious influence, and that robustness depends strongly on the aggregation mechanism ( Gaitonde et al. 2020 ; Amir et al. 2025 ) . Motivated by these observations, we study insider manipulation in multi-agent LLM consensus systems and ask whether a malicious LLM agent can strategically disrupt collective agreement while remaining superficially similar to benign participants. Our contributions are as follows: • Unlike prior work that relies on prompt-based malicious insiders ( Anthropic 2025 ; Yu et al. 2025 ) , we cast insider manipulation in multi-agent LLM consensus as a sequential decision-making problem and develop a reinforcement learning (RL)-based attack. • To improve sample efficiency, our attack is model-based: a latent-agent world model is inferred from trajectory data to capture heterogeneous benign-agent response patterns and enable efficient attacker optimization. • We instantiate the framework in a 1-D consensus setting and provide preliminary evidence that the learned attacker is stronger than a direct malicious-prompt baseline. A detailed summary of related work is given in Appendix A . 2 System and Threat Models System Model. We consider a multi-turn LLM-agent consensus system with N agents interacting over multiple rounds. Given a task prompt x , agents repeatedly exchange natural-language messages with their neighbors and update their decisions before producing a final collective outcome. Formally, let \mathcal{I}=\{1,\dots,N\} be the set of agents, and let \mathcal{B}\subseteq\mathcal{I} and \mathcal{M}=\mathcal{I}\setminus\mathcal{B} denote the sets of benign and malicious agents, respectively. Each agent i is initialized with a system prompt p_{i}^{\mathrm{sys}} and a behavioral type or role \psi_{i} . Let \mathcal{N}_{i}\subseteq\mathcal{I} denote the set of agents whose messages and decisions are visible to agent i . At round t , agent i observes a local multi-turn context h_{i}^{t}=(x,p_{i}^{\mathrm{sys}},\psi_{i},\{m_{j}^{\ell},y_{j}^{\ell}\}_{j\in\mathcal{N}_{i},\,\ell=0}^{t-1}) , which includes the task prompt, its own system prompt, its role or behavioral profile, and the previous messages and decisions visible from its neighbors in earlier rounds. Based on this local context, agent i jointly generates a natural-language message and a current decision, (m_{i}^{t},y_{i}^{t})\sim\pi_{i}(\cdot\mid h_{i}^{t}) , where m_{i}^{t} is the message sent to its neighbors and y_{i}^{t}\in\mathcal{Y} is the agent’s effective decision state at round t . The interaction context is then updated with the newly generated messages and decisions. The objective of the benign agents is to reach consensus over the decision space \mathcal{Y} within a maximum of T communication rounds, namely y_{i}^{t}=y_{j}^{t} for all i,j\in\mathcal{B} . Equivalently, if d(\cdot,\cdot) is a discrepancy measure on \mathcal{Y} , we define a disagreement functional \Delta(\mathbf{y}_{\mathcal{B}}^{t}) , where \mathbf{y}_{\mathcal{B}}^{t}=\{y_{i}^{t}:i\in\mathcal{B}\} , such that consensus is achieved when \Delta(\mathbf{y}_{\mathcal{B}}^{t})=0 . We define the disagreement functional as the maximum pairwise discrepancy among benign agents, \Delta(\mathbf{y}_{\mathcal{B}}^{t})=\max_{i,j\in\mathcal{B}}d(y_{i}^{t},y_{j}^{t}). Insider Threat Model. We study an insider attack in which one or more malicious agents participate as legitimate members of the multi-turn LLM-agent consensus process. Unlike external attacks that perturb prompts, infrastructure, or communication channels from outside the system, insider attackers operate through the same interface as benign agents: they observe their local multi-turn context, send natural-language messages to their neighbors, and declare decisions as ordinary participants, but do so with a hidden objective that differs from the system goal. For a malicious agent k\in\mathcal{M} , the attacker only observes information available through its neighborhood. At round t , its state is s_{\mathrm{adv}}^{t}=\left(h_{k}^{t},\mathbf{y}_{\mathcal{N}_{k}}^{t},\boldsymbol{\psi}_{\mathcal{N}_{k}}\right), where h_{k}^{t} is its local multi-turn context, \mathbf{y}_{\mathcal{N}_{k}}^{t} denotes the current decisions of visible neighboring agents, and \boldsymbol{\psi}_{\mathcal{N}_{k}} denotes their behavioral attributes. Since malicious agents may coordinate with each other, \mathcal{N}_{k} can include both benign and malicious neighbors. Thus, the attacker does not observe the full population unless the communication graph is fully connected. The attacker’s action at round t is its message–decision pair a_{\mathrm{adv}}^{t}=(m_{k}^{t},y_{k}^{t}) , where m_{k}^{t} is the natural-language message sent to its neighbors and y_{k}^{t} is its declared decision. The attacker’s objective is to prevent, delay, or degrade consensus among the benign agents while remaining behaviorally plausible. The attacker cannot directly modify benign agents’ prompts, memories, or outputs. Instead, it influences the evolving dialogue through misleading arguments, selective agreement, framing, ambiguity, or strategic disagreement. At a high level, the attacker solves a sequential decision-making problem: \max_{\pi_{\mathrm{adv}}}\mathbb{E}\left[\sum_{t=0}^{T-1}\gamma^{t}r_{\mathrm{adv}}^{t}\right], where T denotes the maximum number of communication rounds, and r_{\mathrm{adv}}^{t} is the attacker’s reward at time step t , designed to penalize benign

Read the complete text