Comprehensive Vulnerability Analysis is Necessary for Trustworthy LLM-MAS
Pengfei He*, Yue Xing*, Shen Dong, Juanhui Li, Zhenwei Dai, Xianfeng Tang, Hui Liu, Han Xu, Zhen Xiang, Charu C. Aggarwal, Hui Liu · 2025
Read Comprehensive Vulnerability Analysis is Necessary for Trustworthy LLM-MAS by Pengfei He*, Yue Xing*, Shen Dong, Juanhui Li, Zhenwei Dai, Xianfeng Tang, Hui Liu, Han Xu, Zhen Xiang, Charu C. Aggarwal, Hui Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01245.txt.
Introduction
Pengfei He and co-authors argued in a 2025 paper that multi-agent language systems need a full weakness audit before they can be trusted. Such systems team up many language-model agents that talk, share trust, and call tools as one crew. Firms now field them for high-stakes work, yet their joint form opens attack paths a lone model lacks.
Flaws of single agents are well studied, but the seams between agents are not. Messages can be poisoned, trust can be abused, and tool links can be hijacked. The authors lay out a frame for mapping those seams. Their case is that trust must be earned by audit, not assumed.
Catalog record
| CREATOR | Pengfei He*, Yue Xing*, Shen Dong, Juanhui Li, Zhenwei Dai, Xianfeng Tang, Hui Liu, Han Xu, Zhen Xiang, Charu C. Aggarwal, Hui Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01245
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Comprehensive Vulnerability Analysis is Necessary for Trustworthy LLM-MAS Pengfei He*, Yue Xing*, Shen Dong, Juanhui Li, Zhenwei Dai, Xianfeng Tang, Hui Liu, Han Xu, Zhen Xiang, Charu C. Aggarwal, Hui Liu Abstract This paper argues that a comprehensive vulnerability analysis is essential for building trustworthy Large Language Model-based Multi-Agent Systems (LLM-MAS) . These systems, which consist of multiple LLM-powered agents working collaboratively, are increasingly deployed in high-stakes applications but face novel security threats due to their complex structures. While single-agent vulnerabilities are well-studied, LLM-MAS introduces unique attack surfaces through inter-agent communication, trust relationships, and tool integration that remain significantly underexplored. We present a systematic framework for vulnerability analysis of LLM-MAS that unifies diverse research. For each type of vulnerability, we define formal threat models grounded in practical attacker capabilities and illustrate them using real-world LLM-MAS applications. This formulation enables rigorous quantification of vulnerability across different architectures and provides a foundation for designing meaningful evaluation benchmarks. Our analysis reveals that LLM-MAS faces elevated risk due to compositional effects—vulnerabilities in individual components can cascade through agent communication, creating threat models not present in single-agent systems. We conclude by identifying critical open challenges: (1) developing benchmarks specifically tailored to LLM-MAS vulnerability assessment, (2) considering new potential attacks specific to multi-agent architectures, and (3) implementing trust management systems that can enforce security in LLM-MAS. This research provides essential groundwork for future efforts to enhance LLM-MAS trustworthiness as these systems continue their expansion into critical applications. 1 Introduction Large Language Model-based Multi-Agent Systems (LLM-MAS) represent a significant advancement in AI collaboration and automation. In an LLM-MAS, multiple LLM-based agents, assigned specialized roles and equipped with various tools, can communicate, reason and collaborate with each other [ 1 , 2 , 3 ] . Therefore, compared with LLMs and a single LLM agent, LLM-MAS shows more advanced capabilities in tackling complex tasks and already powers non - trivial deployments in software engineering [ 4 , 5 , 6 ] , embodied agents [ 7 , 8 ] , and scientific research [ 9 , 10 ] . Moreover, the advanced capabilities of LLM-MAS are driving their adoption in high-stakes domains—from fintech conversational agents (e.g., FinRobot) [ 11 ] to medical triage assistants (e.g., TriageAgent, MDAgents) [ 12 , 13 ] —further highlighting their potential and the growing momentum of their development. Despite the effectiveness and growing adoption of LLM-MAS, an unreliable and untrustworthy LLM-MAS can cause substantial safety consequences, especially for the security-critical domains. On the one hand, with the access to various tools, existing single-agent systems have already demonstrated potential vulnerability in outputting harmful outputs or executing malicious programs. For example, ChatGPT was exploited in a recent Cybertruck explosion incident in Las Vegas [ 14 ] , and OpenAI’s operator agent reportedly executed unauthorized $31.43 transactions despite safety protocol [ 15 ] . On the other hand, as demonstrated by [ 16 ] , the vulnerability in LLM-MAS can be further exaggerated as the system is exposed with more vulnerable components. This will cause harmful consequences such as users’ privacy leakage or system crash [ 17 ] . In addition, with the rise of Agent-to-agent (A2A) protocol, agents from different sources will collaborate in a system, which can be vulnerable if some agents are not verified properly. While prior work has addressed safety concerns for individual LLMs and single-agent systems [ 18 , 19 , 20 ] , LLM-MAS introduces fundamentally new and unique security challenges, yet these challenges remain significantly underexplored. Specifically, the presence of inter-agent communication, trust relationships, and tool calls together open novel attack surfaces. Current security discussions on LLM-MAS remain narrow in scope, often focusing on limited attack surfaces such as malicious agents [ 21 , 22 , 23 ] , or specific scenarios like error injection [ 23 , 22 ] . While these studies uncover some critical risks, they typically explore only a small subset of possible vulnerabilities and adopt relatively basic techniques—often adapted from general LLMs or single-agent settings. As a result, they do not reveal the full spectrum of weaknesses in LLM-MAS and are insufficient for systematic security evaluation. Moreover, many of these works introduce attack methods without a clear problem formulation, which hinders a deeper understanding of the system’s security landscape and limits the progress in both attack development and defense design. In particular, there is a lack of: (1) a broad taxonomy of potential vulnerabilities within LLM-MAS; (2) well-justified threat models; and (3) formal definitions of attack objectives that can guide the design of meaningful evaluations. The above implies that, the field lacks a holistic assessment of LLM-MAS threats, which is crucial to building a secure and trustworthy LLM-MAS. To address the aforementioned challenges, we argue that a comprehensive vulnerability analysis is necessary for trustworthy LLM-MAS . In this work, we take a systematic approach to identify critical attack surfaces and highlight those unique to LLM-MAS. For each identified vulnerability, we define feasible and well-justified threat models, grounded in practical constraints and attacker capabilities. These models are illustrated using real-world LLM-MAS applications and widely adopted frameworks such as MetaGPT [ 24 ] and ChatDev [ 6 ] , ensuring their relevance and applicability. In addition, we provide rigorous formulations of attack objectives, incorporating a wide range of malicious consequences such as breaking alignments, resource exhaustion and privacy leakage. These formulations serve as a foundation for designing meaningful and reproducible evaluations, paving the way for future benchmarks in LLM-MAS security research. The structure of this paper is as follows: In Section 2 , we review the basic architecture of LLM-MAS and its key components. In Section 3 , we introduce the proposed analysis framework, provide the basic mathematical formulation, summarize the general malicious goals, and comprehensively analyze the vulnerability in each component in LLM-MAS. Finally, Section 4 discusses open challenges and future directions based on the proposed analysis. 2 An Overview of the Architecture of LLM-MAS An LLM-MAS is a collaborative system composed of multiple LLM agents, each capable of autonomous reasoning, communication, and task execution. As shown in Figure 1 , the LLM-MAS can be decomposed into the following critical components: Individual Agents ( \mathcal{A}=\{A_{i}\}_{i=1}^{n} ). Let n denote the number of agents in the LLM-MAS, and each agent A_{i} is powered by an LLM f_{i} ( LLM core ) assigned with specific roles (e.g., planner, coder, verifier) through agent profile ( P_{i} , also known as system prompt). Each agent has access to a set of tools , T_{i}=\{t_{i,j}\}_{j=1}^{n_{i}} , such as the retriever to external databases and the calculator, where n_{i} represents the total number of available tools for the agent A_{i} . In this work, in addition to maintaining the tools in the local server, we also consider the Model Context Protocol (MCP) [ 25 ] : the LLM-MAS requests tools from MCP servers (hosted by third parties) and can obtain various tools from various MCP servers. Finally, the agent can also maintain its local memory, which contains received messages and previous experience, and interacts with other agents to finish tasks. Inter-agent Communication ( \mathcal{C}=(\mathcal{S},\mathcal{M},\mathcal{T}) ). Communication is a fundamental mechanism in LLM-MAS, allowing individual agents to interact with each other. The communication includes the communication structure, exchanged messages and trust management.