When AI Agents Collude Online: Financial Fraud Risks by Collaborative LLM Agents on Social Platforms WARNING: This paper contains model outputs that may be considered offensive.
Qibing Ren Zhijie Zheng Jiaxuan Guo Junchi Yan Lizhuang Ma Jing Shao, [2mm] Shanghai Jiao Tong University Shanghai Artificial Intelligence Laboratory Beihang University[3mm] [email protected] [email protected] · 2025
Read When AI Agents Collude Online: Financial Fraud Risks by Collaborative LLM Agents on Social Platforms WARNING: This paper contains model outputs that may be considered offensive. by Qibing Ren Zhijie Zheng Jiaxuan Guo Junchi Yan Lizhuang Ma Jing Shao, [2mm] Shanghai Jiao Tong University Shanghai Artificial Intelligence Laboratory Beihang University[3mm] [email protected] [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06448.txt.
Introduction
Qibing Ren and co-authors test whether language model agents can team up to commit financial fraud online. Multi-agent systems raise a fresh worry: agents might coordinate scams rather than act alone. The team built MultiAgentFraudBench, a benchmark simulating realistic online interactions across twenty-eight common fraud scenarios and the full fraud life cycle.
Experiments measured how conversation depth, activity levels, and patterns of collaboration shape fraud success, including where teamwork breaks down. Defenses tested include content warnings, monitor models that block malicious agents, and group information sharing. Malicious agents adapted to some defenses, underlining real-world risk while pointing to practical countermeasures.
Catalog record
| CREATOR | Qibing Ren Zhijie Zheng Jiaxuan Guo Junchi Yan Lizhuang Ma Jing Shao, [2mm] Shanghai Jiao Tong University Shanghai Artificial Intelligence Laboratory Beihang University[3mm] [email protected] [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-06448
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
When AI Agents Collude Online: Financial Fraud Risks by Collaborative LLM Agents on Social Platforms WARNING: This paper contains model outputs that may be considered offensive. Qibing Ren Zhijie Zheng Jiaxuan Guo Junchi Yan Lizhuang Ma Jing Shao, [2mm] Shanghai Jiao Tong University Shanghai Artificial Intelligence Laboratory Beihang University[3mm] [email protected] [email protected] Abstract In this work, we study the risks of collective financial fraud in large-scale multi-agent systems powered by large language model (LLM) agents. We investigate whether agents can collaborate in fraudulent behaviors, how such collaboration amplifies risks, and what factors influence fraud success. To support this research, we present MultiAgentFraudBench, a large-scale benchmark for simulating financial fraud scenarios based on realistic online interactions. The benchmark covers 28 typical online fraud scenarios, spanning the full fraud lifecycle across both public and private domains. We further analyze key factors affecting fraud success, including interaction depth, activity level, and fine-grained collaboration failure modes. Finally, we propose a series of mitigation strategies including adding content-level warnings to fraudulent posts and dialogues, using LLMs as monitors to block potentially malicious agents, and fostering group resilience through information sharing at the societal level. Notably, we observe that malicious agents can adapt to environmental interventions. Our findings highlight the real-world risks of multi-agent financial fraud and suggest practical measures for mitigating them. Code is available at https://github.com/zheng977/MutiAgent4Fraud . † † footnotetext: ⋆ Equal contribution † Corresponding author 1 Introduction Multi-agent systems have already been widely deployed in real-world systems, ranging from coding tasks to general-purpose tasks ( 36 ; 45 ; 47 ) . These tasks are typically handled by several agents working together with a precise division of labor. In parallel, another line of research explores agent societies, where agents are given autonomy and self-interest, and large-scale interactions may give rise to emergent social phenomena such as cooperation ( 42 ; 13 ; 12 ) . These societies can be used to study complex social dynamics, and they can also be used to simulate activities that involve ethical risks. Among such risks, financial fraud is one of the most damaging. The rapid growth of social media platforms has further amplified this threat by providing fertile ground for fraud to scale ( 6 ) . Most prior research on agent societies has focused on collective intelligence, where agents collaborate to achieve beneficial outcomes ( 28 ; 37 ; 38 ) . Yet a critical question remains: what happens when such intelligence is directed toward malicious goals? Could the harm exceed the sum of individual capabilities? Financial fraud is often conducted collectively in human society, with groups coordinating to maximize success ( 39 ; 10 ) . Whether multi-agent systems may also exhibit similar collusive fraud behaviors has not been sufficiently studied. Considering the growing autonomy of LLM-based agents, malicious actors may exploit groups of agents to create scaling risks. This makes the study of collective fraud not a theoretical concern but an urgent, practical problem. In this work, we present a systematic study of financial fraud collusion in LLM-driven multi-agent systems, addressing three fundamental questions: ( i ) Can multi-agents collaborate in fraud? Does this amplify the risks? ( ii ) What factors are critical to the success of a fraud operation? ( iii ) How can we mitigate these risks? To answer Question ( i ), we propose MultiAgentFraudBench (Section 3 ), a large-scale multi-agent collective financial fraud benchmark, which builds on the OASIS simulation framework ( 42 ) . Our benchmark covers 28 fraud scenarios drawn from the Stanford fraud taxonomy ( 7 ) , encompassing a wide spectrum of online fraud cases, and contains 2800 posts. Crucially, we extend OASIS beyond the public domain by introducing private peer-to-peer communication, enabling more realistic simulations of the fraud lifecycle. As shown in Figure 1 , malicious agents start by attracting attention on social media, build hype, gain trust in private messages, and finally deceive people to steal their money. Benign users can also inform the community about how they were scammed. To make our simulation faithfully mirror real-world conditions, we construct a threat model to define our simulation boundary, including realistic ratios of malicious to benign agents, comparable knowledge and activity levels, and freedom to interact through standard social media actions. We define two quantitative metrics to evaluate performance: conversation-level fraud success and population-level fraud impact. Building on the insights from our investigation (Section 4.2 ), we address Question ( ii ) by examining two key factors: interaction depth and activity level (Section 5 ) and conducting fine-grained analyses to uncover common failure modes in fraudulent behaviors by malicious agents (Section 5.3 ). To address Question ( iii ), we find that malicious agents can adapt to simple environmental interventions, such as adding warnings to private chat contexts. Interestingly, those powered by DeepSeek-V3 even achieve higher fraud success rates under such conditions (Section 7 ). Therefore, we further explore two prompt-based strategies to mitigate fraud risks: (1) developing agents as monitors to detect and block potential malicious agents (Section 6.2 ), and (2) enhancing group resilience by encouraging reporting and information sharing among benign agents (Section 6.3 ). This comprehensive analysis aims to reveal the potential risks of multi-agent financial fraud in human society, identify their root causes, and propose effective intervention strategies. In a nutshell, our contributions are threefold: 1. We propose MultiAgentFraudBench, the first large-scale benchmark to systematically study collective financial fraud in multi-agent societies, covering realistic scenarios and the full fraud lifecycle across public and private domains. 2. We present a comprehensive empirical study of collective fraud, evaluated with conversation-level and population-level success metrics. 3. We analyze key factors behind fraud success and investigate potential mitigation strategies, offering insights into the misuse risks of collaborative AI systems in society. Figure 1: ( left ): a diagram of fraud activities on social media: multiple malicious actors targeting benign users. ( middle ): at each time step, the recommendation system distributes posts to users, and users react to the posts or to messages from other users; ( right ): examples of agents evolving and colluding, and the three levels of mitigation we propose. 2 Related Work The difference between multi-agent systems (MAS) and agent societies lies in autonomy, scale, and goals. MAS research typically focuses on multiple agents cooperating with role specialization to complete one well-defined task, such as designing software or developing websites. In contrast, agent societies emphasize granting agents sufficient autonomy and studying the dynamics of large-scale interactions. These agents have their own interests and personalities, and pursue individual goals. In this paper, we evaluate the risks posed by malicious agents collaborating within an agent society to conduct financial fraud. 2.1 Safety of Multi-Agent Systems Most existing work examines whether the introduction of malicious agents disrupts MAS collaboration. For example, PsySafe ( 46 ) and Evil Geniuses ( 34 ) study how malicious prompts can be injected into MAS. Agent Smith ( 15 ) investigates the spread of harmful behaviors among agents, and other work shows how toxic information may propagate within multi-agent systems ( 19 ) . Additional studies explore the robustness of different topologies under adversarial conditions ( 17 ) . Closer to our evaluation setting, ( 43 ) analyzes