Preventing Rogue Agents Improves Multi-Agent Collaboration

Ohav Barbi Ori Yoran Mor Geva · 2025

Read Preventing Rogue Agents Improves Multi-Agent Collaboration by Ohav Barbi Ori Yoran Mor Geva (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-05986.txt.

Introduction

Ohav Barbi, Ori Yoran, and Mor Geva study how one rogue agent can sink a whole team of cooperating AI agents. In multi-agent systems, members share information and any one of them may end the task with a final answer, even when unsure. A wrong move by a single agent can then fail the entire system.

The team proposes watching agents as they form their answers and stepping in when an error looks likely. To test the idea, they built WhoDunitEnv, a collaboration setting where task complexity and communication structure can be tuned. Experiments there, plus tests on code writing and a resource sustainability simulation, showed gains of up to about 17, 2.5, and 20 percent. Their monitors caught the moments when agents grew confused, and the interventions stopped errors from spreading. The authors argue that guarding against rogue moves should be a standard part of multi-agent design.

Catalog record

CREATOROhav Barbi Ori Yoran Mor Geva
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-05986

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Preventing Rogue Agents Improves Multi-Agent Collaboration Ohav Barbi Ori Yoran Mor Geva Abstract Multi-agent systems, where specialized agents collaborate to solve a shared task hold great potential, from increased modularity to simulating complex environments. However, they also have a major caveat – a single agent can cause the entire system to fail. Consider a simple game where the knowledge to solve the task is distributed between agents, which share information in a communication channel. At each round, any of the agents can terminate the game and make the final prediction, even if they are uncertain about the outcome of their action. Detection of such rogue agents before they act may prevent the system’s failure. In this work, we propose to monitor agents during action prediction and intervene when a future error is likely to occur. To test our approach, we introduce WhoDunitEnv , a multi-agent collaboration environment that allows modular control over task complexity and communication structure. Experiments on two variants of WhoDunitEnv and the GovSim environment for resource sustainability show that our approach leads to substantial performance gains up to 17.4% and 20%, respectively. Moreover, a thorough analysis shows that our monitors successfully identify critical points of agent confusion and our interventions effectively stop agent errors from propagating. We release WhoDunitEnv and our code for future studies on multi-agent collaboration at https://github.com/Ohav/rogue-agents . 1 Introduction Recently there has been a growing interest in leveraging large language models (LLMs) to solve complex multi-agent tasks Guo et al. 2024a ; Tran et al. 2025 , such as improving factuality and reasoning, Du et al. 2024 ; Liang et al. 2024 simulating software development teams Hong et al. 2024 ; Qian et al. 2024 ; Liu et al. 2024c , and human interactions Park et al. 2023 . A key requirement for succeeding at such cooperative tasks is the agent’s ability to communicate effectively. Figure 1: An overview of our approach . We propose to improve multi-agent collaboration by monitoring agent communication and applying interventions to the environment in case rogue agents are detected. However, establishing effective communication is a major challenge for current LLMs, which often ignore critical information in the communication Liu et al. 2024b ; Levy et al. 2024 , become distracted by irrelevant messages Shi et al. 2023 ; Amayuelas et al. 2024 , and introduce hallucinations Xiao and Wang 2021 ; Zhang et al. 2023 ; Jiang et al. 2024 ; Lin et al. 2024 ; Huang et al. 2025 . Such failures introduce noise to the communication, which can be amplified as LLM responses are generated for many steps Zhang et al. 2024a ; Ivgi et al. 2024 ; Yoran et al. 2024 ; Jimenez et al. 2024 . Consequently, a single agent can cause an entire system failure. To improve multi-agent collaboration, recent works have proposed introducing changes to the communication protocol Guo et al. 2024b ; Mehta et al. 2024 , augmenting the communication with agent beliefs Li et al. 2023b , and adding specific modules for reasoning and grounding Agashe et al. 2023 ; Hong et al. 2024 . While these approaches often yield improvements in the overall task performance, they do not prevent cases where one rogue agent introduces noise to the communication and drags the system into failure D’Arcy et al. 2024 ; Hong et al. 2024 . In this work, we propose to improve multi-agent collaboration through live monitoring and interventions in agent communication. Specifically, we hypothesize that many failures can be detected and prevented early on — before the agent becomes non-functional and the task fails — based on the state of the agent and how it observes the environment (see Fig. 1 for an illustration). This approach is widely used in industrial systems, such as intrusion detection software and critical manufacturing processes Stavropoulos et al. 2013 ; Baldauf et al. 2021 ; Olajiga et al. 2024 ; Woods and Seymour 2023 ; Salem et al. 2024 , and is also observed in natural systems such as the human immune system, which monitors faulty behavior and prevents infections from spreading throughout the body. To test this hypothesis, we present a monitor and intervention framework (§ 2 ). To monitor an agent’s state, we employ simple classifiers that detect rogue agents based on intrinsic signals of uncertainty, such as the prediction’s entropy during action selection. This monitor then triggers an intervention mechanism, which prevents the communication from being distorted and the agents from failing at the task. Specifically, we experiment with simple interventions of restarting certain parts of the communication or the entire environment. To evaluate our approach, we create a modular collaborative environment, called WhoDunitEnv , which allows analyzing different agent communication protocols at varying task-complexity levels (§ 3 ). WhoDunitEnv is inspired by the well-known game of Guess Who , 1 1 1 https://en.wikipedia.org/wiki/Guess_Who where players (here agents) collaborate to identify a culprit among a group of suspects based on their appearance properties. Next (§ 4 ), we conduct experiments with Llama-3.1-70B Grattafiori et al. 2024 , Qwen-2.5-72B Yang et al. 2024 , and GPT-4o Achiam et al. 2023 on two variants of WhoDunitEnv and the GovSim environment Piatti et al. 2024 , where agents should consume resources in a cooperative and sustainable manner. Notably, these environments cover symmetric and asymmetric agents, structured and free-form communication protocols, and varying levels of task complexities. Our results (§ 5 ) show that, across all settings, monitoring rogue agents leads to substantial improvements in multi-agent collaboration. Specifically, we observe gains of up to 20%. Finally, we conduct ablations and qualitative analysis (§ 6 ), showing that strong monitors and interventions are needed to improve performance, and that our monitors are effective at identifying a range of rogue agents, including when agents hallucinate information in the communication channel. To summarize, our work makes the following contributions: (a) we propose the notion of live monitoring and interventions to prevent failures due to rogue agents in multi-agent systems, (b) we introduce WhoDunitEnv , a modular environment for studying collaborative multi-agent systems where agents are either symmetrical/asymmetrical, (c) we show that our approach leads to substantial performance gains on WhoDunitEnv and GovSim across models and complexity levels, and (d) we show that improvements are due to strong monitors and interventions. 2 Monitoring and Intervening in Multi-Agent Systems In multi-agent systems, agents collaborate in order to solve a task or use shared resources. The system is composed of a group of agents G=\{g_{i}\} , a communication channel C that stores messages shared by the agents, and a shared task T . Agents can differ in the knowledge they posses K_{i} and the actions they can perform A_{i} . For example, agents that simulate human behavior have different memories based on their experiences Park et al. 2023 , and different personas in a software development team have different actions, such as designing the code, programming, or writing tests Qian et al. 2024 . Let C_{j} be the communication channel at the j -th turn, an agent g_{i} chooses their next action using a probability distribution over actions conditioned on their knowledge and the shared information: P_{a\in A_{i}}(a,j,g_{i},T)=P(\texttt{Action}=a\,|\,K_{i},C_{j},T) Similarly, agents share information in the channel with a distribution

Read the complete text