Talk Isn’t Always Cheap: Understanding Failure Modes in Multi-Agent Debate
Andrea Wynn, Harsh Satija, Gillian Hadfield · 2025
Read Talk Isn’t Always Cheap: Understanding Failure Modes in Multi-Agent Debate by Andrea Wynn, Harsh Satija, Gillian Hadfield (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-05396.txt.
Introduction
Letting several AI models debate can make answers worse, Andrea Wynn, Harsh Satija, and Gillian Hadfield find in a 2025 paper on multi-agent debate. Debate is often sold as a way to improve reasoning, and earlier work mostly studied debates among models of similar strength.
Mixing models of different capability changes the picture. Across experiments, accuracy sometimes fell over time even when stronger models outnumbered weaker ones. Models often moved from right answers to wrong ones after hearing peers, favouring agreement over challenge. The authors probe causes including flattery-like agreement, social conformity, and task type. Naive debate, they conclude, can degrade performance unless agents are equipped to resist persuasive errors.
Catalog record
| CREATOR | Andrea Wynn, Harsh Satija, Gillian Hadfield |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-05396
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Talk Isn’t Always Cheap: Understanding Failure Modes in Multi-Agent Debate Andrea Wynn, Harsh Satija, Gillian Hadfield Abstract While multi-agent debate has been proposed as a promising strategy for improving AI reasoning ability, we find that debate can sometimes be harmful rather than helpful. The prior work has exclusively focused on debates within homogeneous groups of agents, whereas we explore how diversity in model capabilities influences the dynamics and outcomes of multi-agent interactions. Through a series of experiments, we demonstrate that debate can lead to a decrease in accuracy over time — even in settings where stronger (i.e., more capable) models outnumber their weaker counterparts. Our analysis reveals that models frequently shift from correct to incorrect answers in response to peer reasoning, favoring agreement over challenging flawed reasoning. These results highlight important failure modes in the exchange of reasons during multi-agent debate, suggesting that naive applications of debate may cause performance degradation when agents are neither incentivized nor adequately equipped to resist persuasive but incorrect reasoning. † † affiliationnotice: Equal contribution 1 Introduction Large Language Model (LLM) agents have demonstrated remarkable problem-solving abilities across a wide array of complex reasoning tasks ( Brown et al. 2020 ) . Recently, a new line of research on interactive reasoning among multiple LLMs through debate has promoted the multi-agent debate framework as a promising approach to enhancing the reasoning and decision-making capabilities of LLM agents ( Du et al. 2023 ; Chan et al. 2023 ; Liang et al. 2023 ; Khan et al. 2024 ) . Various forms of multi-agent debate have been shown to improve performance on multiple arithmetic and strategic reasoning benchmarks ( Du et al. 2023 ; Subramaniam et al. 2025 ) , produce more truthful answers and evaluations ( Chan et al. 2023 ; Khan et al. 2024 ) , and enhance tasks such as machine translation ( Liang et al. 2023 ) and negotiation ( Fu et al. 2023 ) . The core concept of these studies is that by engaging LLM agents through structured argumentation or discourse, we can facilitate the exchange of reasoning among different agents and guide them toward more accurate answers. However, most of these techniques incur significant computational overhead, as they are inference or test-time methods that rely on the strong zero-shot or few-shot capabilities of LLMs. Intuitively, greater exchanges of reasoning should lead to better decisions—allowing multiple agents to challenge flawed reasoning, highlight overlooked details, and reduce individual biases. But is this always the case? In this work, we show that the benefits of multi-agent debate are not as universal as commonly assumed. Through a series of empirical studies, we show that multi-agent debate can sometimes degrade performance, leading to worse final answers than those generated by a single agent acting alone. These failures are not rare edge cases, but arise systematically in settings where agents amplify each other’s errors – agreeing reflexively rather than challenging flawed reasoning. These findings hold true even when there is variation in the abilities of the participating LLM agents. For instance, we discover that introducing a weak or less capable (lower-performing) LLM agent into a debate with a strong or more capable (higher-performing) agent can detrimentally affect the debate outcome, producing results worse than if the agents had not engaged in discussion. The presence of a weaker agent disrupts the performance of the stronger agent. Moreover, in certain cases, the longer a debate continues, the more performance can degrade. In other words: talk isn’t always cheap – and in some cases, it’s actively harmful. We present a systematic evaluation of multi-agent debate across multiple tasks, showing that debate can sometimes harm group performance, particularly with heterogeneous LLM agents engaged in debate. Our findings challenge the prevailing narrative that more discussion between agents is inherently beneficial. Instead, we uncover several key factors that mediate the success or failure of debate, including task type and complexity, agent diversity and capability. In doing so, we offer a nuanced view of when and why debate helps – and when it hurts. Together, these results suggest that while debate remains a promising tool for improving model reasoning, its success is far from guaranteed. 2 Related Work Debate and multi-agent reasoning. Multi-agent debate was initially proposed as method for scalable oversight problem where a judge or verifier can interject and elicit hidden contradictions, using structured back-and-forth conversations ( Irving et al. 2018 ; Khan et al. 2024 ; Michael et al. 2023 ; Kenton et al. 2024 ) . More recently, another form of multi-agent debate, sometimes referred to as multi-agent deliberation, investigates at leveraging different LLM agents to surface better answers by having them exchange reasoning via iterative discussion ( Du et al. 2023 ; Chan et al. 2023 ; Liang et al. 2023 ; Subramaniam et al. 2025 ) . Most of these studies focus on a homogeneous setting where all LLM agents utilize the same underlying language model, finding that this approach enhances accuracy across various Question-Answering (QA) tasks. Estornell & Liu 2024 examine the theoretical properties and effects of opinion diversity within the debate framework, reporting a“tyranny of the majority” effect. They found that if the majority of agents provide the same answer—regardless of its correctness—minority agents tend to conform, creating an echo chamber effect. Their analysis was limited to a homogeneous setting, whereas we extend our analysis to heterogeneous settings. Our findings reveal that even when the majority consists of stronger models, introducing a weaker model can diminish overall performance, thereby complementing their results. Some works explores debates between agents of diverse nature or abilities. For instance, Estornell et al. 2025 ; Subramaniam et al. 2025 propose training LLM agents to debate collaboratively with distinct roles (actors/generators and critics), demonstrating that this approach can surpass previous unsupervised debate setups in reasoning benchmarks. Finally, studies such as Amayuelas et al. 2024 investigate whether the collaborative nature holds if an explicit adversary is introduced into the debate process—where the adversary actively seeks to reduce performance. These works suggest that, when agents constructively challenge each other, answers can improve. However, other studies caution that debate can fail when agents emphasize persuasion over truth. Agarwal & Khanna 2025 introduce a single-round debate on factual questions (using TruthfulQA) where one agent states a true answer calmly and another delivers a confident, emotional false answer. They show the LLM judge often chooses the persuasive falsehood with high confidence, suggesting that a vivid but incorrect argument can override a correct one. These results highlight a risk: unless the judge is well-calibrated, debate may amplify bluster, mirroring human misinformation scenarios. Collaborative multi-agent frameworks. Beyond explicit debates, many recent systems assume collaboration among LLMs improves reasoning ( Li et al. 2023 ; Tran et al. 2025 ) . For instance, Wu et al. 2023 provides a general multi-agent conversation framework: developers can define many agents (assistant, user-proxy, tools, etc.) that autonomously chat to solve complex tasks. These role-based and decentralized systems often yield richer interactions (e.g. multi-turn planning) than a single LLM alone. Sequential revision: Frameworks that utilize interactive reasoning at inference time in a sequential manner are employed for either self-refinement ( Madaan et al. 2023 ) or self-consistency ( Wang et al. 2022 )