Revisiting Multi-Agent Debate as Test-Time Scaling: A Systematic Study of Conditional Effectiveness
Yongjin Yang, Euiin Yi, Jongwoo Ko, Kimin Lee, Zhijing Jin, Se-Young Yun, KAIST AI, MPI for Intelligent Systems, University of Toronto, Vector Institute · 2025
Read Revisiting Multi-Agent Debate as Test-Time Scaling: A Systematic Study of Conditional Effectiveness by Yongjin Yang, Euiin Yi, Jongwoo Ko, Kimin Lee, Zhijing Jin, Se-Young Yun, KAIST AI, MPI for Intelligent Systems, University of Toronto, Vector Institute (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22960.txt.
Introduction
Having several language model agents debate an answer is often sold as a better way to spend compute at test time. Yet its edge over a single agent refining its own work is unclear. Yang and co-authors compare debate with strong solo methods on math problems and safety tasks, varying problem hardness, model size, and how much the agents differ.
For math, debate helped little overall, though it gained as problems got harder and models got weaker, and agent variety made little difference. For safety tasks, debate raised attack success at first, though more varied agents slowly cut it back. The results guide when debate is worth its cost.
Catalog record
| CREATOR | Yongjin Yang, Euiin Yi, Jongwoo Ko, Kimin Lee, Zhijing Jin, Se-Young Yun, KAIST AI, MPI for Intelligent Systems, University of Toronto, Vector Institute |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-22960
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Revisiting Multi-Agent Debate as Test-Time Scaling: A Systematic Study of Conditional Effectiveness Yongjin Yang, Euiin Yi, Jongwoo Ko, Kimin Lee, Zhijing Jin, Se-Young Yun, KAIST AI, MPI for Intelligent Systems, University of Toronto, Vector Institute Abstract The remarkable growth in large language model (LLM) capabilities has spurred exploration into multi-agent systems, with debate frameworks emerging as a promising avenue for enhanced problem-solving. These multi-agent debate (MAD) approaches, where agents collaboratively present, critique, and refine arguments, potentially offer improved reasoning, robustness, and diverse perspectives over monolithic models. Despite prior studies leveraging MAD, a systematic understanding of its effectiveness compared to self-agent methods, particularly under varying conditions, remains elusive. This paper seeks to fill this gap by conceptualizing MAD as a test-time computational scaling technique, distinguished by collaborative refinement and diverse exploration capabilities. We conduct a comprehensive empirical investigation comparing MAD with strong self-agent test-time scaling baselines on mathematical reasoning and safety-related tasks. Our study systematically examines the influence of task difficulty, model scale, and agent diversity on MAD’s performance. Key findings reveal that, for mathematical reasoning, MAD offers limited advantages over self-agent scaling but becomes more effective with increased problem difficulty and decreased model capability, while agent diversity shows little benefit. Conversely, for safety tasks, MAD’s collaborative refinement can increase vulnerability, but incorporating diverse agent configurations facilitates a gradual reduction in attack success through the collaborative refinement process. We believe our findings provide critical guidance for the future development of more effective and strategically deployed MAD systems. 1 Introduction The capabilities of large language models (LLMs) have grown remarkably, enabling complex task-solving. Recently, beyond scaling individual models, researchers are increasingly exploring multi-agent systems where multiple LLMs collaborate or interact. One promising approach involves structured interactions using debate frameworks, where agents can present, critique, and refine arguments to arrive at a better solution [ 10 ] . These multi-agent debate approaches hold the potential to enhance reasoning and generate more diverse perspectives compared to monolithic models. Several prior studies have leveraged multi-agent debate to enhance LLM performance [ 10 , 20 , 5 ] . These methods use either the same models [ 10 ] , the same models with different roles or personas [ 20 ] , or entirely different models [ 5 ] . Recent approaches have been developed to encourage more diverse thinking [ 23 , 5 , 6 ] , to compete with self-agent methods that utilize more computing like self-refinement [ 25 , 26 ] and self-consistency [ 32 ] . However, a comprehensive understanding of multi-agent debate (MAD) frameworks is currently lacking. The reported effectiveness over using a self-agent varies considerably across studies and task settings, making it difficult to generalize the benefits [ 31 , 27 , 7 ] . Humans typically cooperate when tasks are difficult, individuals lack expertise, or diverse perspectives are needed. In contrast, LLMs are inherently multi-domain experts, which may lead to different conditions under which debate is beneficial. Key questions remain unanswered: under what conditions does MAD truly outperform a highly capable self-agent? In this paper, we aim to bridge the gap in understanding MAD by systematically investigating its vanilla framework. We conceptualize MAD not just as a cooperation strategy, but as a method of test-time computational scaling, similar to techniques such as self-consistency and self-refinement. However, MAD distinguishes itself through two key features as illustrated in Figure 1 : The first is collaborative refinement, where agents share the contextual results of parallel sampling in each round to jointly refine the response. The second is a diverse exploration stage, particularly utilizing different agent configurations. Our objective is to determine if these two additional features prove beneficial for two prominent tasks: mathematical reasoning and safety. To evaluate this, we conduct a comprehensive empirical study comparing MAD with strong self-agent test-time scaling baselines across several critical dimensions. First, we examine task type , including (1) mathematical reasoning tasks, where agents collaborate to solve a single problem, and (2) safety-related reasoning tasks, such as jailbreak detection, where agents must decide how to respond to adversarial queries. Second, we consider task difficulty to assess whether more complex problems necessitate MAD. Third, we explore model scale , analyzing how agent capability affects the utility of MAD. Finally, we investigate agent diversity , evaluating scenarios involving from different model families. Figure 1 : Overview comparing self-agent scaling (sequential: self-refinement [ 25 ] ; parallel: self-consistency [ 32 ] ) with Multi-Agent Debate (MAD) [ 10 ] which mixes parallel generation within rounds and sequential refinement across rounds with shared context. MAD has two distinct features: diverse exploration through different agent configurations, and collaborative refinement, where the full context from previous rounds is shared among all agents. Our extensive experiments yield several key observations regarding the utility and limitations of the MAD framework as the test-time scaling method. On mathematical reasoning tasks, MAD generally shows no clear benefit over parallel scaling methods using a self-agent. However, MAD’s collaborative refinement nature becomes relatively more effective as problem difficulty increases and model capability decreases (Takeaway 4.1 and 4.2 ), where a single, rare correct solution may emerge through parallel scaling, and can be verified and adopted as the final answer through consensus in sequential refinement. Additionally, diversifying exploration spaces does not significantly help in mathematical reasoning (Takeaway 4.3 ), since the strongest model is usually capable of solving problems that weaker models can also solve. For safety tasks, sequential scaling through collaborative refinement leads to a higher attack success rate for MAD compared to zero-shot prompting (Takeaway 5.1 and 5.2 ), although it is less vulnerable than self-refinement. However, unlike in mathematical reasoning, incorporating diverse paths via different agent configurations makes collaborative refinement effective for safety tasks (Takeaway 5.3 ), because models actively incorporate safe responses from others—unlike the tendency to adopt unsafe but helpful responses when using identical model copies. Contributions. In summary, our contributions include: 1. Conceptualizing MAD as test-time scaling: Defining Multi-Agent Debate as a test-time computational scaling method characterized by collaborative refinement and diverse exploration. 2. Systematic empirical comparison: Evaluating MAD against strong self-agent test-time scaling baselines across task types (reasoning, safety), difficulty levels, model scales, and agent diversity. 3. Key insights on MAD’s utility: Identifying specific conditions under which MAD proves beneficial or limited for mathematical reasoning and safety, highlighting the impact of its core features of collaborative refinement and diverse exploration. We believe that the insights presented in this work offer valuable guidance for advancing the design and deployment of MAD systems by analyzing the conditional effectiveness of MAD as a test-time scaling method. 2 Motivation of systematic investigation 2.1 Preliminary: self-agent Standard prompting. Zero-shot prompting queries the language model p with a question q to directly generate an output o , i.e., o=p(q) , without any examples. Chain-of-Thought (CoT; [ 33 ] ) prompting improves performance on complex tasks by generating intermediate reasoning steps r before the final output o , typically using a prompt like q_{\text{CoT}} (e.g., “Think step-by-step”), yielding (r,o)=p(q_{CoT}) . Sequential scaling: self-refinement. Self-refinement (SR; [ 25 , 26 ] ) is an iterative, sequential process where the model p refines its output over multiple rounds. Starting from an initial output o_{0} from zero-shot (i.e., o_{0}=p(q) ) or CoT