Mixture of Debaters: Learn to Debate at Architectural Level in Multi-Agent Reasoning ISBN: 978-1-4503-XXXX-X/2026/06 CCS: Computing methodologies Natural language processing CCS: Computing methodologies Neural networks CCS: Computing methodologies Computer vision CCS: Computing methodologies Knowledge representation and reasoning

Dayong Liang, Kaisong Gong, Yi Cai, Changmeng Zheng, Xiao-Yong Wei · 2026

Read Mixture of Debaters: Learn to Debate at Architectural Level in Multi-Agent Reasoning ISBN: 978-1-4503-XXXX-X/2026/06 CCS: Computing methodologies Natural language processing CCS: Computing methodologies Neural networks CCS: Computing methodologies Computer vision CCS: Computing methodologies Knowledge representation and reasoning by Dayong Liang, Kaisong Gong, Yi Cai, Changmeng Zheng, Xiao-Yong Wei (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29425.txt.

Introduction

Liang and four colleagues build debate directly into a single model's architecture, instead of running several model copies that argue with each other. Multi-agent debate can improve reasoning, but fixed roles and repeated copies carry heavy computing costs. Their framework, Mixture of Debaters, adapts the Mixture-of-Experts paradigm, where different expert modules inside one network handle different work.

Three ideas make self-debate work. Dual routing decides both which persona argues and when the model should debate or synthesize an answer. Momentum switching smooths choices between tokens, the text pieces a model processes, to cut jitter. Lightweight expert modules hold the debating personas, so no messages pass between separate agents. On multimodal benchmarks, tests that mix images and text, the system beats single models and conventional multi-agent setups. It also runs much faster, with latency down sharply and token use cut by most of its former level.

Catalog record

CREATORDayong Liang, Kaisong Gong, Yi Cai, Changmeng Zheng, Xiao-Yong Wei
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.29425

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Mixture of Debaters: Learn to Debate at Architectural Level in Multi-Agent Reasoning ISBN: 978-1-4503-XXXX-X/2026/06 CCS: Computing methodologies Natural language processing CCS: Computing methodologies Neural networks CCS: Computing methodologies Computer vision CCS: Computing methodologies Knowledge representation and reasoning Dayong Liang, Kaisong Gong, Yi Cai, Changmeng Zheng, Xiao-Yong Wei Abstract . Existing multi-agent debate frameworks suffer from two critical limitations: they rely on static architectures where agent roles and coordination patterns are fixed at design time, and they require instantiating multiple model copies, incurring substantial computational overhead. We propose Mixture of Debaters (MoD) , a unified framework that enables dynamic self-debate within a single model by leveraging the Mixture-of-Experts paradigm. We address three key challenges in adapting MoE for dialectical reasoning: (1) dual-routing that decouples role allocation from process flow, dynamically determining when to debate versus when to synthesize; (2) momentum switching that smooths token-level routing with local context, reducing expert-switch jitter; and (3) unified self-debate that encapsulates diverse debating personas into lightweight expert modules, eliminating inter-agent communication while preserving behavioral diversity. Extensive experiments on multimodal benchmarks demonstrate that MoD outperforms both single-model baselines and conventional multi-agent systems, achieving superior accuracy with 3.7 × \times lower latency and 87% reduction in token consumption. The source code can be accessed at https://github.com/YongLD/MoD . 1. Introduction Large Language Models (LLMs) have demonstrated remarkable capabilities in complex reasoning tasks, ranging from mathematical problem solving to logical deduction and code generation ( 8 ; 35 ; 45 ; 34 ) . Despite the success of prompting strategies like Chain-of-Thought (CoT), reasoning traces generated by a single monolithic model remain prone to degeneration and hallucination, particularly when navigating intricate logical landscapes ( 51 ) . To address these limitations, recent research has pivoted toward multi-agent debate frameworks, where diverse agents improve reasoning fidelity through adversarial critique and cross-examination ( 27 ; 21 ; 52 ) . Figure 1. Internal vs. external debate architectures. (A) Conventional multi-agent debate requires instantiating independent model copies ( A_{1} – A_{4} ), incurring substantial runtime complexity. (B) Our Mixture-of-Debaters internalizes diverse agents into a shared expert pool with dynamic routing, improving framework flexibility while eliminating inter-agent communication overhead. Internal vs. external debate architectures. (A) Conventional multi-agent debate requires instantiating independent model copies ($A_1$–$A_4$), incurring substantial runtime complexity. (B) Our Mixture-of-Debaters internalizes diverse agents into a shared expert pool with dynamic routing, improving framework flexibility while eliminating inter-agent communication overhead. However, we argue that current debate paradigms are structurally constrained. Existing systems operate with static architectures where agent roles and communication patterns are fixed at design time. For instance, a standard setup might rigidly enforce a cycle of “propose \rightarrow critique \rightarrow refine.” Real-world multimodal problems, however, are fluid and multifaceted: an advanced physics problem might initially appear as a simple kinematics question, evolve into a complex thermodynamics puzzle upon closer inspection of a diagram, and ultimately require mathematical synthesis. In such dynamic contexts, a fixed topology fails to adapt to the shifting epistemic needs of the problem. To bridge the gap between static architectures and fluid reasoning requirements, the Mixture-of-Experts (MoE) framework offers a promising structural foundation ( 55 ; 31 ; 23 ) . By dynamically activating different subsets of parameters (experts) based on the input, MoE has the potential to realize robust and generalized reasoning. Nevertheless, directly applying generic MoE architectures to the dialectical process of debate is non-trivial and exposes distinct architectural gaps. In this work, we propose Mixture-of-Debaters ( MoD ), a pilot framework that unifies dynamic self-debate within a single architecture. We articulate our contributions by identifying three critical challenges in adapting MoE for debate and presenting our corresponding architectural advantages: \bullet Dual-Routing for Dialectical Flow Control. Standard MoE architectures typically rely on a single router to dispatch tokens to experts. This design is insufficient for debate, which requires orchestration across different dimensions: assigning specific roles (e.g., proposer vs. critic) and managing the process flow (e.g., debating vs. concluding). A single router cannot simultaneously optimize for these orthogonal objectives. We propose a dual-routing mechanism that decouples role allocation from stage management. By introducing specialized gating functions, our model dynamically regulates when to debate (invoking dialectical confrontation) and when to summarize (invoking synthesis). This allows the system to switch strategies adaptively, skipping unnecessary argumentation for simple sub-problems or invoking early synthesis to resolve conflicts, thereby optimizing the reasoning trajectory. \bullet Momentum Switching for Consistency. Conventional MoE gating operates at the token level and is sensitive to local noise and minor representation fluctuations. In a debate context, such jittery routing may swap experts too frequently (e.g., flipping between a “pro” and a “critic” stance within a short span), fragmenting the reasoning trace and hurting coherence. We introduce Momentum Switching , which smooths routing decisions via local contextual aggregation so expert usage evolves more gradually over time, improving short-range persistency while remaining responsive to genuine context shifts. This stabilization is lightweight, autoregressive-friendly, and complements our dual-routing design by reducing unnecessary role flips during multi-turn reasoning. \bullet Unified Efficient Self-Debate. Prior multi-agent debate frameworks rely on instantiating multiple distinct models or maintaining several heavy copies of LLMs to simulate interaction. This inter-model communication is computationally expensive, memory-intensive, and introduces significant latency due to network overhead or context switching. We propose a unified self-debate mode based on the MoE architecture. By encapsulating diverse debating personas into lightweight, specialized expert modules within a single backbone, we eliminate the need for external multi-model orchestration. This design reduces inter-agent communication cost while retaining the behavioral diversity required for effective self-debate, allowing for efficient, high-quality reasoning within a compact computational budget. Figure 2. Overview of Mixture-of-Debaters (MoD). Left: Viewpoint-Shift Data Synthesis constructs debate trajectories from correct and incorrect samples for belief revision. Right: MoD architecture with dual-routing, momentum switching, and decoupled A-side/B-side expert pools for diverse reasoning pathways. Overview of Mixture-of-Debaters (MoD). Left: Viewpoint-Shift Data Synthesis constructs debate trajectories from correct and incorrect samples for belief revision. Right: MoD architecture with dual-routing, momentum switching, and decoupled A-side/B-side expert pools for diverse reasoning pathways. 2. Related Work Multi-Agent Reasoning and Debate. Multi-agent debate has emerged as an effective paradigm for improving reasoning by exposing latent inconsistencies through adversarial critique and cross-examination ( 49 ; 27 ; 2 ) . Recent studies extend this idea to multimodal settings, showing that interaction among multiple vision-language agents can improve reasoning reliability and reduce hallucination ( 46 ; 24 ; 25 ; 16 ; 53 ) . This line is further generalized by Mixture-of-Agents (MoA), which aggregates responses from multiple external models through layered coordination ( 43 ) . Despite their effectiveness, these approaches typically require multiple model instances or external APIs as independent agents, leading to substantial memory, communication, and latency overhead. Moreover, their performance often depends on the zero-shot debate capability of the underlying models, which may limit gains when base models are weak or prone to shallow contradiction and paraphrastic disagreement ( 36 ; 6 ) . Internalized Deliberation and Self-Correction. To reduce the cost of explicit multi-agent orchestration, a parallel line of work seeks to internalize deliberative behavior within a single model. Early efforts such as Debate, Reflect, and Distill (DRD) distill multi-agent debate trajectories into a student model ( 54 ) , while SMoA emulates MoA-style

Read the complete text