Maestro : Learning to Collaborate via Conditional Listwise Policy Optimization for Multi-Agent LLMs
Wei Yang, Jiacheng Pang, Shixuan Li, Paul Bogdan, Stephen Tu, Jesse Thomason · 2025
Read Maestro : Learning to Collaborate via Conditional Listwise Policy Optimization for Multi-Agent LLMs by Wei Yang, Jiacheng Pang, Shixuan Li, Paul Bogdan, Stephen Tu, Jesse Thomason (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06134.txt.
Introduction
Wei Yang and co-authors orchestrate teams of language models so that wild idea generation and careful synthesis stop interfering. Multi-agent systems can beat lone models on hard problems, but they tend to agree too early, spread errors, and misassign credit between real reasoning and plausible fluff. Their framework, Maestro, splits the two modes structurally.
Parallel execution agents explore widely, while one central agent evaluates and converges on an answer. A new training objective, CLPO, tunes that synthesis step. It separates the signal for strategic choices from the signal for written rationales, ranking justifications against each other for cleaner credit assignment. On math and general problem-solving benchmarks, Maestro with CLPO improved results consistently, the authors report.
Catalog record
| CREATOR | Wei Yang, Jiacheng Pang, Shixuan Li, Paul Bogdan, Stephen Tu, Jesse Thomason |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-06134
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Maestro : Learning to Collaborate via Conditional Listwise Policy Optimization for Multi-Agent LLMs Wei Yang, Jiacheng Pang, Shixuan Li, Paul Bogdan, Stephen Tu, Jesse Thomason Abstract Multi-agent systems (MAS) built on Large Language Models (LLMs) are being used to approach complex problems and can surpass single model inference. However, their success hinges on navigating a fundamental cognitive tension: the need to balance broad, divergent exploration of the solution space with a principled, convergent synthesis to the optimal solution. Existing paradigms often struggle to manage this duality, leading to premature consensus, error propagation, and a critical credit assignment problem that fails to distinguish between genuine reasoning and superficially plausible arguments. To resolve this core challenge, we propose the Multi-Agent Exploration–Synthesis framework Through Role Orchestration ( Maestro ) , a principled paradigm for collaboration that structurally decouples these cognitive modes. Maestro uses a collective of parallel Execution Agents for diverse exploration and a specialized Central Agent for convergent, evaluative synthesis. To operationalize this critical synthesis phase, we introduce Conditional Listwise Policy Optimization (CLPO) , a reinforcement learning objective that disentangles signals for strategic decisions and tactical rationales. By combining decision-focused policy gradients with a list-wise ranking loss over justifications, CLPO achieves clean credit assignment and stronger comparative supervision. Experiments on mathematical reasoning and general problem-solving benchmarks demonstrate that Maestro , coupled with CLPO, consistently outperforms existing state-of-the-art multi-agent approaches, delivering absolute accuracy gains of 6% on average and up to 10% at best. 1 Introduction The rise of large language models (LLMs) have enabled a new type of multi-agent system (MAS) ( Park et al. 2023 ; Chen et al. 2023a ; Zhu et al. 2025 ) , where multiple model instances collaborate to tackle problems that exceed the capacity of any single model ( Zhang et al. 2024a ; Qiao et al. 2024 ; Han et al. 2025 ) . By distributing roles and enabling structured interaction, MASs hold the promise of achieving robustness, creativity, and reliability that emerge from collective intelligence ( Cheng et al. 2024 ; Pezeshkpour et al. 2024 ) . At the heart of any effective collaborative system lies a fundamental cognitive tension. Early work in the psychology of creativity ( Runco & Chand 1995 ; Brophy 2001 ; Zhang et al. 2020 ) emphasizes that intelligent problem-solving requires a dynamic balance between two seemingly contradictory modes of thought: Divergent Creativity and Convergent Critique . Guilford’s theory of divergent and convergent thinking ( Guilford 1967 ) formalizes this duality: divergence is the generative process of exploring a wide array of alternative hypotheses, while convergence is the evaluative process of comparing, refining, and synthesizing these options. Without the former, a system risks premature closure; without the latter, it risks incoherence and indecision ( Sternberg & Lubart 1991 ; Cropley 2006 ) . Achieving a principled and effective synergy between these two capabilities is the essential challenge for effective LLM agent collaboration. Despite their diversity, the limitations of existing paradigms point to a set of recurring requirements for advancing multi-agent collaboration. First, an effective system should strike a balance between divergent exploration and convergent synthesis , ensuring that creativity is not stifled by premature agreement yet also not lost in unbounded search. Second, it should enable disentangled credit assignment across structured outputs ( Li et al. 2025 ; He et al. 2025 ) , so that strategic decisions and supporting rationales receive distinct and targeted learning signals rather than being conflated into a single monolithic reward. Third, a robust framework requires transparent and scalable interaction protocols ( Qian et al. 2024 ; Hu et al. 2024c ) , where information is propagated in analyzable ways that remain efficient as the number of agents and rounds increases ( Yang et al. 2025 ) . Together, these desiderata highlight the limitations of existing approaches and motivate the need for a new paradigm that integrates principled exploration, evaluative precision, and collaborative scalability. To address these desiderata, we propose the Multi-Agent Exploration–Synthesis framework Through Role Orchestration ( Maestro ) , a principled paradigm for multi-agent collaboration (Figure 1 ). The effectiveness of Maestro arises not from any single component, but from the synergistic orchestration of specialized roles. Maestro explicitly operationalizes the divergent–convergent duality through a structured role orchestration: ( i ) Divergence as Collective Exploration , where multiple Execution Agents generate a broad and diverse candidate pool; ( ii ) Convergence as List-wise Bayesian Synthesis , where a Central Agent evaluates these candidates to identify and endorse the most promising solution; and ( iii ) Broadcast as Public Conditioning , where the endorsed solution is propagated back to all agents, guiding the next round of exploration. This cycle of divergence, convergence, and broadcast structures collaboration into analyzable and scalable phases. To further optimize the convergence phase, we introduce Conditional Listwise Policy Optimization (CLPO) , a reinforcement learning objective that disentangles decision-making from rationale generation. Unlike standard GRPO-style sequence-level training, CLPO allocates learning signal separately to decisions (which candidate to endorse) and reasons (why this choice is defensible). Maestro and CLPO constitute a new paradigm for multi-agent collaboration that integrates cognitive inspiration with principled optimization. The main contributions of this paper are: • We introduce the Multi-Agent Exploration–Synthesis framework Through Role Orchestration ( Maestro ), a principled paradigm for multi-agent collaboration that explicitly operationalizes the divergent–convergent duality through three coordinated phases. • We propose Conditional Listwise Policy Optimization (CLPO), an RL objective that decouples signals for decisions and reasons . CLPO combines group-relative decision optimization with listwise rationale ranking for clean credit assignment and stable convergence. • Extensive experiments on mathematical and general reasoning benchmarks show that Maestro with CLPO achieves significant improvements over state-of-the-art baselines. Figure 1: Overview of the Maestro framework. First, N execution agents each generate K candidate reasoning-answer pairs, forming a broad solution pool. A central agent then governs exploitation by applying discriminative selection over the candidate set. The decision policy \pi_{\theta} is trained under Conditional Listwise Policy Optimization (CLPO) , which integrates a choice-aware objective, a reasoning-rank objective, and regularization terms including KL divergence and entropy. 2 Related Work We highlight representative related works in multi-agent LLM collaboration and RL for multi-agent LLMs. For a more in-depth account of related work, see Appendix A . Multi-Agent LLM Collaboration. Large language model (LLM) based multi-agent systems have been proposed to overcome the inherent limits of single models in context length, sequential reasoning, and skill breadth ( Abdelnabi et al. 2023 ; Wu et al. 2024 ; Yan et al. 2025 ; Dai et al. 2025 ) . By coordinating multiple agents, these systems can decompose tasks, critique candidate solutions, and integrate diverse perspectives ( Hong et al. 2023 ; Chen et al. 2023b ; Qiao et al. 2024 ; Pan et al. 2024 ) . One common design follows prestructured coordination , where communication topologies and protocols are fixed in advance ( Chen et al. 2024 ; Mukobi et al. 2023 ; Wang et al. 2023 ; Abdelnabi et al. 2024 ) . Debate and peer-review frameworks ( Du et al. 2023 ; Chan et al. 2023 ; Liu et al. 2024 ) encourage agents to cross-examine one another, while chain or graph structures regulate message flow ( Qian et