When Does Multi-Agent RL Improve LLM Workflows? Workflow, Scale, and Policy-Sharing Tradeoffs

Yifan Zeng, Yiran Wu, Yaolun Zhang, Wentian Zhao, Kun Wan, Qingyun Wu, Huazheng Wang · 2026

Read When Does Multi-Agent RL Improve LLM Workflows? Workflow, Scale, and Policy-Sharing Tradeoffs by Yifan Zeng, Yiran Wu, Yaolun Zhang, Wentian Zhao, Kun Wan, Qingyun Wu, Huazheng Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24202.txt.

Introduction

Zeng and six co-authors ask when multi-agent reinforcement learning improves language-model workflows. They compare training with one shared policy for all roles against training where each role keeps its own policy. Across three workflow shapes, maths and code tasks, and three model sizes, multi-agent reinforcement learning usually beats base models, but gains hinge on workflow, task, and scale, not sharing. Isolated policies reach higher peaks yet fall off a terminal accuracy cliff more often, while shared policies merely shift failure into different patterns.

Catalog record

CREATORYifan Zeng, Yiran Wu, Yaolun Zhang, Wentian Zhao, Kun Wan, Qingyun Wu, Huazheng Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53990
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.24202

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When Does Multi-Agent RL Improve LLM Workflows? Workflow, Scale, and Policy-Sharing Tradeoffs Yifan Zeng, Yiran Wu, Yaolun Zhang, Wentian Zhao, Kun Wan, Qingyun Wu, Huazheng Wang Abstract Multi-agent LLM workflows route inference through specialized roles to lift end-task accuracy, but jointly training those roles with reinforcement learning is unstable in ways that are poorly understood. We study when end-to-end RL training of multi-agent LLM workflows improves over their base models, comparing Shared-Policy training, where all roles update one policy, with Isolated-Policy training, where each role has its own parameters. Our experimental matrix spans Eval-Opt, Voting, and Orch-Workers workflows, math and code tasks, and three model scales (0.6B, 1.7B, 4B). We find that multi-agent RL usually improves over base models, but gains depend jointly on workflow, task, and scale, not on policy sharing alone. IP tends to reach higher peak accuracy yet more often falls off a terminal accuracy cliff, while SP training does not eliminate failure; it redistributes failure into qualitatively different patterns. We then explain the strongest of these patterns through role-level gradient dynamics induced by workflow topology and policy routing: under IP , parallel same-role agents on shared prompts amplify per-role gradients and drive terminal degradation in Voting and Orch-Workers workflows; under SP , asymmetric per-step gradient mass causes the shared policy to be captured by the dominant role, producing different failure signatures by task and workflow. Together, the empirical map and its underlying mechanisms show that policy sharing routes training pressure through different channels rather than offering uniform stability, making it a design choice with workflow- and task-conditional tradeoffs. XHMY/marl-llm-workflows 1 Introduction Multi-agent LLM workflows have become a common inference-time way to extract more capability than a single forward pass can deliver, often by decomposing generation, evaluation, aggregation, or delegation across multiple model calls ( 20 ; 17 ; 10 ; 14 ) . In parallel, reinforcement learning with verifiable rewards (RLVR) has become a standard way to improve the reasoning capabilities of large language models (LLMs), with notable success on mathematics and code ( 12 ; 3 ; 21 ; 23 ) . More recently, RLVR has been applied to train multi-step LLM agents, where task success can be evaluated by an outcome signal at the end of an episode ( 5 ; 16 ; 15 ) . This progression naturally extends to multi-agent systems, where the workflow consists of multiple interacting components and the final verifiable outcome can provide a reward signal for optimizing the system end to end. Recent work has begun extending Group Relative Policy Optimization ( 12 ) to multi-agent settings ( 24 ; 7 ; 4 ; 1 ; 2 ) . These efforts show that RL can be applied to specific multi-agent workflows, such as debate, search, or hierarchical tool use, but they do not provide a systematic picture of when multi-agent RL improves performance or why training succeeds or fails. This paper therefore asks a broader question: when does reinforcement learning improve a multi-agent workflow, and what training dynamics explain the outcome ? To answer this question, we run a systematic study across three workflows (Eval-Opt, Voting, and Orch-Workers), three model scales (0.6B, 1.7B, 4B), two tasks (math and code), and two policy-sharing strategies (a shared policy used by every role, and an isolated policy per role). Every cell is compared against a base-model control and a single-agent RL control at matched scale and task, so we can separate the gain attributable to multi-agent training from the gain that single-agent RL alone would already produce. Figure 1 shows the workflow topology and policy routing under study. Figure 1: Workflow topology and policy routing. Three workflows (Eval-Opt, Voting, Orch-Workers), each trained under Isolated-Policy (one \pi_{\text{role}} per role) or Shared-Policy (one \pi_{\text{shared}} for all roles), with a shared outcome-reward GRPO loop. From this design, we gain three empirical insights about when multi-agent RL training improves over the base workflow. (1) multi-agent RL training usually improves over base models, but the choice of policy-sharing strategy trades ceiling against floor: Isolated-Policy routes more often reach higher peak accuracy yet are more prone to late-training degradation, while Shared-Policy routes are conservative on the upside and still subject to late-training drift. (2) whether and how much joint training helps depends jointly on workflow and task, not on the policy-sharing axis alone: the same policy-sharing choice has different consequences across Eval-Opt, Voting, and Orch-Workers workflows, and across math and code. (3) SP training redistributes degradation rather than eliminating it, and some SP failure patterns hide from token-level training metrics and surface only at the episode-correctness layer (§ 3 , § 4 ). We then explain the strongest of these insights through two role-level gradient mechanisms induced by workflow topology and policy routing: gradient_amplification under Isolated-Policy , and sp_role_capture under Shared-Policy (§ 5 ). 2 Related Work Multi-agent RL training of LLM workflows. A growing body of work extends reinforcement learning, especially GRPO ( 12 ) , to multi-agent LLM systems. One line of work develops GRPO-style training objectives for different multi-agent workflow structures. AT-GRPO ( 24 ) introduces agent- and turn-wise grouping with tree-structured sampling, MAGRPO ( 7 ) formulates LLM collaboration as a Dec-POMDP with centralized group-relative advantages and decentralized execution, M-GRPO ( 4 ) addresses hierarchical multi-agent systems through trajectory alignment for variable-length sub-agent invocations, and MHGPO ( 1 ) designs heterogeneous group advantages for multi-agent search pipelines. A second line studies policy adaptation and reinforcement fine-tuning at the system level, including MPDF ( 19 ) , which learns adaptive meta-policies for agent deliberation through rank-based RL, and MARFT ( 6 ) , which provides a conceptual framework for multi-agent reinforcement fine-tuning with LoRA adapters. Another recent direction analyzes optimization stability in multi-agent RL; for example, Dr. MAS ( 2 ) identifies gradient-norm imbalance caused by global advantage normalization across agents and proposes agent-wise normalization. Overall, these methods advance multi-agent RL by introducing algorithmic components tailored to specific workflow patterns or optimization issues. Our work is complementary: rather than proposing a new training algorithm, we conduct a controlled cross-workflow, cross-scale, cross-task empirical study with base-model and single-agent-RL controls, and explain the strongest observed patterns through role-level gradient dynamics induced by workflow topology and policy routing. Diversity collapse and role drift in LLM RL. Several recent threads address related symptoms in single-policy and multi-agent LLM RL. 8 treat diversity collapse under reinforcement learning with verifiable reward as a divergence-choice problem and argue for replacing forward KL with alternatives that better preserve sample diversity. 13 detect and repair role drift in multi-agent collaboration through lightweight protocol-level monitors. 22 address the converse pathology of agent under-contribution in multi-agent reasoning, where one agent dominates and others degenerate into passive roles. Our work is structural: we identify which workflow topologies and policy-routing choices produce role drift in the first place, and explain the strongest observed patterns through gradient mechanisms ( gradient_amplification under Isolated-Policy , sp_role_capture under Shared-Policy ). 3 Experimental Setup Our experiments are organized as a controlled grid over four dimensions: task, model scale, policy-routing strategy, and workflow (Figure 1 ). The task dimension covers mathematical reasoning and code-generation settings; the model dimension varies the size of the underlying base model; the policy-routing dimension compares shared and isolated

Read the complete text