UnityMAS-O: A General RL Optimization Framework for LLM-Based Multi-Agent Systems

Renmin University of China Xiaohongshu Inc. Code: https://github.com/chenyiqun/UnityMAS-O See Contributions section for the full author list. · 2026

Read UnityMAS-O: A General RL Optimization Framework for LLM-Based Multi-Agent Systems by Renmin University of China Xiaohongshu Inc. Code: https://github.com/chenyiqun/UnityMAS-O See Contributions section for the full author list. (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26646.txt.

Introduction

Yiqun Chen and sixteen others present UnityMAS-O, a training frame for language-model teams that split jobs across roles. Most teams stay hand-run by prompts and rules, while existing trainers optimize one policy at a time. UnityMAS-O treats the whole workflow as the unit, with roles, graph paths, rewards, and model maps as first-class parts. A star-topology runtime runs workflows and PPO-style updates. On question answering, search, and code tasks, team training lifts hand-set workflows, most for small models.

Catalog record

CREATORRenmin University of China Xiaohongshu Inc. Code: https://github.com/chenyiqun/UnityMAS-O See Contributions section for the full author list.
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66652
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.26646

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

UnityMAS-O: A General RL Optimization Framework for LLM-Based Multi-Agent Systems Renmin University of China Xiaohongshu Inc. Code: https://github.com/chenyiqun/UnityMAS-O See Contributions section for the full author list. Abstract LLM-based multi-agent systems decompose complex tasks into interacting roles, but most existing systems remain manually orchestrated: workflows are specified by prompts, tools, and control rules, while the agents themselves are rarely optimized through a unified reinforcement learning interface. Meanwhile, modern RL post-training frameworks mainly target single-policy optimization and do not directly expose the abstractions needed to optimize arbitrary user-defined multi-agent systems with structured interaction, role-specific credit assignment, and configurable parameter sharing. We present UnityMAS-O , a general RL optimization framework for LLM-based multi-agent systems. UnityMAS-O treats a complete multi-agent workflow as the unit of optimization, rather than a single response or a single policy trajectory. It represents arbitrary task workflows through four first-class objects: logical agent roles, graph-structured trajectories, user-defined reward functions, and explicit agent–model mappings. This design decouples logical agents from physical model parameters, enabling full parameter sharing, full separation, and partial sharing within one interface, while allowing rewards to be assigned at role, turn, and trajectory levels. At the system level, UnityMAS-O extends verl with a Ray-based star-topology runtime. A central controller executes the user-defined workflow, invokes tools or environments, records structured trajectories, and assembles rewards, while model-local worker groups perform rollout, buffering, advantage computation, and distributed PPO-style policy updates. The framework is task-agnostic: users can define the workflow, agents, model mapping, and rewards for different multi-agent systems without rewriting the underlying optimization infrastructure. We instantiate UnityMAS-O on retrieval-augmented QA, iterative agentic search, and reflective code generation as representative workflow families. Across Natural Questions, HotpotQA, and held-out code tasks, multi-agent RL improves the same manually specified workflows before and after optimization, with especially large gains for smaller models and strict code all-passed metrics. These results demonstrate that UnityMAS-O can serve as a reusable optimization substrate for converting diverse LLM-based multi-agent workflows into trainable multi-agent RL systems. 1 Introduction Large language models (LLMs) have made it possible to build multi-agent systems in which role-specialized agents communicate through natural language and tools to solve complex tasks ( 22 ; 1 ; 12 ; 24 ; 17 ) . A typical LLM-based multi-agent system defines a workflow over roles such as planners, retrievers, executors, critics, coders, and answerers. This decomposition is attractive because many real tasks are naturally procedural: information must be searched, filtered, verified, revised, and finally synthesized. Despite this promise, the optimization interface of current multi-agent LLM systems remains limited. Most workflows are manually specified through prompts, routing rules, and hand-designed interaction protocols. When training is introduced, it is often applied to a single model or a single role, leaving the broader interaction pattern largely fixed. This creates a mismatch: the behavior that makes a multi-agent system succeed is not only the quality of an individual response, but also whether upstream agents produce useful intermediate states, whether downstream agents can use them, and whether rewards can be assigned to the roles that influence final outcomes. Recent post-training frameworks have made reinforcement learning for LLMs much more practical by providing scalable rollout, optimization, and distributed execution infrastructure. TRL ( 23 ) , OpenRLHF ( 7 ) , slime ( 14 ) , and verl/HybridFlow ( 20 ) offer important building blocks for large-scale LLM training. However, these systems are primarily organized around single-policy optimization. Multi-turn interaction, tool use, or agentic behavior can be incorporated through customized environments, but the framework abstraction is still usually centered on one trainable policy rather than on a graph of interacting logical agents. This limitation reflects a missing abstraction. A general framework for multi-agent LLM optimization should jointly represent four objects: the user-defined workflow graph, the logical agent roles in that graph, the mapping from logical agents to physical trainable models, and the reward functions defined over structured multi-agent trajectories. It should also provide the systems machinery needed to execute and optimize such workflows at scale. Without this abstraction, each multi-agent training pipeline becomes a task-specific implementation, making it difficult to compare parameter-sharing regimes, assign credit across roles, or reuse training infrastructure across workflows. To address this gap, we present UnityMAS-O , a general-purpose optimization framework for LLM-based multi-agent systems developed by extending verl ( 20 ) . The name reflects the goal of a unified optimization interface for multi-agent systems : “Unity” emphasizes a common abstraction across workflows, agents, model mappings, and rewards, while “O” denotes optimization. UnityMAS-O provides a general optimization interface for user-defined LLM multi-agent workflows. We describe its abstractions, system runtime, reward interface, and demonstrate its effectiveness on QA/search and code-generation workflows. A central controller executes the workflow, records graph-structured trajectories, invokes tools or environments, and assembles role-specific rewards. Model-local worker groups then perform inference, buffering, advantage computation, and distributed optimization for the physical LLM instances assigned to the logical agents. The framework is built around three design requirements. First, UnityMAS-O must optimize over multi-agent trajectories rather than isolated input–output pairs. The relevant learning signal may depend on intermediate evidence, verifier scores, tool outputs, or state updates produced by different agents at different stages. The framework therefore records the structured execution process and exposes rewards at the agent and workflow levels. Second, it must decouple logical agents from physical model instances . The same workflow should support full parameter sharing, full parameter separation, and partial sharing without rewriting the training pipeline. This is essential for studying the trade-offs among specialization, coordination, stability, and computational cost. Third, it must support heterogeneous distributed execution. Multi-agent workflows may invoke different LLMs, tools, and environments within one trajectory. UnityMAS-O therefore separates workflow-level control from model-local optimization, allowing different worker groups to manage different trainable models while the central controller preserves global trajectory and reward consistency. We evaluate UnityMAS-O on two families of verifiable tasks: retrieval-augmented QA and reflective code generation. The QA experiments cover three search workflows on Natural Questions and HotpotQA, while the code experiments study a three-round plan–code–verify–reflect workflow with executable tests. Across these settings, MARL training improves the same workflow before and after optimization. The gains are especially pronounced for smaller QA models and strict code all-passed evaluation, and the trained code workflow also uses fewer verification turns on average. These results show that UnityMAS-O can instantiate different workflow families and improve them through multi-agent RL. Contributions. This report makes the following contributions: • We present UnityMAS-O , a general-purpose optimization framework that extends verl from single-policy post-training to MARL over LLM-based multi-agent workflows. • We introduce a unified abstraction that jointly represents user-defined workflow graphs, logical agent roles, agent-to-model mappings, role-specific rewards, and structured multi-agent trajectories. • We provide native support for full parameter sharing , full parameter separation , and partial parameter sharing , enabling controlled study of specialization, resource cost, and optimization behavior. • We design a Ray + verl -based distributed runtime with central workflow control, asynchronous rollout, reward assembly, model-local buffering, advantage computation, and per-model policy updates. • We instantiate UnityMAS-O on QA/search and code workflows, showing before/after MARL gains, competitive parameter-shared training, and improved execution efficiency in reflective code generation. • We release the open-source implementation at https://github.com/chenyiqun/UnityMAS-O . Overall, UnityMAS-O is intended to serve as a practical systems

Read the complete text