R3DM: Enabling Role Discovery and Diversity Through Dynamics Models in Multi-agent Reinforcement Learning
Harsh Goel, Mohammad Omama, Behdad Chalaki, Vaishnav Tadiparthi, Ehsan Moradi Pari, Sandeep Chinchali · 2025
Read R3DM: Enabling Role Discovery and Diversity Through Dynamics Models in Multi-agent Reinforcement Learning by Harsh Goel, Mohammad Omama, Behdad Chalaki, Vaishnav Tadiparthi, Ehsan Moradi Pari, Sandeep Chinchali (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24265.txt.
Introduction
Harsh Goel and five co-authors teach teams of game agents to invent their own roles. Their method appeared in twenty twenty-five. In multi-agent learning, agents that split into roles coordinate better, much like animals in groups, but most methods assign roles from past behavior alone.
The new method, called R3DM, argues a role should also shape what an agent does next. It links each role to past paths and expected future paths, and trains the link by contrast. A learned model of the world then rewards agents for acting out distinct futures by role. On two standard test suites for team games, the method beat leading rivals. Win rates rose by up to twenty percent.
Catalog record
| CREATOR | Harsh Goel, Mohammad Omama, Behdad Chalaki, Vaishnav Tadiparthi, Ehsan Moradi Pari, Sandeep Chinchali |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-24265
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
R3DM: Enabling Role Discovery and Diversity Through Dynamics Models in Multi-agent Reinforcement Learning Harsh Goel, Mohammad Omama, Behdad Chalaki, Vaishnav Tadiparthi, Ehsan Moradi Pari, Sandeep Chinchali Abstract Multi-agent reinforcement learning (MARL) has achieved significant progress in large-scale traffic control, autonomous vehicles, and robotics. Drawing inspiration from biological systems where roles naturally emerge to enable coordination, role-based MARL methods have been proposed to enhance cooperation learning for complex tasks. However, existing methods exclusively derive roles from an agent’s past experience during training, neglecting their influence on its future trajectories. This paper introduces a key insight: an agent’s role should shape its future behavior to enable effective coordination. Hence, we propose Role Discovery and Diversity through Dynamics Models (R3DM), a novel role-based MARL framework that learns emergent roles by maximizing the mutual information between agents’ roles, observed trajectories, and expected future behaviors. R3DM optimizes the proposed objective through contrastive learning on past trajectories to first derive intermediate roles that shape intrinsic rewards to promote diversity in future behaviors across different roles through a learned dynamics model. Benchmarking on SMAC and SMACv2 environments demonstrates that R3DM outperforms state-of-the-art MARL approaches, improving multi-agent coordination to increase win rates by up to 20%. The code is available at https://github.com/UTAustin-SwarmLab/R3DM . Figure 1 : In a fire-fighting scenario with two drones, standard role-based multi-agent RL methods fail to distribute drones effectively, as roles are inferred from exhibited behavior. By linking roles to future expected behavior via a dynamics model, R3DM achieves better role differentiation and coordination. 1 Introduction Multi-agent Reinforcement Learning (MARL) has seen increasing progress in board games ( Meta Fundamental AI Research Diplomacy Team et al.(2022)Meta Fundamental AI Research Diplomacy Team (FAIR), Bakhtin, Brown, Dinan, Farina, Flaherty, Fried, Goff, Gray, Hu, et al. FAIR ; Perolat et al. 2022 ) , traffic signal control ( Chu et al. 2019 ; Chinchali et al. 2018 ; Goel et al. 2023 ; Zhang et al. 2025 ) , autonomous vehicles ( Zhao et al. 2023 ; Han et al. 2023 ) , stock markets ( Bao & Liu 2019 ) , and collaborative robotics ( Krnjaic et al. 2022 ; Zhang et al. 2022 ) . A significant challenge is to learn policies for agents that enable effective coordination. Centralized Training Decentralized Execution (CTDE) is a common paradigm for training collaborative policies. CTDE methods such as QMIX ( Rashid et al. 2020 ) , QPlex ( Wang et al. 2020a ) , MAPPO ( Yu et al. 2022 ) , MAAC ( Iqbal & Sha 2019 ) and VDN ( Sunehag et al. 2018 ) commonly learn shared policy parameters from local observations across multiple agents. These approaches have been effective on benchmarks such as the Starcraft Multi-agent Challenge (SMAC) ( Samvelyan et al. 2019 ; Ellis et al. 2024 ) . However, these methods, which rely primarily on learning shared policy parameters for agents, often hinder the learning of diverse behaviors that individual agents need to exhibit to complete tasks reliably. To address this limitation, recent methods have explored various approaches: (i) encouraging individualized behaviors through diversity or skill-driven intrinsic rewards ( Jiang & Lu 2021a ; Liu et al. 2023 ; Li et al. 2021 ; Liu et al. 2022 ) , and (ii) explicitly training heterogeneous policies ( Zhong et al. 2024 ) . While these methods promote the learning of diverse behaviors, the focus on developing individualized behaviors often compromises effective coordination ( Hu et al. 2024 ) . Furthermore, sacrificing parameter sharing in heterogeneous setups ( Liu et al. 2022 ; Zhong et al. 2024 ) reduces sample efficiency. To balance diversity with efficient cooperation, role-based MARL methods like ACORM ( Hu et al. 2024 ) , CIA ( Liu et al. 2023 ) , RODE ( Wang et al. 2021 ) , GoMARL ( Zang et al. 2023 ) , and ROMA ( Wang et al. 2020b ) learn emergent roles from agents’ past observations. These approaches encourage complementary and distinct behaviors within the parameter-sharing framework of CTDE training. However, deriving roles solely from past observations often fails to enforce effective coordination and cooperative behavior. For example, in a fire-fighting task involving drones extinguishing two separate fires (Fig. 1), if both drones have similar initial observations, they are likely to adopt identical roles early on. This leads to redundant behaviors, such as both drones moving toward the same fire, rather than distributing effectively to address both fires. To address this issue, the central insight of this paper is that an agent’s role should shape its future behavior, meaning an agent adopting different roles at any given moment will naturally follow distinct trajectories. For example, in a multi-drone firefighting scenario, a drone taking up a role to target any specific building would exhibit future observations and actions that naturally diverge as its targets change. This motivates our approach, Role Discovery and Diversity through Dynamics Models (R3DM), which maximizes the mutual information (MI) between agents’ roles at a timestep, their observed trajectories up to that timestep, and expected future trajectories. By linking the current roles of agents to their future expected behaviors or trajectories, R3DM ensures agents learn more distinct yet complementary roles. Our key contributions are summarized as follows: 1. We propose a novel information-theoretic objective between agents’ roles, observed trajectories, and expected future trajectories. This objective is optimized jointly by a contrastive learning framework ( Hu et al. 2024 ) for learning intermediate roles, which shapes intrinsic rewards devised from observation dynamics models to promote role-specific behavior. 2. We integrate this framework with an existing role-based MARL method, ACORM ( Hu et al. 2024 ) . By balancing task rewards with intrinsic rewards, our approach enhances the diversity of role-specific behavior, thereby enabling exploration. 3. We validate R3DM on SMAC ( Samvelyan et al. 2019 ) and SMACv2 ( Ellis et al. 2024 ) environments. Our method achieves superior coordination capabilities compared to existing role-based MARL approaches by showing up to 20% improvement in the final win rates in the challenging SMAC and SMACv2 environments such as 3s5z_vs_3s6z and protoss_10_vs_11. 2 Background 2.1 Preliminaries Multi-agent tasks are modelled as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP) G=\langle I,S,O,A,P,R,\Omega,n,\gamma\rangle , where I=\{0,1,\dots,n-1\} is a finite set of n agents, s^{t}\in S is the global state from the continuous state space S . Each agent i receives observation o_{i}^{t}\in O_{i} where O_{i} is its observation space, via the observation model \Omega(s^{t},i):S\times I\to O_{i} . This results in a joint observation space O\equiv(O_{1}\times O_{2}\times\dots\times O_{n}) . Each agent selects an action a_{i}^{t}\in A_{i} from its discrete action space A_{i} by learning a policy \pi_{i}:T_{i}^{t}\times A_{i}\to[0,1] , where \tau_{i}^{t}\in(A_{i}\times O_{i})^{t}\equiv T_{i}^{t} is the agent’s observation-action history. This yields a joint action \mathbf{a}^{t}=[a_{1}^{t},\ldots,a_{n}^{t}]\in A , where A\equiv(A_{1}\times A_{2}\times\dots\times A_{n}) is the global action space. The joint action policy is given by \boldsymbol{\pi} . The team transitions to a next state s^{t+1} through a transition dynamics model P(s^{t+1}|s^{t},\mathbf{a}^{t}):S\times A\times S\to[0,1] , and receives a reward r^{t}=R(s^{t},\mathbf{a}^{t}) shared by all agents. Here, the discount factor is \gamma\in[0,1] . The joint policy induces the joint state action-value function Q_{\text{tot}}^{\pi}(s,\mathbf{a};\phi) at state s and action \mathbf{a} parameterized by a neural network \phi as Q^{\pi}_{\text{tot}}(s,\mathbf{a};\phi)=\mathbb{E}_{a^{0:\infty},s^{0:\infty}\sim\boldsymbol{\pi}}\left[\sum_{j=0}^{\infty}\gamma^{j}r^{j}|s^{0}=s,\mathbf{a}^{0}=\mathbf{a}\right]. (1) 2.2 Value Function