Revisiting Multi-Agent World Modeling from a Diffusion-Inspired Perspective
Yang Zhang, Xinran Li, Jianing Ye, Shuang Qiu, Delin Qu, Xiu Li, Chongjie Zhang, Chenjia Bai · 2025
Read Revisiting Multi-Agent World Modeling from a Diffusion-Inspired Perspective by Yang Zhang, Xinran Li, Jianing Ye, Shuang Qiu, Delin Qu, Xiu Li, Chongjie Zhang, Chenjia Bai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20922.txt.
Introduction
Yang Zhang, Xinran Li, and their co-authors rethink how groups of learning agents model their shared world. Their 2025 paper works in multi agent reinforcement learning, where world models can cut the data needed to learn good policies. Modeling a joint world is hard: the shared action space explodes, and outcomes stay uncertain.
Their fix models the state alone at each step, revealing agents one by one in sequence. Each reveal trims uncertainty and captures how agents depend on each other. The team noticed this stepwise reveal mirrors the reverse process of diffusion models, famed for expressive power and stable training. Borrowing that lens yields a world model design for agent teams. The paper builds the case for the approach and its structure.
Catalog record
| CREATOR | Yang Zhang, Xinran Li, Jianing Ye, Shuang Qiu, Delin Qu, Xiu Li, Chongjie Zhang, Chenjia Bai |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 88 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-20922
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Revisiting Multi-Agent World Modeling from a Diffusion-Inspired Perspective Yang Zhang, Xinran Li, Jianing Ye, Shuang Qiu, Delin Qu, Xiu Li, Chongjie Zhang, Chenjia Bai Abstract World models have recently attracted growing interest in Multi-Agent Reinforcement Learning (MARL) due to their ability to improve sample efficiency for policy learning. However, accurately modeling environments in MARL is challenging due to the exponentially large joint action space and highly uncertain dynamics inherent in multi-agent systems. To address this, we reduce modeling complexity by shifting from jointly modeling the entire state-action transition dynamics to focusing on the state space alone at each timestep through sequential agent modeling. Specifically, our approach enables the model to progressively resolve uncertainty while capturing the structured dependencies among agents, providing a more accurate representation of how agents influence the state. Interestingly, this sequential revelation of agents’ actions in a multi-agent system aligns with the reverse process in diffusion models—a class of powerful generative models known for their expressiveness and training stability compared to autoregressive or latent variable models. Leveraging this insight, we develop a flexible and robust world model for MARL using diffusion models. Our method, D iffusion- I nspired M ulti- A gent world model (DIMA), achieves state-of-the-art performance across multiple multi-agent control benchmarks, significantly outperforming prior world models in terms of final return and sample efficiency, including MAMuJoCo and Bi-DexHands. DIMA establishes a new paradigm for constructing multi-agent world models, advancing the frontier of MARL research. Codes are open-sourced at https://github.com/breez3young/DIMA . 1 Introduction Learning accurate world models to capture environmental dynamics is crucial for effective decision-making. In the realm of model-based reinforcement learning (MBRL), such models play a pivotal role by enabling policy training through learning in imagination [ 1 , 2 , 3 , 4 ] , facilitating planning with look-ahead search [ 5 , 6 ] , or combining both approaches [ 7 , 8 ] . While MBRL has achieved significant success in single-agent settings, extending these methodologies to multi-agent scenarios presents unique challenges, necessitating new approaches for multi-agent world modeling. In multi-agent settings, where multiple agents simultaneously interact within a shared environment, two primary challenges emerge. First, the joint action space grows exponentially with the number of agents [ 9 , 10 ] , making it computationally expensive to directly handle joint dynamics. Second, the complex interdependencies among agents [ 11 ] make it difficult to accurately capture how individual actions impact global state transitions. Current multi-agent world modeling approaches face a fundamental tradeoff. On one end of the spectrum, centralized modeling schemes directly capture full joint dynamics but incur computational costs that scale exponentially with the number of agents. On the other end, decentralized approaches [ 12 , 13 , 14 ] model individual agent dynamics separately and rely on additional mechanisms, such as sophisticated communication or aggregation modules, to recover the global state. However, this misalignment between decentralized model structure and the global Markov decision process (MDP) can impose inherent limitations on model accuracy and those communication or aggregation modules do not have explicit signal for supervision, further hindering the training. This tradeoff motivates a fundamental rethinking of the world model structure: Can we develop a centralized modeling scheme that maintains global consistency without auxiliary components in decentralized methods, while keeping computational complexity manageable as the number of agents increases? Figure 1: Illustration of the DIMA world model. From the temporal perspective, each environmental timestep is modeled as a complete denoising process, analogous to diffusion models. Within each timestep, we further consider an agent-wise perspective, where the introduction of each individual agent’s action information represents a single denoising step, progressively reducing uncertainty about the next state. To address this challenge, we adopt a sequential agent modeling perspective that processes agents’ actions incrementally, as illustrated in Figure 1 . Specifically, consider a multi-agent system at timestep t with global state s_{t} . When all agents’ actions a_{t}^{1:n} are unknown, the next state s_{t+1} remains highly uncertain. As agents’ actions are progressively revealed, this uncertainty gradually decreases. This sequential uncertainty reduction process bears striking similarity to the reverse process in diffusion models [ 15 , 16 , 17 ] , where generation is framed as iterative denoising from noise to clean samples. Inspired by this conceptual similarity and the recent success of diffusion models in image-based world modeling [ 18 , 19 , 20 ] , we propose D iffusion- I nspired M ulti- A gent world model (DIMA), which reformulates multi-agent dynamics prediction as a modified conditional denoising process. Despite employing a centralized modeling scheme, DIMA achieves computational complexity that scales linearly with the state space dimensionality, regardless of the number of agents. We summarize our contributions as follows: • We leverage the connection between sequential agent modeling and diffusion processes to reformulate multi-agent dynamics prediction as a conditional denoising process. This enables a centralized modeling scheme that reduces complexity without additional communication mechanisms. • We propose DIMA, a centralized multi-agent world model tailored for model-based MARL, and derive its corresponding evidence lower bound (ELBO), providing theoretical insights. We then instantiate DIMA within the EDM training framework [ 21 ] and integrate it into the learning-in-imagination paradigm for policy optimization. • We evaluate DIMA on challenging continuous MARL benchmarks, including MAMuJoCo [ 22 ] and Bi-DexHands [ 23 ] , in low-data regimes. Experimental results show that DIMA consistently improves the prediction accuracy of environment dynamics and outperforms both model-free and strong model-based MARL baselines in terms of sample efficiency and overall performance. 2 Preliminaries 2.1 Multi-Agent Systems as Dec-POMDP We focus on fully cooperative multi-agent systems where all agents share a team reward signal. We formulate the system as a decentralized partially observable Markov decision process (Dec-POMDP) [ 11 ] , which can be described by a tuple (\mathcal{N},\mathcal{S},\boldsymbol{\mathcal{A}},P,R,\boldsymbol{\Omega},\boldsymbol{\mathcal{O}},\gamma) . \mathcal{N}=\{1,...,n\} denotes a set of agents, \mathcal{S} is the finite global state space, \boldsymbol{\mathcal{A}}=\prod_{i=1}^{n}\mathcal{A}^{i} is the product of finite action spaces of all agents, i.e., the joint action space, P:\mathcal{S}\times\boldsymbol{\mathcal{A}}\times\mathcal{S}\rightarrow[0,1] is the global transition probability function, R:\mathcal{S}\times\boldsymbol{\mathcal{A}}\rightarrow\mathbb{R} is the shared reward function, \boldsymbol{\Omega}=\prod_{i=1}^{n}\Omega^{i} is the product of finite observation spaces of all agents, i.e., the joint observation space, \boldsymbol{\mathcal{O}}=\{\mathcal{O}^{i},i\in\mathcal{N}\} is the set of observing functions of all agents. \mathcal{O}^{i}:\mathcal{S}\rightarrow\Omega^{i} maps global states to the observations for agent i , and \gamma is the discount factor. Given a global state s_{t} at timestep t , agent i is restricted to obtaining solely its local observation o_{t}^{i}=\mathcal{O}^{i}(s_{t}) , takes an action a_{t}^{i} drawn from its policy \pi^{i}(\cdot|o_{\leq t}^{i}) based on the history of its local observations o_{\leq t}^{i} , which together with other agents’ actions gives a joint action \boldsymbol{a}_{t}=(a_{t}^{1},...,a_{t}^{n})\in\boldsymbol{\mathcal{A}} , equivalently drawn from a joint policy \boldsymbol{\pi}(\cdot|\boldsymbol{o}_{\leq t})=\prod_{i=1}^{n}\pi^{i}(\cdot|o_{\leq t}^{i}) . Then the agents receive a shared reward r_{t}=R(s_{t},\boldsymbol{a}_{t}) , and the environment moves to next state s_{t+1} with probability P(s_{t+1}|s_{t},\boldsymbol{a}_{t}) . The aim of all agents is to learn a joint policy \boldsymbol{\pi} that maximizes the expected discounted return J(\boldsymbol{\pi})=\mathbb{E}_{s_{0},\boldsymbol{a}_{0},...\sim\boldsymbol{\pi}}\left[\sum_{t=0}^{\infty}\gamma^{t}R(s_{t},\boldsymbol{a}_{t})\right] . Note that recent approaches [ 12 , 13 , 14 ] build the multi-agent world models via modeling P(\boldsymbol{o}_{t+1}|\boldsymbol{o}_{t},\boldsymbol{a}_{t}) in the joint observation and action space, which mismatches with the transition formulation in Dec-POMDP. However, DIMA is trained to recover the well-defined global state transition P(s_{t+1}|s_{t},\boldsymbol{a}_{t}) according to the proposed