Multi-Agent Craftax: Benchmarking Open-Ended Multi-Agent Reinforcement Learning at the Hyperscale
Bassel Al Omari, Michael Matthews, Alexander Rutherford, Jakob N. Foerster · 2025
Read Multi-Agent Craftax: Benchmarking Open-Ended Multi-Agent Reinforcement Learning at the Hyperscale by Bassel Al Omari, Michael Matthews, Alexander Rutherford, Jakob N. Foerster (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-04904.txt.
Introduction
Bassel Al Omari leads four authors with Multi-Agent Craftax, a benchmark for open-ended multi-agent learning. Existing tests stress narrow, short-horizon skills in teams of agents. Long-term dependencies and broad generalization rarely get measured in one place.
Craftax-MA extends the Craftax world to many agents at once. Written in JAX, it runs two hundred fifty million interactions in under an hour. A harder companion variant pushes current methods further still. One environment grades a wide spread of general abilities together. Fast, open-ended team benchmarks like this expose whether multi-agent methods truly scale.
Catalog record
| CREATOR | Bassel Al Omari, Michael Matthews, Alexander Rutherford, Jakob N. Foerster |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-04904
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multi-Agent Craftax: Benchmarking Open-Ended Multi-Agent Reinforcement Learning at the Hyperscale Bassel Al Omari, Michael Matthews, Alexander Rutherford, Jakob N. Foerster Abstract Progress in multi-agent reinforcement learning (MARL) requires challenging benchmarks that assess the limits of current methods. However, existing benchmarks often target narrow short-horizon challenges that do not adequately stress the long-term dependencies and generalization capabilities inherent in many multi-agent systems. To address this, we first present Craftax-MA : an extension of the popular open-ended RL environment, Craftax, that supports multiple agents and evaluates a wide range of general abilities within a single environment. Written in JAX, Craftax-MA is exceptionally fast with a training run using 250 million environment interactions completing in under an hour. To provide a more compelling challenge for MARL, we also present Craftax-Coop , an extension introducing heterogeneous agents, trading and more mechanics that require complex cooperation among agents for success 1 1 1 Code is available at https://github.com/BaselOmari/MA-Craftax . . We provide analysis demonstrating that existing algorithms struggle with key challenges in this benchmark, including long-horizon credit assignment, exploration and cooperation, and argue for its potential to drive long-term research in MARL. 1 Introduction Figure 1 : Example pixel-based observation of Craftax-Coop with a summary of player specializations. We also provide a symbolic observation to focus research on multi-agent challenges. Progress in reinforcement learning (RL) goes hand in hand with the development of increasingly sophisticated environments. Such environments serve as benchmarks focusing research towards well-defined problems and enabling direct comparison between methods. In the effort towards increasingly general agents, a community has arisen focused on environments with open-ended dynamics ( 41 ) . These dynamics, such as procedurally generated worlds as seen in environments like Procgen ( 8 ) and NetHack ( 25 ) , help evaluate the systematic generalization of RL methods across diverse scenarios. This field has further advanced with the introduction of hardware-accelerated benchmarks, such as Craftax ( 29 ) , enabling rapid evaluation of scalable and adaptable RL methods using minimal computational resources. Despite these advancements in the single agent setting, the transition of these explorations to the multi-agent setting has remained limited. Existing multi-agent reinforcement learning (MARL) benchmarks often focus on narrow challenges under short time-horizons, such as micromanagement tasks in the popular SMAC benchmark ( 39 ) or cooperation under imperfect information in Hanabi ( 1 ) . While such benchmarks have driven significant algorithmic development, their limited scopes prevent them from capturing the rich dependencies inherent in open-ended multi-agent systems with extended time horizons. To address this gap, we present Craftax-MA , a multi-agent extension of the Craftax benchmark. This extension preserves the rich dynamics of the original Craftax, enabling agents to gather resources, craft advanced tools and combat enemies while learning to navigate complex procedurally generated worlds. By providing a flexible testbed, these dynamics support the study of diverse multi-agent interactions over long time horizons, including competition over resources, coordinated exploration and dynamic role allocation. Built with JAX, Craftax-MA is also fast, enabling rapid experimentation of complex multi-agent behavior even with limited computational resources. Building on Craftax-MA , we present Craftax-Coop , an enhanced multi-agent environment designed specifically to test cooperation among agents. Craftax-Coop introduces agent specialization, where each agent is assigned one of three unique roles (Forager, Miner, or Warrior) each with distinct abilities and responsibilities. Agents must trade essential resources, coordinate their actions and maintain their health through collaborative strategies. Craftax-Coop is thus an ideal testbed for exploring complex cooperation, long horizon planning and emergent teamwork in MARL. Through experimental results, we demonstrate that popular MARL adaptations of algorithms fail to solve Craftax-MA and Craftax-Coop , particularly struggling with long-horizon credit assignment, efficient exploration and cooperation among heterogeneous agents. In some of our settings naive independent learning beats the popular MARL adaptation of PPO, which shows that they are overfit to a small number of popular MARL environments. We believe our environment suite will serve as a much needed challenging and robust benchmark for future MARL research, driving the development of more adaptable and cooperative multi-agent systems. In summary, our contributions are: 1. We introduce Craftax-MA , a multi-agent extension of the popular open-ended RL environment Craftax, which allows for individual or shared reward settings. 2. We introduce Craftax-Coop , a co-operative extension of Craftax-MA featuring specialized agent roles, trading among other mechanics that require continual cooperation among agents. 3. We demonstrate that existing MARL algorithms achieve limited success in Craftax-MA and Craftax-Coop , struggling specifically with credit assignment, exploration and cooperation. 2 Background 2.1 Multi-Agent Reinforcement Learning Multi-Agent Reinforcement Learning (MARL) extends the RL paradigm to facilitate the co-learning of multiple agents simultaneously in a single environment. In particular, we focus on fully co-operative MARL, a paradigm formalized in the decentralized partially observable Markov decision process (Dec-POMDP) ( 33 ) . This is defined as \langle\mathcal{S},\mathcal{A},\mathcal{O},\mathcal{R},\mathcal{T},n\rangle , where \mathcal{S} is the set of states; \mathcal{A} is the set of actions shared between all agents; \mathcal{O} is the observation function \mathcal{O}(s,i) , mapping a state and agent index to a local observation; \mathcal{T}:\mathcal{S}\times\mathcal{A}^{n}\to\Delta\mathcal{S} is the transition function, defining the distribution over next states \mathcal{T}(s,a_{1},...,a_{n}) given a current state s and tuple of actions (a_{1},...,a_{n}) ; \mathcal{R}:\mathcal{S}\to\mathbb{R} is the shared reward function and n is the number of agents. Note that, since the agents all receive the same reward, they are incentivized to behave entirely cooperatively. The Dec-POMDP can be further generalized to a partially observable stochastic game (POSG), where the reward function additionally conditions on the agent index, allowing for the representation of general-sum games. We follow the centralized training decentralized execution (CTDE) paradigm ( 34 ; 23 ) , where it is assumed that data can be shared between agents at training time to facilitate learning (for example through shared critics ( 10 ; 13 ) ), but each agent must behave with only local information at execution time. 2.2 Hardware-Accelerated Reinforcement Learning The recent advent of training pipelines entirely on a hardware accelerator ( 16 ; 28 ) has driven a revolution in RL, allowing for experiments to be run on orders of magnitude more samples than was previously possible. The key to unlocking this scale is the development of simulation environments that can run natively on hardware accelerators ( 11 ; 27 ; 4 ; 22 ; 29 ; 30 ; 35 ; 20 ) , allowing for massive parallelization of workers and the elimination of CPU-GPU data transfer. The vast majority of these environments have been implemented in JAX ( 5 ) . 2.3 Craftax and Crafter Craftax ( 29 ) is a challenging, single-agent, JAX-based environment that takes inspiration from the original Crafter benchmark ( 14 ) and the NetHack Learning Environment ( 26 ) . Beating the game requires navigating nine floors of increasingly difficult enemies, making of use of game mechanics like crafting, building and combat. In particular, Craftax tests the capability of algorithms to explore, generalize to new settings and to perform long-term reasoning over many thousands of timesteps. At the time of writing, the benchmark is currently unsolved, with the best performing agent ( 15 ) averaging only 18% of the maximum reward. 3 Multi-Agent Craftax 3.1 Craftax-MA We first present Craftax-MA , a rewrite of