Modular Reinforcement Learning For Cooperative Swarms Conference: Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026); May 25 – 29, 2026; Paphos, Cyprus 0

Erel Shtossel, Gal A. Kaminka · 2026

Read Modular Reinforcement Learning For Cooperative Swarms Conference: Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026); May 25 – 29, 2026; Paphos, Cyprus 0 by Erel Shtossel, Gal A. Kaminka (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-04939.txt.

Introduction

Erel Shtossel and Gal Kaminka propose a modular way for cooperative robot swarms to learn how to work together. Each small robot can only interact with a few neighbours and cannot see how its choices affect the whole group's goal. Earlier learning methods forced each robot to track a huge number of combined interaction states, straining tiny memories.

Their alternative splits the spatial state into features, with a separate learning procedure handling each feature and the results combined afterwards. Simulated swarms carrying out foraging tasks show the approach works well across many experiments. It offers swarm designers a memory-light route to coordinated behaviour in simple robots.

Catalog record

CREATORErel Shtossel, Gal A. Kaminka
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.04939

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Modular Reinforcement Learning For Cooperative Swarms Conference: Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026); May 25 – 29, 2026; Paphos, Cyprus 0 Erel Shtossel, Gal A. Kaminka Abstract . A cooperative robot swarm is a collective of computationally-limited robots that share a common goal. Each robot can only interact with a small subset of its peers, without knowing how this affects the collective utility. Recent advances in distributed multi-agent reinforcement learning have demonstrated that it is possible for robots to learn how to interact effectively with others, in a manner that is aligned with the common goal, despite each robot learning independently of others. However, this requires each robot to represent a potentially combinatorial number of interaction states, challenging the memory capabilities of the robots. This paper proposes an alternative approach for representing spatial interaction states for multi-robot reinforcement learning in swarms. A modular (decomposed) representation is used, where each feature of the state is handled by a separate learning procedure, and the results aggregated. We demonstrate the efficacy of the approach in numerous experiments with simulated robot swarms carrying out foraging . C. Amato, L. Dennis, V. Mascardi, J. Thangarajah (eds.) 1. Introduction A cooperative robot swarm is a collective of robots that work towards a common goal, despite each being limited in the range of its perception and the reach of its actions. These limitations allow only local interactions for each robot, with a small subset of its peers ( Correll and Martinoli 2009 ; Brambilla et al. 2013 ; Bayındır 2016 ; Hamann 2018 ; Dorigo et al. 2021 ) . Cooperative swarms pose a significant challenge to swarm robot developers. The maximization of the collective utility by the swarm’s members, in aggregate, is encumbered by each individual robot’s lack of knowledge about its peers or the state of the collective as a whole. Often, this is addressed by carefully designing the individual decision-making procedures, such that local decisions are forced to align with the collective goal ( Rybski et al. 1998 ; Zuluaga and Vaughan 2005 ; Song and Vaughan 2013 ; Ren et al. 2023 ) . Alternatively, multi-agent reinforcement learning ( MARL ) can be used to learn individual action-selection policies ( Kapetanakis and Kudenko 2002 ; Hernandez-Leal et al. 2019 ; Kober et al. 2013 ; Zhang et al. 2021 ; Kuckling 2023 ) , in particular when matched with appropriate aligned rewards ( Devlin et al. 2014 ; Kaminka 2025 ) . A significant challenge to the use of MARL in swarm robots is raised by the limited computational resources of the robots. Limited memory prohibits explicit representation of the large combinatorial number of states (a state explosion problem). Limited computation capabilities prohibit the use of deep neural networks as function approximations to generalize and represent states implicitly. Finally, the restrictions on communication range and bandwidth prohibit sharing of information at the swarm level. Thus robots act, for the most part, as independent learners , unaware of the decisions of others, nor the individual impact (reward) of these decisions. This is a particularly challenging setting for MARL ( Kapetanakis and Kudenko 2002 ; Matignon et al. 2012 ; Nowé et al. 2012 ) . While these challenges are known in general ( Lesort et al. 2018 ; Mohan et al. 2023 ; Sahni et al. 2017 ; Wong et al. 2021 ) , the limited computational resources of swarm robots make most solution approaches impractical or even irrelevant. For perspective, research swarm robots are commonly built around microcontrollers running at a few hundred MHz cycles, using total RAM well under 512KB (the capabilities of popular off-the-shelf robots are presented in Table 1 ). Robot Controller Clock Speed RAM e-puck2 ( Generation Robots 2025 ) STM32F4 168 MHz 192 KB Elisa-3 ( GCTronic 2025 ) ATmega2560 8 MHz 8 KB Pololu3Pi ( Pololu Robotics and Electronics 2025 ) ATmega328 20 MHz 2 KB Krembot ( Particle.io 2025 ) Particle Photon (ARM Cortex M3) 120 MHz 128 KB Arduino Robot ( Arduino 2025 ) Two ATmega32u4 16 MHz 2.5 KB each Kilobots ( Rubenstein et al. 2012 ) ATmega328 8 MHz 32 KB Table 1. Specifications of popular off-the-shelf robots. Multi-robot swarm foraging, a canonical cooperative swarm task, is a good example of how the severe computational limitations of swarm robots pose a challenge to reinforcement learning approaches. In foraging, multiple robots search a shared arena for items that they pick up and bring to their base. The performance of the swarm is measured by the sum total of collected items. Robots have limited sensor and communication ranges, and cannot communicate with all other robots; in some cases, they cannot communicate at all. To navigate, each robot typically represents its spatial interaction state, composed of the range and bearing to all robots and obstacles in its sensed surroundings. As long as only the current state is maintained, this is not a challenge. However, if a naive reinforcement learning mechanism needs to learn how to act in each state, then all possible spatial states must be represented. The number of states grows combinatorially large with the number of entities and the resolution of measurements. A latent representation using deep neural networks is not feasible given the limited computation available (some controllers do not support floating-point operations in hardware). To address the state explosion problem, this paper presents a modular (decomposed) state representation, where each state feature (e.g., the position of each neighbour) is represented separately and handled by a separate learning process. In this way, rather than a single learner addressing a k -factor state representation ( \mathcal{O}(2^{k}) states to be stored), there are k simple learners, requiring storing a total of \mathcal{O}(k) states. The learners’ recommendations are aggregated via a fixed action-fusion mechanism. We describe the modular representation in detail and evaluate its use in a comprehensive set of experiments with simulated robots in a cooperative foraging task. The experiments show that the modular representation works on par with a full state learning algorithm, despite its modest memory and computational requirements. 2. Background Research on robot swarms focuses on the collective behaviour of decentralized, self-organized systems composed of numerous computationally-limited robots. The research area is very broad, covering many different tasks and approaches. We refer the reader to comprehensive surveys of the area ( Correll and Martinoli 2009 ; Brambilla et al. 2013 ; Bayındır 2016 ; Hamann 2018 ; Dorigo et al. 2021 ) . We focus on cooperative swarm foraging , an operation where a group of swarm robots is tasked with continually searching for objects of interest ( items ), and when found, transporting them to collection points ( bases ). This is a canonical swarm task with many studied variants: see ( Shell and Mataric 2006 ; Winfield 2009 ; Zedadra et al. 2017 ; Lu et al. 2020 ; Ren et al. 2023 ) for comprehensive surveys. This paper focuses on using multi-agent reinforcement learning (MARL) ( Kapetanakis and Kudenko 2002 ; Yang and Gu 2004 ; Busoniu et al. 2008 ; Kober et al. 2013 ; Hernandez-Leal et al. 2019 ; Zhang et al. 2021 ; Kuckling 2023 ; Fatima et

Read the complete text