HyperMARL: Adaptive Hypernetworks for Multi-Agent RL

Kale-ab Abebe Tessera, Arrasy Rahman, Amos Storkey, Stefano V. Albrecht · 2024

Read HyperMARL: Adaptive Hypernetworks for Multi-Agent RL by Kale-ab Abebe Tessera, Arrasy Rahman, Amos Storkey, Stefano V. Albrecht (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-04233.txt.

Introduction

Kale-ab Tessera and three co-authors introduce in 2024 HyperMARL, a method that lets teams of learning agents share one policy yet still specialize. Sharing parameters speeds multi-agent learning but tends to flatten the variety teams need. The team traces much of the trouble to gradients that clash when agent identity is glued to observations.

A hypernetwork that reads agent identity generates custom weights for each agent, splitting the tangled gradients apart. Across 22 test scenarios with up to 30 agents, the method matched six strong baselines. Behavioral variety stayed close to that of agents trained fully apart. Adaptive cooperation, it turns out, need not require preset diversity tricks or staged updates.

Catalog record

CREATORKale-ab Abebe Tessera, Arrasy Rahman, Amos Storkey, Stefano V. Albrecht
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE101 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-04233

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HyperMARL: Adaptive Hypernetworks for Multi-Agent RL Kale-ab Abebe Tessera, Arrasy Rahman, Amos Storkey, Stefano V. Albrecht Abstract Adaptive cooperation in multi-agent reinforcement learning (MARL) requires policies to express homogeneous, specialised, or mixed behaviours, yet achieving this adaptivity remains a critical challenge. While parameter sharing (PS) is standard for efficient learning, it notoriously suppresses the behavioural diversity required for specialisation. This failure is largely due to cross-agent gradient interference, a problem we find is surprisingly exacerbated by the common practice of coupling agent IDs with observations . Existing remedies typically add complexity through altered objectives, manual preset diversity levels, or sequential updates – raising a fundamental question: can shared policies adapt without these intricacies? We propose a solution built on a key insight: an agent-conditioned hypernetwork can generate agent-specific parameters and decouple observation- and agent-conditioned gradients, directly countering the interference from coupling agent IDs with observations. Our resulting method, HyperMARL , avoids the complexities of prior work and empirically reduces policy gradient variance. Across diverse MARL benchmarks (22 scenarios, up to 30 agents), HyperMARL achieves performance competitive with six key baselines while preserving behavioural diversity comparable to non-parameter sharing methods, establishing it as a versatile and principled approach for adaptive MARL. The code is publicly available at https://github.com/KaleabTessera/HyperMARL . 1 Introduction Specialist and generalist behaviours are critical to collective intelligence, enhancing performance and adaptability in both natural and artificial systems [ 52 , 43 , 44 , 23 , 50 ] . In Multi-Agent Reinforcement Learning (MARL) [ 2 ] , this translates to a critical need for policies that can flexibly exhibit specialised, homogeneous, or mixed behaviours to meet diverse task demands [ 26 , 6 ] . Figure 1 : HyperMARL Policy Architecture. Common agent-ID conditioned shared MARL policy (FuPS+ID, left) vs HyperMARL (right), which uses an agent-conditioned hypernetwork to generate agent-specific weights and decouples observation- and agent-conditioned gradients. Optimal MARL performance thus hinges on being able to represent the required behaviours. While No Parameter Sharing (NoPS) [ 30 ] enables specialisation by using distinct per-agent networks, it suffers from significant computational overhead and sample inefficiency [ 11 ] . Conversely, Full Parameter Sharing (FuPS) [ 45 , 18 , 15 ] , which trains a single shared network, improves efficiency but typically struggles to foster the behavioural diversity necessary for many complex tasks [ 24 , 17 , 26 ] . This failure of FuPS, particularly for diverse behaviours, was hypothesised to be gradient interference among agents, whereby their updates negatively impact each other’s learning [ 11 , 54 ] . We not only empirically validate this hypothesis but also demonstrate a critical insight: this conflict is significantly exacerbated by the common practice of coupling observations with agent IDs within a shared network (Fig. 1 for coupling, Sec. 3.2 for results). Balancing FuPS efficiency with the capacity for diverse behaviours therefore remains a central open problem in MARL. Prior works have explored intrinsic-rewards [ 26 , 22 ] , role-based allocations [ 48 , 49 ] , specialised architectures [ 24 , 28 , 6 ] , sequential updates [ 54 ] , and sharing parameters within clusters of agents [ 11 ] . However, these remedies introduce their own intricacies: they often alter the learning objective, require prior knowledge of optimal diversity levels, necessitate maintaining agent-specific parameters or require sequential updates. This raises a fundamental question: Can we design a shared MARL architecture that flexibly supports both specialised and homogeneous behaviours—without altered learning objectives, manual preset diversity levels, or sequential updates? Guided by our observation-ID coupling insight, we propose HyperMARL , a novel agent-conditioned hypernetwork [ 19 ] architecture. HyperMARL generates per-agent weights on the fly (Fig. 1 ) and explicitly decouples observation- and agent-conditioned gradients (Sec. 4.3 ). This choice is motivated by hypernetworks’ proven effectiveness at resolving gradient conflicts in multi-task RL [ 33 ] and continual learning [ 47 ] . Our work establishes their effectiveness for the problem of cross-agent interference in MARL. Indeed, HyperMARL empirically attains lower policy gradient variance than FuPS, and we show this decoupling is critical for specialisation (Sec. 5.2 , 6.1 ), confirming its role in mitigating interference. We validate HyperMARL on diverse MARL benchmarks – including Dispersion and Navigation (VMAS) [ 5 ] , Multi-Agent MuJoCo (MAMuJoCo) [ 37 ] , SMAX [ 41 ] , and Blind-Particle Spread (BPS) [ 11 ] – across environments with two to thirty agents that require homogeneous, heterogeneous, or mixed behaviours. HyperMARL consistently matches or outperforms NoPS, FuPS, and diversity-promoting methods such as Diversity Control (DiCo) [ 6 ] , Heterogeneous-Agent Proximal Policy Optimisation (HAPPO) [ 54 ] , Kaleidoscope [ 28 ] and Selective Parameter Sharing (SePS) [ 11 ] , while achieving NoPS-level behavioural diversity while using a shared architecture. Our contributions are as follows: • We identify that cross-agent gradient interference in shared policies is critically exacerbated by the common practice of coupling agent IDs with observations (Sec. 3.2 ). • We propose HyperMARL (Sec. 4 ), an agent-conditioned hypernetwork architecture, to test the hypothesis that explicitly decoupling these gradients enables adaptive (diverse, homogeneous, or mixed) behaviours without the complexities of prior remedies (e.g., altered objectives, preset diversity levels, or sequential updates). • Our extensive evaluation (Sec. 5 ) across 22 diverse scenarios (up to 30 agents) shows HyperMARL achieves competitive returns against six strong baselines, while achieving NoPS-level behavioural diversity. We further show this decoupling is empirically linked to reduced policy gradient variance and is critical for specialisation (Sec. 5.2 ; Sec. 6.1 ). 2 Background We formulate the fully cooperative multi-agent systems addressed in our work as a Dec-POMDP [ 34 ] . A Dec-POMDP is a tuple, \langle\mathbb{I},\mathbb{S},\{\mathbb{A}^{i}\}_{i\in\mathbb{I}},R,\{\mathbb{O}^{i}\}_{i\in\mathbb{I}},O,T,\rho_{0},\gamma\rangle , where \mathbb{I} is the set of agents of size n=|\mathbb{I}| , \mathbb{S} is the set of global states with an initial state distribution \rho_{0} , \mathbb{A}^{i} is the action space for agent i where \mathbb{A}=\times_{i}\mathbb{A}^{i} is the joint action space, R:\mathbb{S}\times\mathbb{A}\to\mathbb{R} is the shared reward function, \mathbb{O}^{i} is the observation space for agent i with the joint observation space \mathbb{O}=\times_{i}\mathbb{O}^{i} , O:\mathbb{O}\times\mathbb{A}\times\mathbb{S}\to[0,1] is the probability of joint observation \mathbf{o}\in\mathbb{O} , i.e. O(\mathbf{o},\mathbf{a},s)=\Pr(\mathbf{o_{t}}|s_{t},\mathbf{a_{t-1}}) , T:\mathbb{S}\times\mathbb{A}\times\mathbb{S}\to[0,1] is the state transition function i.e. T\left(s,\mathbf{a},s^{\prime}\right)=\Pr\left(s_{t+1}\mid s_{t},\mathbf{a}_{t}\right) and \gamma is the discount factor. In this setting, each agent i receives a partial observation o_{t}^{i}\in\mathbb{O}^{i} . These observations are accumulated into an action-observation history h_{t}^{i}=\left(o_{0}^{i},a_{0}^{i},\ldots,o_{t-1}^{i},a_{t-1}^{i},o_{t}^{i}\right) . Each agent i acts based on their decentralised policies \pi^{i}(a^{i}|h^{i}) . The joint history and joint action are defined as follows \mathbf{h_{t}}=(h_{t}^{1},\ldots,h_{t}^{n}) and \mathbf{a_{t}}=(a_{t}^{1},\ldots,a_{t}^{n}) . The goal is to learn an optimal joint policy \boldsymbol{\pi^{*}}=(\pi^{1*},...,\pi^{n*}) that maximizes the expected discounted return as follows, 1 1 1 We use simplified notation here omitting explicit dependence on state transitions and distributions for brevity. \boldsymbol{\pi}^{*}=\arg\max_{\boldsymbol{\pi}}\mathbb{E}_{s_{0}\sim\rho_{0},\;\mathbf{h}\sim\boldsymbol{\pi}}\left[G(\mathbf{h})\right] , where G(\mathbf{h})=\sum_{t=0}^{\infty}\gamma^{t}R(s_{t},\mathbf{a}_{t}) . Specialised Policies and Environments. We say an environment is specialised if its optimal joint policy contains at least two distinct, non-interchangeable agent policies ( Def. 1 in App. C ). Under this mild condition, tasks such as Dispersion ( 5.2 ) or our Specialisation Game ( E.1 ) require agents to learn complementary roles rather than identical behaviours. 3 Are Independent or Fully Shared Policies Enough? Standard independent ( NoPS ) and fully parameter-shared ( FuPS ) policies face

Read the complete text