AIR: Unifying Individual and Collective Exploration in Cooperative Multi-Agent Reinforcement Learning

Guangchong Zhou, Zeren Zhang, Guoliang Fan · 2024

Read AIR: Unifying Individual and Collective Exploration in Cooperative Multi-Agent Reinforcement Learning by Guangchong Zhou, Zeren Zhang, Guoliang Fan (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-15700.txt.

Introduction

Guangchong Zhou, Zeren Zhang, and Guoliang Fan propose a new exploration method for team learning agents in a 2024 paper. Cooperative multi-agent learning struggles with exploration. Value-based agents lack an explicit policy to guide the search.

Past methods split into two camps. One explores by uncertainty for each agent alone. The other seeks diversity across the team. Extra structures often slow training and resist combination. Their method, called AIR, pits two parts against each other. A classifier guesses agent identity from behavior paths. A selector then tunes exploration mode and degree. Proofs support the design. Readers learn a unified way to explore alone and together.

Catalog record

CREATORGuangchong Zhou, Zeren Zhang, Guoliang Fan
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-15700

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AIR: Unifying Individual and Collective Exploration in Cooperative Multi-Agent Reinforcement Learning Guangchong Zhou, Zeren Zhang, Guoliang Fan Abstract Exploration in cooperative multi-agent reinforcement learning (MARL) remains challenging for value-based agents due to the absence of an explicit policy. Existing approaches include individual exploration based on uncertainty towards the system and collective exploration through behavioral diversity among agents. However, the introduction of additional structures often leads to reduced training efficiency and infeasible integration of these methods. In this paper, we propose Adaptive exploration via Identity Recognition (AIR), which consists of two adversarial components: a classifier that recognizes agent identities from their trajectories, and an action selector that adaptively adjusts the mode and degree of exploration. We theoretically prove that AIR can facilitate both individual and collective exploration during training, and experiments also demonstrate the efficiency and effectiveness of AIR across various tasks. Code — https://github.com/Jugg1er/AIR Introduction Multi-agent reinforcement learning (MARL) has achieved outstanding results in complex cooperative tasks, such as flocking control ( Xu et al. 2018 ; Gu et al. 2023 ) , autonomous driving ( Shamsoshoara et al. 2019 ; Zhang et al. 2023b ) , and sensor network ( Zhao et al. 2023 ) . To deal with the partial observability and the vast joint spaces of agents, most MARL methods follow the centralized training and decentralized execution (CTDE) paradigm. Especially, value-based approaches ( Sunehag et al. 2017 ; Rashid et al. 2018 ; Yang et al. 2020 ; Wang et al. 2020a ) enjoy high sample efficiency and great scalability, ultimately achieving superior performance in popular benchmarks. However, most value-based methods suffer from insufficient exploration due to the adoption of the vanilla \epsilon -greedy exploration strategy, and how to effectively enhance the exploration of value-based agents in cooperative tasks remains a challenging problem. In some studies, the exploration of an agent is guided by its uncertainty towards the system. Zhang et al. 2023a define the uncertainty as the correlation between an agent’s action and other agents’ observations, based on which they adjust exploration rate \epsilon dynamically. EITI & EDTI ( Wang et al. 2019 ) and EMC ( Zheng et al. 2021 ) refer to the concept of “curiosity”, which is measured by the model prediction error in practice and used as an intrinsic reward to facilitate exploration. These approaches introduce perturbations into individual policies but do not explicitly consider coordination, and we call them individual exploration. Compared to single-agent domain, multi-agent cooperative tasks typically require various skills and have vastly larger policy space, rendering comprehensive exploration infeasible for a single agent. An ideal solution is to keep the diversity of agents’ behaviors, which enables individuals to acquire distinct skills, explore disparate regions of the policy space, and share their experiences by parameter sharing technique. Recently, a group of methods revealed the importance of behavioral diversity in collective exploration. As shown in Figure 1(a) , all agents learn to compete for the ball and form terrible cooperation, while the globally optimal policy in Figure 1(b) induces different behaviors among agents for tacit cooperation ( Li et al. 2021 ) . Such sub-optimal situations arise exclusively in multi-agent tasks and can not be alleviated through individual exploration. MAVEN ( Mahajan et al. 2019 ) proposes committed exploration that learns a diverse ensemble of monotonic approximations with the help of a latent space to explore. CDS ( Li et al. 2021 ) maximizes an information-theoretical regularizer to produce diverse individualized behaviors for extensive exploration. In role-based methods ( Wang et al. 2020b ; Wang et al. 2020c ; Yang et al. 2022 ; Zhou et al. 2023 ) , agents are required to deal with distinct subtasks based on their assigned roles, thereby reducing conflicts and accelerating the formation of cooperation. (a) Sub-optimal policy. (b) Optimal policy. Figure 1: The importance of behavioral diversity in Google Research Football. (a) Agents all compete for the ball, exhibiting homogeneous behaviors and poor coordination. (b) Agents behave differently to achieve coordination. Both individual skills and group collaboration are crucial in cooperative tasks. However, even without methodological conflicts, a direct integration of the above individual and collective exploration methods is infeasible, as excessive extra modules make the overall model too heavy to train. To the best of our knowledge, no research has integrated individual and collective exploration into a unified framework. To address this gap, we propose Adaptive exploration via Individual Recognition (AIR). We first learn an identity classifier that distinguishes corresponding agents based on given trajectories. Based on the classifier, we conduct theoretical analysis and design a unified exploration mechanism that can switch the exploration mode between individual and collective exploration according to the training stage as well as tuning the degree of exploration. Our main contributions can be summarized as follows: • We analyze the impact and limitations of traditional exploration strategies within multi-agent systems and reveal that multi-agent exploration accounts for both individual behaviors and the joint behaviors of agents. • We propose a novel multi-agent exploration framework for value-based agents called AIR, which enables dynamic adjustments of exploration during training. To the best of our knowledge, we are the very first to address individual and collective exploration in a single unified framework. • Our proposed method incorporates only the minimum additional structure and is computationally efficient. Experiments demonstrate the remarkable effectiveness of AIR across various multi-agent tasks. Related Works Value-based MARL Methods Independent Q-learning ( Tampuu et al. 2017 ) directly migrates DQN into multi-agent tasks, but it completely ignores the interactions between agents, leading to various training issues. Value decomposition factorizes the global value into individual ones and alleviates the instability during training. VDN ( Sunehag et al. 2017 ) and QMIX ( Rashid et al. 2018 ) aggregate individual Q-values into a global one through additive and monotonic functions respectively. Qatten ( Yang et al. 2020 ) replaces the dense network in QMIX with the attention mechanism. To further enhance the capability of the mixing network to express richer joint action-value function classes, QTRAN ( Son et al. 2019 ) constructs two soft regularizations to align the greedy action selections between the joint and individual value functions. Weighted-QMIX ( Rashid et al. 2020 ) introduces a weighting mechanism into the projection of monotonic value factorization to place more importance on better joint actions. QPLEX ( Wang et al. 2020a ) proposes a duplex dueling network architecture to loose the monotonicity constraints in QMIX. QPro ( Mei, Zhou, and Lan 2024 ) casts the factorization problem as regret minimization over the projection weights of different state-action values. Exploration for Value-based Agents in MARL Despite their success, the simple \epsilon -greedy exploration strategy used in the above methods has been found ineffective in solving coordination tasks with complex state and reward transitions. Some studies adjust the intensity of individual exploration dynamically according to the metric of uncertainty towards the system. SMMAE ( Zhang et al. 2023a ) refers to the correlation between each agent’s action and others’ observations. EITI & EDTI ( Wang et al. 2019 ) quantify and characterize the influence of one agent’s behavior on others using mutual information and the difference of expected returns respectively, while EMC ( Zheng et al. 2021 ) uses prediction errors

Read the complete text