Tackling Uncertainties in Multi-Agent Reinforcement Learning through Integration of Agent Termination Dynamics

Somnath Hazra, Pallab Dasgupta, Soumyajit Dey · 2025

Read Tackling Uncertainties in Multi-Agent Reinforcement Learning through Integration of Agent Termination Dynamics by Somnath Hazra, Pallab Dasgupta, Soumyajit Dey (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-12061.txt.

Introduction

Somnath Hazra, Pallab Dasgupta, and Soumyajit Dey teach packs of game agents to weigh risk and shun faults while they learn team tasks, in a paper from 2025. Spread learn, which tracks whole prize spreads not means, works for lone agents but lags for packs. Faults hid in the rig give the team a risk read to learn from.

A wall shaped loss turns those fault marks into a safe pull on each choice. The pull counts most in the wild first phase of learn, then lets skill rule. On a hard war game test bed, the pack learns faster and ends past top past packs on both safe play and wins. Safe care lifts pack learn, they say.

Catalog record

CREATORSomnath Hazra, Pallab Dasgupta, Soumyajit Dey
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-12061

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Tackling Uncertainties in Multi-Agent Reinforcement Learning through Integration of Agent Termination Dynamics Somnath Hazra, Pallab Dasgupta, Soumyajit Dey Abstract . Multi-Agent Reinforcement Learning (MARL) has gained significant traction for solving complex real-world tasks, but the inherent stochasticity and uncertainty in these environments pose substantial challenges to efficient and robust policy learning. While Distributional Reinforcement Learning has been successfully applied in single-agent settings to address risk and uncertainty, its application in MARL is substantially limited. In this work, we propose a novel approach that integrates distributional learning with a safety-focused loss function to improve convergence in cooperative MARL tasks. Specifically, we introduce a Barrier Function based loss that leverages safety metrics, identified from inherent faults in the system, into the policy learning process. This additional loss term helps mitigate risks and encourages safer exploration during the early stages of training. We evaluate our method in the StarCraft II micromanagement benchmark, where our approach demonstrates improved convergence and outperforms state-of-the-art baselines in terms of both safety and task completion. Our results suggest that incorporating safety considerations can significantly enhance learning performance in complex, multi-agent environments. Y. Vorobeychik, S. Das, A. Nowé (eds.) Conference: Proc. of the 24th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2025); May 19 – 23, 2025; Detroit, Michigan, USA <<OpenReview submission id>> CCS: Theory of computation Multi-agent reinforcement learning CCS: Theory of computation Multi-agent learning 1. Introduction Multi-Agent Systems (MAS) remain a highly challenging domain, owing to the inherent stochasticity in the environment, and concurrently learning agents; resulting in uncertain reward outcomes. Therefore the contemporary Multi-Agent Reinforcement Learning (MARL) literature has shifted from point estimates derived from expected global rewards ( Sunehag et al., 2017 ; Rashid et al., 2020 ) to distributional analysis over returns ( Bellemare et al., 2017 ; Dabney et al., 2018a ) which provide a more informed understanding of the variability in future returns. However, the predicted distributions are prone to errors at the initial phases of training because the policy has not yet explored and trained enough to predict accurate distributions. In the context of MARL, the prediction errors become even more critical due to the collective uncertainty from multiple agents, where each agent’s decisions can drastically impact both individual and collective outcomes. Our focus lies specifically on cooperative MARL, where a team of agents collaborates to achieve a shared objective in a stochastic environment ( Albrecht et al., 2024 ) . Despite the effectiveness of the Centralized Training and Decentralized Execution (CTDE) paradigm ( Lowe et al., 2017 ) for cooperative MARL most CTDE-based algorithms maximize returns and overlook the uncertainty due to untrained parameters in the policy that captures the inherent stochasticity in MAS. Here agents must contend with not only uncertain rewards but also additional safety constraints resulting from the stochasticity. Our contribution is to identify the innate fault tolerant MARL formulation from the system such that the system does not go beyond a pre-defined safety boundary. For example, a team game cannot be won if players keep getting eliminated. These safety objectives hold importance during the early phases of training since it is hard to estimate accurate distributions by a distributional model owing to exploration strategies and partial observability of the agents. Figure 1. Integration of the safety constraints along with loss functions calculated with respect to returns, helps reduce the uncertainity, especially during the initial phase of training. As an instance of such a situation, let us consider a battle scenario between two agent groups. The reward can be based on intermediate strategies decided by the teams. But it is unsure whether their strategy will work, since any agent army does not know about the strategies planned by the other army. Moreover the agents may not be experts in carrying out their decided objective. However, one property that is overlooked, is that the battle cannot be won if there are not enough agents left to fight; i.e., there is an inherent fault tolerant formulation in the system. For example, in battle simulations like StarCraft II, minimizing ally casualties can be as important as maximizing reward ( Samvelyan et al., 2019 ) . Motivated by the gap in current research, our work addresses the uncertainties of the system via integration of safety considerations from the inherent faults in the MAS. While previous efforts have successfully combined Distributional RL with MARL to improve risk sensitivity ( Qiu et al., 2020 ; Sun et al., 2021a ) , these approaches largely focus on algorithmic modifications rather than optimizing the learning process by using the available information regarding the safety-critical objectives. Our contributions are summarized as follows: • We propose a novel loss function that incorporates safety constraints into the optimization process, derived from the safety boundary of the inherent faults in the system. By explicitly accounting for safety alongside reward maximization, we aim to accelerate learning, tackle environmental uncertainties, and improve long-term policy performance in risk-sensitive environments. We also use the Dueling Networks ( Wang et al., 2016 ) in our architecture to eliminate ineffective actions. • We also suggest an architectural improvement for the local policy network in the CTDE paradigm to utilize the return distribution for prioritizing the important components from the local observation. • We validated our approach on the StarCraft II micromanagement ( Samvelyan et al., 2019 ) and MetaDrive ( Li et al., 2021 ) benchmarks, demonstrating its efficacy compared to existing baselines and state-of-the-art distributional MARL algorithms. More specifically, our work contributes to bridging the gap between theoretical advancements in Distributional MARL and practical needs in real-world MARL environments, particularly those with critical safety concerns. 2. Preliminaries This section outlines some of the preliminary concepts. 2.1. Decentralized POMDP The de facto standard for modeling cooperation in MARL is a Decentralized Partially Observable MDP or Dec-POMDP ( Oliehoek et al., 2008 ) . It is defined using the tuple \mathcal{M}=\langle\mathcal{N},\mathcal{S},\mathcal{U},\Omega,P,\mathcal{O},r,\gamma\rangle , where \mathcal{N}\equiv\{a_{1},a_{2},...,a_{n}\} is a finite set of agents, and s\in\mathcal{S} represents the global state of the system. At each time-step, the individual agents choose an action u_{i}\in\mathcal{U} for i\in\mathcal{N} based on observation o_{i}\in\Omega , with probability \mathcal{O}(o_{i}|s,u_{i}) . The joint action, \textbf{u}=[u_{i}]_{i=1}^{n} , changes the state of the system from s to s^{\prime}\sim P(\cdot|s,\textbf{u}) ; yielding a joint reward r(s,\textbf{u}) . \gamma\in[0,1) is the discount factor. Value functions provide an estimate of the reward that can be obtained given a state, or when taking an action from a state. Accordingly, the state value function for the system is given by, V_{\bm{\pi}}(s)=\mathbb{E}[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},\textbf{u}_{t})|s_{0}=s,\bm{\pi}] ; and the state-action value function is given by, Q_{\bm{\pi}}(s,\textbf{u})=r(s,\textbf{u})+\gamma\mathbb{E}_{s^{\prime}}V_{\bm{\pi}}(s^{\prime}) . The formal objective of the co-operative system is to find a joint policy \bm{\pi}\equiv\langle\pi_{1},...,\pi_{n}\rangle that maximizes the expected return given by: (1) \bm{\pi}*=\arg\max_{\bm{\pi}}\mathbb{E}_{(s_{t},\textbf{u}_{t})\sim(P,\bm{\pi})}\left[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},\textbf{u}_{t})\right] 2.2. Centralized Training and Decentralized Execution Multi-agent algorithms generally follow the Centralized Training Decentralized Execution (CTDE) paradigm ( Lowe et al., 2017 ) . Agents learn collaboratively during training using feedback from the environment. This phase may involve information exchange with a central approximation network. This phase is termed as the Central Training phase. The Central Value function accumulates the values of the individual policies \pi_{i} in a forward propagation determined by the Credit Assignment strategy ( Chang et al., 2003 ) of the algorithm. The strategies

Read the complete text