Multi-Agent Deep Reinforcement Learning under Constrained Communications

Shahil Shaik, Jonathon M. Smereka, Yue Wang · 2026

Read Multi-Agent Deep Reinforcement Learning under Constrained Communications by Shahil Shaik, Jonathon M. Smereka, Yue Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-17069.txt.

Introduction

Shahil Shaik, Jonathon Smereka, and Yue Wang train bot teams that talk on tight links. Team learn oft trains in one hub, then splits acts out. That hub needs full state facts to train well. Such need makes teams frail and dear to retrain. Fresh mates or new scenes can break them. Their 2026 preprint trains teams in a spread way in place of hub style.

Spread teams learn from local facts and peer chats. Tight links curb how much peers can share. Teams must then learn to share what counts most. Such teams suit drone fleets and field robots. They can adapt when mates join or drop.

Catalog record

CREATORShahil Shaik, Jonathon M. Smereka, Yue Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-17069

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multi-Agent Deep Reinforcement Learning under Constrained Communications Shahil Shaik, Jonathon M. Smereka, Yue Wang Abstract Centralized training with decentralized execution (CTDE) has been the dominant paradigm in multi-agent reinforcement learning (MARL), but its reliance on global state information during training introduces scalability, robustness, and generalization bottlenecks. Moreover, in practical scenarios such as adding/dropping teammates or facing environment dynamics that differ from the training, CTDE methods can be brittle and costly to retrain, whereas distributed approaches allow agents to adapt using only local information and peer-to-peer communication. We present a distributed MARL framework that removes the need for centralized critics or global information. Firstly, we develop a novel Distributed Graph Attention Network (D-GAT) that performs global state inference through multi-hop communication, where agents integrate neighbor features via input-dependent attention weights in a fully distributed manner. Leveraging D-GAT, we develop the distributed graph-attention MAPPO (DG-MAPPO) – a distributed MARL framework where agents optimize local policies and value functions using local observations, multi-hop communication, and shared/averaged rewards. Empirical evaluation on the StarCraftII Multi-Agent Challenge, Google Research Football, and Multi-Agent Mujoco demonstrates that our method consistently outperforms strong CTDE baselines, achieving superior coordination across a wide range of cooperative tasks with both homogeneous and heterogeneous teams. Our distributed MARL framework provides a principled and scalable solution for robust collaboration, eliminating the need for centralized training or global observability. To the best of our knowledge, DG-MAPPO appears to be the first to fully eliminate reliance on privileged centralized information, enabling agents to learn and act solely through peer-to-peer communication. 1 Introduction Multi-agent reinforcement learning (MARL) has emerged as a powerful framework for training multiple agents to learn cooperative and competitive behaviors in complex dynamic environments ( 27 ) . However, learning effective collaborative policies remains challenging, as each agent simultaneously seeks to maximize its own return, giving rise to the fundamental issue of non-stationarity: the environment is constantly changing due to the evolving behaviors of other agents from the perspective of any single agent. Recent works such as MAPPO ( 26 ) , MADDPG ( 15 ) , and HAPPO/HATRPO ( 11 ; 29 ) alleviate this challenge by using the centralized training decentralized execution (CTDE) framework, where agents assume access to global state information during training but rely on local information during execution. Although effective, the CTDE approach suffers from several drawbacks that limit its applicability in practical settings. First, it requires access to global information during training, which may be infeasible in large-scale systems due to communication bandwidth, latency, or privacy constraints —for example, wireless and ad-hoc networks often exhibit strong trade-offs between communication range, throughput, and latency ( 21 ) , making long-range low-latency communication difficult to sustain. These limitations are particularly pronounced in off-road robotics, distributed sensing networks, and search-and-rescue settings, where cluttered terrain and unreliable links limit agents to short-range communication ( 4 ; 6 ) . In such environments, relying on global information is impractical, motivating the need for learning frameworks that operate effectively with only local communication. Moreover, CTDE methods often suffer from a train–test mismatch: agents are optimized with privileged global information that is unavailable during execution, which can lead to poor generalization once this information is removed. These limitations highlight the need for distributed MARL approaches that enable agents to learn cooperative strategies using only local observations and peer-to-peer communication among neighboring agents. However, fully distributed learning techniques remain relatively underexplored compared to the centralized training approaches, partly due to the inherent complexity of the problem. Existing studies in this domain typically retain some form of centralization. For instance, 28 proposed a decentralized multi-agent actor–critic algorithms that use average consensus protocols ( 23 ) to approximate global returns and value functions via neighbor communication, while actors update their policies independently. Although effective, this approach relies on simple averaging of value functions, which can yield suboptimal performance in heterogeneous teams with non-i.i.d. dynamics. Moreover, their framework assumes access to global state information for advantage estimation and thus cannot directly handle partial observability. In parallel, graph-based methods have been introduced to better capture structured communication among agents. 9 proposed graph convolution RL (DGN), where agents are represented as nodes in a dynamic graph and leverage Graph Attention Networks (GATs) ( 24 ) to process node-level observations and actions. However, DGN shares both the Q-network and GAT parameters among all agents, preventing fully distributed training. In addition to these decentralized methods, GNN augmented MARL approaches have also explored richer communication structures. For instance, the recent survey by 14 outlines a broad class of GNN-based communication architectures (GNNComm-MARL) that enhance message routing, neighborhood selection, and multi-hop reasoning in cooperative tasks, but these methods continue to rely on CTDE and centralized critics. The attentional communication mechanism ATOC ( 10 ) also adopts this paradigm: agents use a learned attention module to decide when to communicate; however, training still depends on a centralized critic and shared parameterization, which prevents fully distributed learning. Similarly, 7 proposed a GNN-based patrolling framework where agents use deep message passing to overcome partial observability and communication disturbances; nevertheless, their actor–critic structure follows the CTDE paradigm, and training remains centralized. Another relevant line of work integrates information-theoretic objectives: 3 introduced mutual-information–guided GNN communication to enhance representation quality in value-decomposition MARL. Despite its strong empirical performance, MARGIN requires centralized mixing of value functions and thus is not fully distributed. In the context of UAV coordination, 5 employed GNN observers to handle dynamic neighbor sets, and used transfer learning to accelerate QMIX-based training—again relying on centralized value mixing. Overall, existing GNN augmented MARL works demonstrate that graph-structured message passing improves coordination and robustness, especially under partial observability. However, none of these approaches enable fully distributed policy optimization, as they all depend on centralized critics, centralized value decomposition, or shared GNN parameterization across all agents. This leaves a significant gap: how to design a MARL framework in which agents learn cooperative behaviors using only local observations, peer-to-peer communication, and fully distributed updates, without any reliance on centralized components or privileged information. We could bridge this gap by grounding MARL in distributed optimization techniques. For instance, decentralized stochastic gradient descent (D-SGD) ( 13 ; 1 ) and classical distributed averaging protocols ( 16 ; 23 ) provide strong theoretical foundations for consensus optimization over networks. Building on these insights, we introduce Distributed Graph Attention Networks (D-GATs) , which couple the expressiveness of GATv2 ( 2 ) with neighbor-averaged parameter sharing inspired by D-SGD. This design preserves dynamic, input-dependent attention while promoting consensus among agents in a fully distributed setting. While related work, such as GATTA ( 22 ) , has applied graph attention to distributed supervised learning and personalization, it incurs higher computational overhead that scales poorly with the number of agents—making it less suitable for multi-agent reinforcement learning settings where efficiency is critical. In contrast, our Distributed Graph Attention Network (D-GAT) is specifically designed for MARL, focusing on global state inference via lightweight, input-dependent attention mechanisms that remain tractable even in large teams. Building on D-GAT, we introduce distributed graph-attention MAPPO (DG-MAPPO) , a principled distributed MARL framework that removes the need for centralized training or global observability. DG-MAPPO integrates agents’ local observations with global state inference from D-GAT and a

Read the complete text