How Exploration Breaks Cooperation in Shared-Policy Multi-Agent Reinforcement Learning
Yi-Ning Weng, Hsuan-Wei Lee · 2026
Read How Exploration Breaks Cooperation in Shared-Policy Multi-Agent Reinforcement Learning by Yi-Ning Weng, Hsuan-Wei Lee (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-05509.txt.
Introduction
Yi-Ning Weng and Hsuan-Wei Lee show, in a 2026 paper, that routine exploration can break cooperation in shared-policy multi-agent learning. Many agents sharing one policy is a common way to scale reinforcement learning. In social dilemmas, stable and payoff-dominant cooperative outcomes may still exist.
Controlled experiments with shared Deep Q-Networks settled into persistent low cooperation. The cause was not misaligned rewards, noise, or short training. Partial observability plus shared parameters biased the common representation toward defection, and that bias spread across agents. The collapse held across network sizes, exploration schedules, and payoffs. It vanished when agents used separate policies or representations, giving designers a clear structural warning.
Catalog record
| CREATOR | Yi-Ning Weng, Hsuan-Wei Lee |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-05509
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
How Exploration Breaks Cooperation in Shared-Policy Multi-Agent Reinforcement Learning Yi-Ning Weng, Hsuan-Wei Lee Abstract Multi-agent reinforcement learning (MARL) in dynamic games poses fundamental challenges due to non-stationary interactions and exploration noise, which are often compounded by common design choices such as parameter sharing. While parameter sharing is commonly adopted to improve sample efficiency and scalability, its implications for cooperative behavior under exploration remain poorly understood. In this work, we conduct a systematic study of Deep Q-Networks (DQN) with parameter sharing in a dynamic Prisoner’s Dilemma environment, focusing on how exploration and payoff harshness jointly shape learning stability and cooperation outcomes. We reveal a robust cooperation collapse in shared DQN as exploration increases. The collapse is most severe when payoffs strongly favor defection. This collapse is not sporadic but forms structured failure regions in the parameter space, with structured failure boundaries that shift systematically with game severity. This phenomenon cannot be explained by payoff incentives or learning instability alone. Cooperation collapse coincides with degradation in learned representations: cooperative and defective behaviors become entangled in the feature space, and action-value gaps shrink systematically. The system converges to stable but low-cooperation solutions. We identify partial observability and representational coupling as key drivers. Grouped policy learning, which limits parameter sharing to subsets of agents, consistently alleviates collapse. State augmentation with learning progress and exploration signals significantly delays collapse and partially reverses it in some regimes. Network topology also matters: spatial structure supports cooperation through local coordination, while random networks eliminate cooperative regimes. These interventions confirm that the failure is representational, not algorithmic. Our findings reveal a fundamental failure mode of parameter-shared Deep Q-Networks in cooperative settings: shared representations cannot maintain stable cooperation under exploration. However, representation-aware state design can substantially improve robustness. These results have implications for designing scalable MARL systems where cooperation is essential. 1 Introduction Multi-agent reinforcement learning (MARL) provides a framework for modeling systems in which multiple autonomous agents learn through repeated interaction. Applications range from distributed control and networked systems to large-scale coordination problems, where individual decision-makers must adapt to both environmental feedback and the evolving behaviors of others. Exploration poses a fundamental challenge in reinforcement learning. This challenge intensifies in multi-agent settings, where concurrent learning agents induce non-stationarity in the effective environment each agent experiences [ 1 , 2 , 3 ] . While structured exploration mechanisms have been proposed to improve sampling efficiency in single-agent DQN [ 4 ] , their interaction with parameter sharing in multi-agent settings remains poorly understood. Understanding how learning architectures interact with non-stationarity and exploration is therefore essential for building scalable and robust MARL systems [ 5 , 6 , 7 , 8 , 9 , 10 ] . Parameter sharing has become a standard approach to address scalability in MARL [ 11 , 12 , 13 ] . By training multiple agents using a shared policy or value function, this design reduces sample complexity and enables learning in systems with large agent populations. Many value-based cooperative MARL methods adopt centralized training with decentralized execution, incorporating information about teammates’ behaviors during training [ 14 , 15 ] . Despite these advantages, parameter sharing imposes a fundamental constraint: all agents are coupled through a single representational and optimization pathway [ 16 , 17 ] . Every agent contributes gradients to, and draws decisions from, the same network, regardless of differences in local interaction contexts. As a result, shared policies are exposed to aggregated exploration noise and non-stationary feedback from many interacting agents [ 18 ] . Parameter sharing is often treated as a benign engineering choice. However, its implications for learning stability and cooperation under exploration remain poorly understood, particularly when agents face heterogeneous and evolving interaction contexts. This gap in understanding is particularly critical in cooperative settings. Sustained coordination requires stable feedback between individual actions and collective outcomes. Small perturbations from exploration or miscoordination can propagate through interactions and disrupt cooperative structures [ 19 , 20 , 21 ] . Cooperation breakdowns are frequently observed in shared-policy MARL under exploration. However, they are rarely studied as systematic phenomena. Three potential explanations exist: unavoidable incentive conflicts, training instability, or fundamental limitations in how shared representations handle non-stationary multi-agent interactions. This work distinguishes among these possibilities. Network structure also plays a role in cooperative learning. Prior work shows that network topology and adaptive rewiring influence the emergence and stability of cooperation in learning systems [ 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 ] . We examine how interaction structure modulates the cooperation collapse we identify. We investigate these questions through a systematic study of parameter-shared Deep Q-Networks in a dynamic Prisoner’s Dilemma. This environment serves as a diagnostic setting that allows controlled manipulation of incentive structure, exploration strength, and network topology [ 30 , 31 , 32 , 33 , 34 ] . We use cooperation rate not as an optimization objective, but as an indicator of learning stability and coordination quality. Our main finding is a robust cooperation collapse in parameter-shared Deep Q-Networks as exploration increases. This collapse occurs across broad regions of the parameter space and forms structured failure boundaries that shift systematically with game severity. To understand this failure, we examine the internal dynamics of the shared network. We find that cooperation collapse coincides with degradation in learned representations. Cooperative and defective behaviors become increasingly entangled in the latent space. At the value level, action-value gaps contract systematically. This suggests convergence to stable but low-cooperation attractors, not training instability. These findings rule out two simple explanations: incentive misalignment and excessive exploration noise. Instead, the failure is rooted in how shared representations handle non-stationary multi-agent learning. If representational coupling causes the failure, then reducing it should help. We test this prediction using grouped policy learning. Grouped policy learning limits parameter sharing to subsets of agents. This consistently improves stability and delays collapse, confirming that global representation sharing contributes to the failure. We next test whether the failure stems from partial observability of learning dynamics. We augment the state with signals encoding learning progress and exploration intensity. This augmentation delays cooperation collapse across all exploration levels. In some regimes, it partially reverses the collapse, restoring stable cooperation over a wider parameter range. These interventions show that state and representation design are critical for shared-policy MARL. Exploration tuning alone cannot achieve robustness. We also examine how network topology affects cooperation collapse. Spatially structured grids support cooperation by providing local coordination scaffolding. Random topologies eliminate this benefit, making cooperation collapse more severe. This reveals a boundary condition: parameter-shared DQN requires environmental structure to sustain cooperation under exploration. Learning dynamics alone are insufficient. This work identifies and characterizes a fundamental failure mode of parameter-shared Deep Q-Networks in cooperative settings. We show that representation-aware state design can mitigate this failure. Our findings link macro-level cooperation failure to internal representation dynamics. This provides guidance for designing more robust shared-policy MARL systems. 2 Methods We study a multi-agent reinforcement learning system in which agents learn to act in a dynamic Prisoner’s Dilemma Game (PDG) using Deep Q-Networks (DQN). The model builds upon established frameworks that integrate reinforcement learning with evolutionary game dynamics, and is designed to examine learning behavior under non-stationary interactions