Q-Learning–Driven Adaptive Rewiring for Cooperative Control in Heterogeneous Networks

Yi-Ning Weng, Hsuan-Wei Lee · 2025

Read Q-Learning–Driven Adaptive Rewiring for Cooperative Control in Heterogeneous Networks by Yi-Ning Weng, Hsuan-Wei Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-01057.txt.

Introduction

Yi-Ning Weng and Hsuan-Wei Lee propose Q-learning-driven adaptive rewiring for cooperation in networks. Cooperation in multi-agent systems is a basic problem in statistical physics. Small learning rules can drive large collective shifts there. Their method builds on adaptive rewiring work in the literature. It joins temporal difference learning with network restructuring.

Agents optimize both strategies and social ties from past interactions. Neighbor-specific Q-learning lets agents manage partnerships with care. Cooperator clusters form, set apart from defective regions in space. Tests use power-law networks that mirror real connectivity patterns. Three regimes appear across rewiring constraint levels. A permissive regime enables rapid cluster formation. A middle regime depends sensitively on dilemma strength. A patient regime optimizes structure only gradually.

Catalog record

CREATORYi-Ning Weng, Hsuan-Wei Lee
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-01057

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Q-Learning–Driven Adaptive Rewiring for Cooperative Control in Heterogeneous Networks Yi-Ning Weng, Hsuan-Wei Lee Abstract Cooperation emergence in multi-agent systems represents a fundamental statistical physics problem where microscopic learning rules drive macroscopic collective behavior transitions. We propose a Q-learning-based variant of adaptive rewiring that builds on mechanisms studied in the literature. This method combines temporal difference learning with network restructuring so that agents can optimize strategies and social connections based on interaction histories. Through neighbor-specific Q-learning, agents develop sophisticated partnership management strategies that enable cooperator cluster formation, creating spatial separation between cooperative and defective regions. Using power-law networks that reflect real-world heterogeneous connectivity patterns, we evaluate emergent behaviors under varying rewiring constraint levels, revealing distinct cooperation patterns across parameter space rather than sharp thermodynamic transitions. Our systematic analysis identifies three behavioral regimes: a permissive regime (low constraints) enabling rapid cooperative cluster formation, an intermediate regime with sensitive dependence on dilemma strength, and a patient regime (high constraints) where strategic accumulation gradually optimizes network structure. Comparative analysis against Bush-Mosteller stimulus-response learning demonstrates that Q-learning’s temporal credit assignment capabilities produce superior cooperation outcomes, particularly under intermediate rewiring constraints where long-term relationship assessment becomes crucial. Simulation results show that while moderate constraints create transition-like zones that suppress cooperation, fully adaptive rewiring enhances cooperation levels through systematic exploration of favorable network configurations. Quantitative analysis reveals that increased rewiring frequency drives large-scale cluster formation with power-law size distributions. Our results establish a new paradigm for understanding intelligence-driven cooperation pattern formation in complex adaptive systems, revealing how machine learning serves as an alternative driving force for spontaneous organization in multi-agent networks. 1 Introduction Ensuring cooperative control in distributed engineered systems and applications is a daunting challenge across diverse domains. In distributed resource management, cooperative agents must dynamically adapt to balance local demands and maintain global performance [ 1 ] ; in urban traffic networks, intersections must exchange information to optimize flows [ 2 , 3 ] ; in robotic swarms, unmanned aerial vehicles or mobile robots must align actions for collective tasks under uncertainty [ 4 , 5 ] . Apparently, in each case, the performance of the overall system, including throughput, latency, reliability, and safety, depends on the ability of autonomous agents to adapt strategies and restructure interactions in dynamic environments. Enhancing cooperation among agents is therefore essential, since insufficient coordination can lead to cascading failures, degraded performance, or even systemic collapse in critical infrastructures. This engineering perspective also mirrors a deeper scientific puzzle: how self-interested agents, across natural, social, and artificial systems, manage to sustain cooperation despite incentives to defect. Deciphering how self-interested agents achieve cooperation remains a central open question spanning statistical physics, evolutionary biology, the social sciences, and complex systems theory [ 6 , 7 , 8 , 9 ] . Despite the apparent contradiction between individual rationality and collective benefit, cooperative behavior pervades natural and artificial systems from microbial colonies [ 10 ] and neural networks [ 11 ] to economic markets [ 12 ] and distributed computing architectures [ 13 ] . This ubiquity suggests that cooperation emergence follows universal physical principles that transcend specific biological or technological implementations. Understanding these fundamental mechanisms has profound implications for designing beneficial artificial societies and explaining the ubiquity of cooperation in nature. The Prisoner’s Dilemma (PD) has served as the canonical framework for investigating cooperation emergence, where individual rationality paradoxically leads to collectively suboptimal outcomes. Traditional studies of PD have primarily considered static networks or well-mixed populations, where agents interact with fixed partners over repeated rounds or are distributed in homogeneous topologies [ 14 , 15 , 16 , 17 , 18 , 19 , 20 ] . These classical approaches revealed fundamental cooperation mechanisms including spatial reciprocity, group selection, and reputation effects. However, real-world systems exhibit inherently dynamic topologies with continuously evolving connections and heterogeneous agent behaviors. This dynamic nature suggests that network plasticity represents a fundamental driver of cooperation emergence rather than a secondary supportive factor. Recognition of this limitation has sparked extensive research into adaptive networks that grant agents the freedom to break unproductive ties and forge new ones. In recent years, researchers have extended the scope of PD studies to dynamic networks, allowing agents to adjust their connections over time through tie breaking and edge formation [ 21 , 22 , 23 , 24 ] . Such network rewiring mechanisms offer agents the ability to sever unproductive links and form new ties, mimicking the adaptive nature of social relationships [ 25 , 26 , 27 , 28 ] . Theoretical and empirical studies have demonstrated that even simple rewiring rules, such as preferentially connecting to cooperators or severing ties with defectors, can dramatically improve population-level cooperation [ 29 , 30 , 31 , 32 ] . These findings established network adaptability as a powerful mechanism for escaping exploitation traps and forming stable cooperative clusters. Despite these advances, most rewiring mechanisms rely on predetermined heuristics or static strategies that lack the flexibility to adapt to diverse local environments through experience. Recent advances have shown that incorporating reinforcement learning into network rewiring decisions can significantly enhance cooperation levels, with adaptive rewiring mechanisms guided by Bush-Mosteller learning [ 33 , 34 , 35 ] creating more stable cooperative structures compared to static strategies [ 36 , 37 ] . However, these approaches typically employ simple stimulus-response learning without the sophisticated temporal credit assignment capabilities of modern reinforcement learning algorithms. Recent advances in machine learning now allow these adaptive rules to be learned rather than hand-coded, linking statistical-physics models to modern artificial intelligence [ 38 , 39 , 40 ] . Reinforcement learning (RL) has emerged as particularly powerful for designing adaptive agents capable of learning optimal strategies through environmental feedback [ 41 , 42 , 43 , 44 , 45 , 46 ] . Q-learning is particularly attractive because it is model-free, converges under mild conditions, and scales to the large heterogeneous graphs typical of social systems [ 47 , 48 , 49 , 50 , 51 , 52 ] . The temporal difference learning mechanism in Q-learning enables sophisticated evaluation of long-term relationship value, making it ideally suited for partnership management in dynamic social networks. Complementary work on Interactive Diversity (ID) shows that agents gain when they tailor their actions to each neighbor rather than deploying a single global strategy [ 53 , 54 , 55 , 56 , 57 ] . Edge-based analytical frameworks reveal that such personalized interaction strategies are crucial for maintaining extensive reciprocal relationships, ultimately generating resilient cooperative clusters that persist across diverse parameter regimes. This Interactive Diversity framework has demonstrated superior performance in sustaining cooperation compared to traditional uniform-strategy approaches, particularly under moderate temptation conditions [ 58 ] . However, the potential for combining such behavioral sophistication with intelligent network adaptation through reinforcement learning remains largely unexplored. Bringing value-based learning together with neighbor-specific decision making therefore offers an unexplored route to agents that optimize both behavior and topology in tandem. While extensive research has focused on learning action strategies (deciding when to cooperate or defect), the potential for reinforcement learning to guide intelligent network rewiring decisions remains largely unexplored. To address this limitation, we propose a novel framework that integrates Q-learning with adaptive network rewiring in dynamic

Read the complete text