Shaping the learning signal in a combined Q-learning rule to improve structured cooperation Journal: Chaos, Solitons & Fractals

Chunpeng Du, Zongyang Li, Yali Zhang, Yikang Lu, Attila Szolnoki · 2026

Read Shaping the learning signal in a combined Q-learning rule to improve structured cooperation Journal: Chaos, Solitons & Fractals by Chunpeng Du, Zongyang Li, Yali Zhang, Yikang Lu, Attila Szolnoki (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21392.txt.

Introduction

Chunpeng Du, Zongyang Li, Yali Zhang, Yikang Lu, and Attila Szolnoki study how cooperation grows when agents learn from a shared score signal. Q-learning is a standard frame for such studies, since it says how agents update the value of each act. Reputation is known to help cooperation grow.

Past models baked reputation into payoffs, states, or partner choice. Those routes blur the effect of the learning signal itself. The authors instead shape the learning signal directly inside a combined Q-learning rule. That clean design shows how much of the cooperation gain comes from the signal alone.

Catalog record

CREATORChunpeng Du, Zongyang Li, Yali Zhang, Yikang Lu, Attila Szolnoki
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-21392

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Shaping the learning signal in a combined Q-learning rule to improve structured cooperation Journal: Chaos, Solitons & Fractals Chunpeng Du, Zongyang Li, Yali Zhang, Yikang Lu, Attila Szolnoki Abstract Q-learning provides a standard reinforcement learning framework for studying cooperation by specifying how agents update action values from repeated local interactions outcomes. Although previous work has shown that reputation can promote cooperation in such systems, most models introduce reputation by modifying payoffs, encoding it directly in the state or changing partner selection, which makes it difficult to isolate the role of the learning signal itself. Here, we construct the reinforcement signal as a weighted combination of reputation and game payoffs, leaving the game and network structure unchanged. We find that increasing the weight on reputation generally promotes cooperation by consolidating clusters, but this effect is conditional on the learning dynamics. Specifically, this promoting effect vanishes in two regimes: when the learning rate is extremely small, which prevents effective information propagation and when the discount factor approaches one, as distant future expectations obscure the immediate reputational advantage. Outside these limiting cases, the efficacy of reputation in promoting cooperation is attenuated by higher learning rates but amplified by larger discount factors. These results advance the understanding of cooperative dynamics by demonstrating that cooperation can be stabilized through the reputational shaping of learning signals alone, providing critical insights into the interplay between social information and individual learning parameters. 1 Introduction Cooperation is a founding pillar of biological and social systems, supporting collective defense in animal groups, division of labor in eusocial species, public goods and in human societies [ 25 , 18 , 38 , 47 ] . Nevertheless, it is difficult for cooperation to emerge and persist because individuals can easily benefit from common goods without contributing to it. More precisely, contributors incur the cost, while the resulting benefits are shared independently of the contribution. As a response, the fear of potential exploitation discourages cooperators, which allows opportunistic behavior to spread [ 1 , 16 , 19 ] . These difficulties have motivated sustained studies of how cooperation emerges and persists. Evolutionary game theory provides a standard framework for studying the emergence and stability of cooperation, linking individual incentives, interaction patterns, and population-level dynamics. [ 39 , 54 , 51 , 22 ] . It specifies a payoff structure and a behavioral update rule at an individual level, allowing researchers to analyze how interactions governed by that rule produce collective population level dynamics, including steady state cooperation levels and characteristic spatial patterns [ 29 , 9 , 27 , 10 ] . A substantial body of evolutionary game theory research has identified mechanisms that promote or sustain cooperation under specified conditions [ 17 , 28 ] . Nowak summarized five classical cases: kin selection, direct reciprocity, indirect reciprocity, network reciprocity and group selection [ 26 , 53 ] . These mechanisms are widely used as benchmark explanations for the evolution of cooperation in evolutionary game theory. Among these, network reciprocity is fundamental when interaction occurs in fixed structures [ 45 , 40 ] . In particular, local interaction allows cooperators to form homogeneous clusters that reduce exposure to exploitation and stabilize cooperative domains on networks [ 2 , 21 ] . Analytical work on regular networks has established a simple selection condition, expressed as the benefit to cost ratio and the degree, which predicts when cooperation spreads under local updating [ 50 , 11 ] . With the development of complex network science, the dynamics of cooperative behavior has been examined in more realistic structures, including heterogeneous networks and interdependent networks [ 62 , 3 ] . In addition to structural effects, mechanisms such as rewards, punishments, memory and expectations have been shown to enhance group cooperation in networks by promoting network reciprocity [ 61 , 13 , 15 ] . In structured networks, interaction is local and repeated within small neighborhoods, making a record of past behavior especially consequential [ 36 , 30 ] . Reputation serves as key information at individual level that acts as a social signal summarizing recent behavior. Depending on how this information is shared and the applied update rule, it can diffuse locally and influence decisions at cooperator defector interfaces, thereby affecting the stability of cooperative domains [ 12 , 41 , 49 , 6 ] . In networks, a body of work instantiates reputation and measures its cooperative impact. In co-evolving social networks, reputation based partner choice sustains cooperation by redirecting ties toward well rated neighbors [ 12 , 46 , 37 ] . Locally available image scoring rules increase cooperation when neighbors can observe or infer the reputation state [ 49 , 23 , 4 ] . In spatial models with second-order reputation, evaluation rules that depend on both a player’s action and the recipient’s reputation result in a higher cooperation level than spatial reciprocity alone [ 52 , 7 , 67 ] . Laboratory experiments further show that gossip mediated diffusion of reputation information and the possibility of ostracism increase cooperation in group settings [ 8 , 59 ] . The aim of our work is to integrate reputation information into value based learning on networks. Some studies encoded the reputation directly in the reward, so agents update action values from returns shaped by their reputation level. These models typically find greater cooperation when the assessment rules are consistent and locally observable [ 63 , 66 , 35 ] . Other works encoded the reputation in the agent’s state, so the Q-values are updated conditional on the reputation level and reported that richer assessment rules, such as second-order reputation, further enhance cooperation on lattice structures [ 57 , 7 ] . There is also evidence that tailoring the learning signal to local interaction improves cooperation relative to imitation updates on networks [ 44 , 65 ] . Laboratory and multi-agent studies show that reputation cues, when observable and evaluated according to stable rules, are used by value-based learners to adjust their strategies and support cooperation [ 8 , 33 ] . Most of the existing combinations of the reputation and the Q-learning protocol change the payoffs in the stage game or modify partnership interactions when introducing reputation [ 58 , 64 ] . Other works encode the reputation directly in the agent’s state using custom features, a method that adds additional parameters and obscures which component produces the gains [ 32 ] . In both cases, it becomes challenging to isolate the effect of learning from the reputation information and the relative influence of the game payoff and reputation is often not on a common numerical scale [ 55 , 5 ] . In contrast, we define the reinforcement signal as a weighted combination of reputation and game payoffs, rather than using reputation to alter either the strategic payoffs or interaction partners. This design follows the standard reward shaping practice by guiding learning without changing the underlying game and keeping the role of reputation explicit [ 20 , 56 ] . In our setting, reputation and game payoff are combined in a single reinforcement signal that updates Q-values in the spatial prisoner’s dilemma. This setup gives cooperator clusters a local advantage at their boundaries

Read the complete text