The Dynamics of Policy Gradient in Social Dilemmas with Partner Selection
Benedict Russell, Chin-wing Leung, Paolo Turrini · 2026
Read The Dynamics of Policy Gradient in Social Dilemmas with Partner Selection by Benedict Russell, Chin-wing Leung, Paolo Turrini (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18185.txt.
Introduction
Benedict Russell's 2026 paper gives an exact account of policy-gradient learning in social dilemmas where agents can choose partners. Simulations suggest partner choice helps cooperation emerge, but this work proves how it shifts the opponent mix and the reward landscape. Under simple literature rules, partner choice helps cooperation emerge, and spread across the population proves necessary for it to emerge at all. A two-dimensional Wiener process adds chance to partner choice, giving a sufficient condition for a cooperation-favouring population and proof of a steady distribution.
Catalog record
| CREATOR | Benedict Russell, Chin-wing Leung, Paolo Turrini |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 81448 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.18185
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Dynamics of Policy Gradient in Social Dilemmas with Partner Selection Benedict Russell, Chin-wing Leung, Paolo Turrini Abstract In social dilemmas self-interested learning agents face the choice between the societal benefit of cooperation and the immediate reward of defection. Significant evidence exists on the benefits of assortment mechanisms such as partner selection for the emergence of cooperation, but this is largely available through agent-based simulations. In this paper, we provide an analytical solution to the problem, studying the policy-gradient dynamics in a multi-agent environment with partner selection. We show how partner selection changes the opponent distribution and hence the reward landscape, and prove this promotes cooperation under simple rules known from the literature. In particular, we find that population variance is a necessary condition for cooperation to emerge. Using a two-dimensional Wiener process, we extend the dynamics to capture the stochastic effects of partner selection and the resulting opponent distribution. We derive a sufficient condition for the population to be cooperation-promoting and prove the existence of a stationary distribution. Simulations confirm that the stochastic model accurately captures the policy-gradient dynamics and clarifies how the learning rate affects the emergence of cooperation. 1 Introduction Social dilemmas capture the tension faced by self-interested agents when given the option to contribute to a common goal, or exploit the contributions of others. Ensuring that autonomous systems can work together for the greater good is currently one of the biggest challenges in the field of Artificial Intelligence. Recent research has explored the cooperation of LLMs [ 48 , 27 ] , multi-agent reinforcement learning (MARL) [ 23 ] , and evolutionary game theory [ 28 ] ; each providing unique insight into how cooperation can be sustained. In MARL, this problem is especially challenging, as agents experience a non-stationary environment induced by other learning agents [ 1 ] . Consequently, defection is a common outcome unless the agents are equipped with an additional mechanism such as behavioural signals of the opponent [ 32 , 37 ] or dynamic interactions [ 2 ] . In his seminal work, 28 identified partner selection as a key mechanism for the emergence of cooperation. This has led to extensive research on cooperation on dynamic networks [ 2 , 46 , 9 ] , as well as pairwise partner selection [ 1 , 23 ] . Researchers in this area commonly find that partner selection rules which increase interaction frequency or duration between cooperative agents are key to the emergence of cooperation. A popular mechanism, coined Out-for-Tat (OFT), enforces cooperative pairs to stay together; whilst re-wiring pairs with a defector [ 16 , 49 ] . Recent contributions have found allowing agents to learn partner selection is sufficient for cooperation to emerge [ 36 , 8 ] ; the agents adopt OFT as a mechanism to avoid exploitation. When random matching is enforced, defection takes hold of the population. Despite the empirical success, the current understanding of partner selection remains heavily reliant on agent-based simulations [ 22 , 21 ] . While these simulations provide robust evidence that cooperation can emerge, they often fail to provide a formal theoretical justification for how these selection rules reshape the underlying learning dynamics. Specifically, there is a lack of analytical work that maps policy gradient (PG) updates onto the non-stationary reward landscapes induced by shifting opponent distributions. Without this theoretical foundation, it remains unclear which population conditions are strictly necessary for cooperation to take hold, or how stochasticity in action and partner selection influences long-term stability. Contribution In this paper, we provide a formal theoretical analysis of policy gradient dynamics in social dilemmas with partner selection. Using mean-field theory, we derive the conditional partner distribution to capture how partner selection rules alter the reward structure for learning agents. We prove that partner selection promotes cooperation under well-known rules (OFT and ROFT) and identify that population variance is a necessary condition for cooperation to emerge from an initially non-cooperative state. We extend the mean dynamics to a stochastic model using a two-dimensional Wiener process to capture the randomness inherent in action selection and partner selection. We derive a sufficient condition for the population to be cooperation-promoting and prove the existence of a stationary distribution under the derived model. Simulations corroborate our theoretical findings, demonstrating that our stochastic model accurately captures the evolution of strategies and clarifies the pivotal role of the learning rate in fostering cooperative clusters. Related Literature Partner selection. In evolutionary and network models, enabling cooperators to avoid defectors alters the payoff landscape, generating more assortative interaction patterns than a well-mixed population [ 29 , 38 , 30 ] . This can take many forms, including active linking [ 29 , 31 , 2 ] , partner switching [ 10 , 24 ] , and conditional dissociation [ 16 , 17 ] . Rewiring acts as a weak form of ostracism; defectors cannot repeatedly exploit cooperators, but cooperators have continued access to each other. These results carry over to human populations, where experimental work shows that dynamic partner updates can increase cooperation [ 33 , 9 , 49 ] . Recent research in MARL has implemented these ideas into learning environments. Anastassacos et al. [ 1 ] show that partner selection can induce cooperation amongst selfish reinforcement learning agents. This has been extended to enable agents to learn partner selection rules themselves, with behavioural signals [ 23 , 21 ] and without any prior information on opponents [ 36 ] . Learning Dynamics. The complexity associated with non-stationary environments has led researchers to model learning dynamics using tools from dynamical systems and evolutionary game theory. Prior work has linked learning to coupled replicator equations [ 39 , 12 ] , selection-mutation models [ 44 , 5 ] , and mean field approximations [ 14 ] . Learning dynamics have been considered in population games, using mean-field theory to derive the dynamics under the concurrent learning protocol [ 13 , 14 ] . This has been extended to agents acting on graphs [ 7 ] , and considering stochastic interactions under the social learning protocol [ 19 ] . In a closely related work, 50 uses a pair approximation in an evolutionary setting to show the emergence of cooperation under a partner selection rule; our work differs by analysing a learning population. Policy Gradient Dynamics. In 2-player symmetric games, 42 provides the mean dynamics of policy gradient and compares against the replicator dynamics. 3 formally establishes the connection between the replicator dynamics and the soft-max policy gradient, and proves asymptotic stability for evolutionarily stable strategies (ESS) in symmetric games. This setting has been extended to analyse the stochastic effects of exploration [ 20 ] , with stochastic stability shown under self-play. Stochastic stability has also been analysed when the payoff faces aggregate shocks [ 11 ] and random perturbations [ 25 ] . Whilst there has been extensive work on symmetric games, including the stochastic effects of shocks and action selection, there has not been theoretical research which shows the role of partner selection on the behaviour of reinforcement learning (RL) agents. This adds an additional layer of complexity, where the opponent distribution will influence both the mean and stochastic dynamics. 2 Background