Decoding fairness: a reinforcement learning perspective
Guozhong Zheng, Jiqiang Zhang, Xin Ou, Shengfeng Deng, Li Chen · 2024
Read Decoding fairness: a reinforcement learning perspective by Guozhong Zheng, Jiqiang Zhang, Xin Ou, Shengfeng Deng, Li Chen (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-16249.txt.
Introduction
Guozhong Zheng and co-authors asked in 2024 why people act fairly when they split money, even though classic theory says they should grab what they can. In lab games, one player offers a split and the other can accept or reject it, leaving both with nothing.
The team let computer players learn from rewards, with each player keeping score for both roles in the game. Fair offers grew on their own when players valued past play and future gains. Deals were more likely to succeed as offers rose, matching lab results with people. The pattern held under other role setups and in a grid of players. Fairness, they argue, needs no outside cause.
Catalog record
| CREATOR | Guozhong Zheng, Jiqiang Zhang, Xin Ou, Shengfeng Deng, Li Chen |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-16249
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Decoding fairness: a reinforcement learning perspective Guozhong Zheng, Jiqiang Zhang, Xin Ou, Shengfeng Deng, Li Chen Abstract Behavioral experiments on the ultimatum game (UG) reveal that we humans prefer fair acts, which contradicts the prediction made in orthodox Economics. Existing explanations, however, are mostly attributed to exogenous factors within the imitation learning framework. Here, we adopt the reinforcement learning paradigm, where individuals make their moves aiming to maximize their accumulated rewards. Specifically, we apply Q-learning to UG, where each player is assigned two Q-tables to guide decisions for the roles of proposer and responder. In a two-player scenario, fairness emerges prominently when both experiences and future rewards are appreciated. In particular, the probability of successful deals increases with higher offers, which aligns with observations in behavioral experiments. Our mechanism analysis reveals that the system undergoes two phases, eventually stabilizing into fair or rational strategies. These results are robust when the rotating role assignment is replaced by a random or fixed manner, or the scenario is extended to a latticed population. Our findings thus conclude that the endogenous factor is sufficient to explain the emergence of fairness, exogenous factors are not needed. I Introduction Fairness is fundamental to the sustainable development of human beings. It fosters social cohesion and well-being by ensuring equitable distribution of resources and opportunities. However, the rising social inequality, manifested through unrest, violence, and tensions between socioeconomic groups, threatens these foundations [ 1 , 2 ] . Therefore, it is crucial to decipher the mechanisms behind the emergence of fairness to achieve and sustain a decent level of fairness [ 3 ] . Evolutionary game theory provides a valuable framework for addressing this issue, and the ultimatum game (UG) [ 4 , 5 ] is the most widely adopted model for understanding fairness. In this game, two players – one proposer and one responder – decide how to divide a fixed amount of money. The proposer makes an offer, and the responder can either accept or reject it. If accepted, the money is distributed as proposed; if rejected, neither player receives anything. With the Homo economicus [ 6 , 7 ] assumption made in orthodox Economics, individuals are fully rational and self-interested; the responder will accept any nonzero offers since “something is better than nothing”, and the proposer will then propose the least possible offer to maximize the remaining proportion for its own. As such, an extremely unfair division is expected. However, abundant behavioral experiments with UG show that most offers fall between 40\% and 50\% of the total, and offers below 20\% are often rejected [ 4 , 8 , 9 , 10 , 5 ] . This indicates that humans strongly prefer fair acts, challenging the prediction made in orthodox Economics. To address this discrepancy, many theoretical studies have explored the mechanisms behind the emergence of fairness [ 11 ] . Research has revealed that the underlying structure of the population [ 12 , 13 , 14 ] , role assignment [ 15 , 16 ] , and noise [ 17 ] significantly contribute to the emergence of fairness. Human factors like reputation [ 18 , 19 , 20 , 20 , 21 ] , spite [ 22 ] , and empathy [ 23 , 24 , 25 ] are also found to be crucial for the evolution of fairness. Interestingly, the presence of a few “good Samaritans” can boost fairness remarkably via a discontinuous phase transition [ 26 ] and may serve as a control strategy [ 27 ] . Notice that these studies typically employ the imitation learning framework [ 28 , 29 ] , where individuals tend to imitate the strategies of neighbors who have higher payoffs. Imitation can be viewed as a simple form of social learning [ 30 ] , where people learn from others in socioeconomic activities by observations or instructions. In contrast, reinforcement learning (RL) [ 31 ] offers a fundamentally distinct paradigm where individuals optimize their strategies through interaction with their environment. Unlike social learning, where decisions are made by observing and imitating peers, players with RL resort to self-reflection by evaluating their actions within different surroundings. This allows them to evolve unique and potentially heterogeneous strategies to better adapt to their surroundings rather than following uniform decision-making rules assumed in social learning. More importantly, individuals with RL aim to maximize the accumulated payoffs instead of immediate rewards as in social learning. Growing evidence shows that RL has distinct working areas and physiological processes in the brain [ 32 , 33 , 34 ] , manifesting itself as a fundamentally different learning paradigm compared to social learning. Only recently, RL started to be utilized to study emergent behaviors such as cooperation [ 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 ] , trust [ 50 ] , resource allocation [ 51 , 52 , 53 ] , collective motion [ 54 , 55 ] , and other collective behaviors for humans [ 56 , 47 ] . For instance, Ding et al. [ 44 ] revealed that individuals with historical experience and expected payoffs learn a “win-stay, lose-shift” strategy to sustain decent cooperation in a two-agent scenario. Zheng et al. [ 50 ] found that trust and trustworthiness arise when individuals appreciate past experiences and future returns. Given its great potential for deciphering many puzzles of human behaviors, it’s intriguing to ask: Does the self-reflective RL learning paradigm provide a fresh perspective on the emergence of fairness, specifically whether the fair acts of people can be explained by endogenous motivation in humans? This fundamentally differs from the current explanations attributed to the exogenous factors in the social learning paradigm. In this study, we investigate the evolution of fairness within the RL paradigm. Specifically, we apply the Q-learning algorithm [ 57 , 58 ] to study the UG, where each individual is associated with two Q-tables: one for the role of a proposer and the other for the responder role. In a two-player scenario, fairness emerges significantly when individuals appreciate both past experiences and future rewards. Particularly, we find that the dependence of a successful deal on the offer is consistent with the observations made in behavioral experiments. Our mechanism analysis reveals a two-phase transition where only strategies of successful deals survive, and ultimately, their preferences evolve mainly to fair strategy together with a small amount of rational type. This paper is organized as follows: Sec. II introduces the Q-learning framework for the UG. Sec. III presents the results for the two-player scenario, while Sec. IV provides a detailed mechanistic analysis. Sec. V discusses the failure of emergence. Sec. VI presents the robustness check and the extension to a latticed population. Finally, Sec. VII concludes with a summary and discussion of our findings. II Model Let’s consider a two-player UG scenario. As a rule of thumb, the total amount of money to be divided in each round is set to be 1. The strategy of each player i is defined by S_{i}=(p_{i},q_{i}) with p,q\in[0,1] , where p_{i} is the offer when acting as a proposer and q_{i}