Inference of Intrinsic Rewards and Fairness in Multi-Agent Systems
Victor Villin, Christos Dimitrakakis · 2025
Read Inference of Intrinsic Rewards and Fairness in Multi-Agent Systems by Victor Villin, Christos Dimitrakakis (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-07650.txt.
Introduction
Judging whether someone acted selfishly or kindly requires knowing what they value, which observers rarely see. Victor Villin and Christos Dimitrakakis tackle that inference problem in a 2025 research paper. They use multi-agent inverse reinforcement learning, which works backward from behavior to the rewards behind it. Their simple model lets each agent weigh its own welfare against the welfare of others.
With new Bayesian techniques, the authors show intrinsic rewards and altruistic tendencies can be told apart. The trick is watching agents placed in different groups. Once separated, the two ingredients can be recombined to synthesize behavior at any desired level of altruism. That ability matters for autonomous agents that must cooperate with people. It could help designers build assistants whose social behavior is understood, not just imitated.
Catalog record
| CREATOR | Victor Villin, Christos Dimitrakakis |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-07650
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Inference of Intrinsic Rewards and Fairness in Multi-Agent Systems Victor Villin, Christos Dimitrakakis Abstract From altruism to antagonism, fairness plays a central role in social interactions. But can we truly understand how fair someone is, especially without explicit knowledge of their preferences? We cast this challenge as a multi-agent inverse reinforcement learning problem, explicitly structuring rewards to reflect how agents value the welfare of others. We introduce novel Bayesian strategies, reasoning about the optimality of demonstrations and characterisation of equilibria in general-sum Markov games. Our experiments, spanning randomised environments and a collaborative cooking task, reveal that coherent notions of fairness can be reliably inferred from demonstrations. Furthermore, when isolating fairness components, we obtain a disentangled understanding of agents preferences. Crucially, we unveil that by placing agents in different groups, we can force them to exhibit new facets of their reward structures, cutting through ambiguity to answer the central question: who is being fair ? 1 Introduction Fairness plays a pivotal role in decision-making in social interactions, influencing how individuals balance their own interests with the welfare of others ( rabin1992incorporating ; fehr1999theory ; reuel2024fairness ) . A simple yet illustrative example is a parent playing chess with their child: rather than playing optimally, the parent might intentionally make suboptimal moves to allow the child to experience the joy of winning or the satisfaction of learning. Psychologically, this decision reflects a value system where utility is derived not only from winning but also from nurturing the child’s growth and happiness. From a game-theoretic perspective, individuals in social interactions can be viewed as rational agents aiming to maximise their rewards through strategic decision-making ( wang2022cooperative ) . In the realm of multi-agent Reinforcement Learning (RL), reward functions are central to optimising agent policies, quantifying the utility of actions given the presence and actions of other agents, and steering agents towards specific behaviours ( eschmann2021reward ) . Within a given environment, an agent can act in a selfish, adversarial, or altruistic manner. Without knowledge of the agent’s reward function, our goal is to infer their motivation by placing them in situations where they must interact with other agents. We assume that each agent acts not only to maximise its intrinsic reward, but may wish to minimise or maximise rewards of other agents to some extent. Then the problem we need to solve is two-fold: Firstly, inferring each agent’s intrinsic reward function, and secondly how they value their own reward vis-a-vis those of the other agents. We choose to tackle this novel challenge through the lens of Multi-Agent Inverse Reinforcement Learning (MAIRL) ( natarajan2010multi ) , a promising tool for interpreting social interactions ( chen2024unveiling ) . Contributions. We propose practical algorithms for inferring the fairness of agents with unknown intrinsic rewards in a multi-agent system. Framing the task as a MAIRL problem, we model agent utilities as a linear combination of their intrinsic rewards and those of other agents. Although simultaneously identifying rewards and fairness might be hard, we show that by placing agents within diverse groups, we can obtain good estimates of the agents’ fairness attitudes. We enumerate our main contributions below: 1. We formalise the fairness inference problem as an MAIRL problem, structuring rewards to reflect how an agent values the welfare of others (Section 4 ). 2. We investigate reward identifiability in MAIRL (Section 5 ), arguing that additional assumptions on the reward structure as well as entropy-regularisation are key elements for inference. 3. In Section 6 , we present two Bayesian methods: one assumes that agents are Boltzmann-rational, while the other makes no assumptions on the policy structure ( dimitrakakis2011bayesian ) . 4. In Section 7 , we assess these methods on fairness and reward identification. We first use randomised Markov Games (MGs) as a general benchmark, before further experimenting on a collaborative cooking task ( carroll_utility_learning_about_2019 ) . Our experiments show that we can learn accurate fairness and intrinsic rewards with sufficient demonstrations and proper optimality modelling. We also show that collecting data from multiple agent groupings improves sample efficiency and reduces any remaining ambiguities about the agent’s preferences. 2 Related Work Multi-Agent Inverse Reinforcement Learning. Inverse RL (IRL) and MAIRL aim to infer reward functions that explain observed behaviours, assuming these are near-optimal ( ng2000algorithms ; natarajan2010multi ) . Defining optimality in multi-agent settings is significantly more complex than in single-agent scenarios. Although multi-agent problems can be decomposed into separate single-agent tasks ( lin2019multi ; fu2021evaluating ) , many studies emphasise the importance of equilibrium concepts ( lin2019multi ) , such as Nash equilibria, for effective reward inference ( reddy2012inverse ) . In entropy-regularised settings, equilibria are unique ( yu2019multi ; jeon2020scalable ; martin2021bayesian ; bergerson2021multi ; mehr2023maximum ; freihaut2025feasiblerewardsmultiagentinverse ; haynam2025multi ) . In our work, agents are assumed to have varying, hidden levels of fairness, rendering any problem general-sum. While MAIRL has been successfully applied in fully cooperative and zero-sum settings ( jeon2020scalable ; fu2021evaluating ; martin2021bayesian ; haynam2025multi ; yu2019multi ) , its application to general-sum games is less straightforward. Nonetheless, effective reward recovery has been demonstrated under structural assumptions on the reward functions ( reddy2012inverse ; lin2019multi ; freihaut2025feasiblerewardsmultiagentinverse ) , including on real-world traffic data ( mehr2023maximum ) . Despite this, MAIRL still faces challenges in identifying true rewards in general-sum environments, even with entropy regularisation ( freihaut2025feasiblerewardsmultiagentinverse ) . One can mitigate this issue by varying the environment in which demonstrations are provided ( cao2021identifiability ; kim2021reward ; rolland2022identifiability ; buening2024environment ; nitschke2024amber ) . Similarly, changing an agent’s policy in a multi-player game, can reveal new aspects of the other agents rewards ( buning2022interactive ) . Our approach takes inspiration from this, but instead of controlling agent policies, we deploy diverse agent combinations to disambiguate reward functions. Bayesian IRL is a framework for inferring rewards through probabilistic reasoning ( ramachandran2007bayesian ) . Despite its success in single-agent scenarios ( choi2011map ; rothkopf2011preference ; dimitrakakis2011bayesian ) , its application to MAIRL remains underexplored ( lin2017multiagent ; martin2021bayesian ) . Common drawbacks to existing MAIRL methods are strong assumption about behaviours, such as strict rationality or a specific amount of entropy regularisation ( yu2019multi ; lin2019multi ) . We propose a Bayesian modelling approach that only assumes \epsilon -optimality of the policy, which we show performs significantly better than approaches which do make behavioural assumptions. Fairness in MARL. Fairness is a natural concern in economics ( fairnessecon86 ) and multi-agent systems ( de2008fairness ; hosseini2024fairness ) , and is typically enforced by encouraging agents to share welfare equitably or by applying constraints ( prosocial2018peysakhovich ; agapiou_melting_pot_2_2023 ; aloor2024cooperation ; siddique2024towards ) . The topic of inferring fairness has been studied in behavioural economics. ( charness2002understanding ) showed that subjects are concerned with increasing payoffs for everybody, based on a linear utility model, while 10.3389/fnhum.2013.00252 ; guth1982experimental proposed utility-based fairness concepts such as envy , altruism and inequity aversion , as well as behavioural concepts such as reciprocity . However, this work is generally limited to surveys and stylised games (cf. guth1982experimental ). In MAIRL, some work considered the related topic of a theory of mind about other agents ( GToM:Yoshida:2008 ; chen2024unveiling ) . However, the specific setting and techniques presented here