Meta-Inverse Reinforcement Learning for Mean Field Games via Probabilistic Context Variables

Yang Chen, Xiao Lin, Bo Yan, Libo Zhang, Jiamou Liu, Neset Özkan Tan, Michael Witbrock · 2025

Read Meta-Inverse Reinforcement Learning for Mean Field Games via Probabilistic Context Variables by Yang Chen, Xiao Lin, Bo Yan, Libo Zhang, Jiamou Liu, Neset Özkan Tan, Michael Witbrock (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-03845.txt.

Introduction

Yang Chen and six co-authors turn to inferring reward functions for many interacting agents with different goals in a 2025 research paper. Inverse reinforcement learning works backward from expert behaviour to the rewards that would explain it. In mean field games, which model huge populations at once, existing inverse methods assume all agents share one objective, an assumption real demonstrations often break.

Their approach builds a deep latent variable model of the mean field game, with probabilistic context variables for each agent's hidden situation. The inverse method can then recover rewards from demonstrations drawn from tasks that differ yet share structure. No prior knowledge of the contexts is needed, and the game model stays unchanged. Experiments cover simulated scenarios plus a real-world pricing problem for spatial taxi rides. Across them, the method beats leading inverse methods for mean field games, handling heterogeneous, unknown objectives that defeat older tools.

Catalog record

CREATORYang Chen, Xiao Lin, Bo Yan, Libo Zhang, Jiamou Liu, Neset Özkan Tan, Michael Witbrock
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-03845

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Meta-Inverse Reinforcement Learning for Mean Field Games via Probabilistic Context Variables Yang Chen, Xiao Lin, Bo Yan, Libo Zhang, Jiamou Liu, Neset Özkan Tan, Michael Witbrock Abstract Designing suitable reward functions for numerous interacting intelligent agents is challenging in real-world applications. Inverse reinforcement learning (IRL) in mean field games (MFGs) offers a practical framework to infer reward functions from expert demonstrations. While promising, the assumption of agent homogeneity limits the capability of existing methods to handle demonstrations with heterogeneous and unknown objectives, which are common in practice. To this end, we propose a deep latent variable MFG model and an associated IRL method. Critically, our method can infer rewards from different yet structurally similar tasks without prior knowledge about underlying contexts or modifying the MFG model itself. Our experiments, conducted on simulated scenarios and a real-world spatial taxi-ride pricing problem, demonstrate the superiority of our approach over state-of-the-art IRL methods in MFGs. Introduction Understanding incentives among interacting agents in real-world decision-making and control tasks is a fundamental challenge in multi-agent systems. Inverse reinforcement learning (IRL) ( Ng, Harada, and Russell 1999 ; Ng and Russell 2000 ) addresses this issue by inferring reward functions from expert demonstrations, offering a succinct representation of tasks. IRL in multi-agent systems serves two primary purposes. First, it aids in comprehending and predicting the objectives of interacting agents, such as determining the destinations of autonomous vehicles ( You et al. 2019 ) . Second, it enables the design of agent environments with known reward signals to guide their behaviour as desired, akin to mechanism design ( Fu et al. 2021 ) . A prominent challenge with IRL is the “curse of the agent number”, i.e., an increase in agent number leads to exponential interaction complexities, resulting in impractical time and memory expenses. Fortunately, recent advancements in IRL within mean field games (MFGs) ( Yang et al. 2018a ; Chen et al. 2022 ; Chen et al. 2023 ) offer a solution. By leveraging mean-field approximation, these methods simplify interactions among many agents to only two agents (individual-population), mitigating the computational burden. These MFG-based approaches have demonstrated compelling outcomes across diverse large-scale multi-agent tasks. Applications include modelling and predicting social media population behaviour ( Yang et al. 2018a ) , product pricing in expansive markets, virus propagation modelling and explaining emerging social norms ( Chen et al. 2022 ; Chen et al. 2023 ) . While appealing, these existing methods typically rely on the theoretically powerful yet practically unideal assumption in MFGs: all agents are homogeneous, i.e., they are identical in the reward function, state-action space and dynamics. Real-world scenarios, however, frequently involve demonstrated behaviour with distinct and unknown rewards. For example, in spatial pricing for taxi rides, drivers may have different preferences based on factors such as distance, origins, and passenger destinations ( Ata, Barjesteh, and Kumar 2019 ) . To address this issue, efforts have been made to generalise mean-field approximation by introducing additional type variables for each agent to differentiate reward functions ( Subramanian and Mahajan 2019 ; Ganapathi Subramanian et al. 2020 ; Ghosh and Aggarwal 2020 ) . However, this approach requires prior knowledge of these variables, making it unsuitable for handling agents with unknown types or contexts, as observed in taxi trajectories. Additionally, incorporating types into mean-field approximation makes the model more complex, necessitating the reevaluation of important theoretical properties, such as the existence and uniqueness of an equilibrium ( Ghosh and Aggarwal 2020 ) , to ensure a well-defined corresponding IRL problem. Although such generalisations are theoretically valued, from a machine learning viewpoint, simpler models with fewer theoretical restrictions are often preferred. Given the considerations above, the question arises: can we enable IRL to handle numerous agents with unknown reward functions without altering the mean-field approximation? On the other hand, an emerging branch of IRL called meta-IRL ( Seyed Ghasemipour, Gu, and Zemel 2019 ; Yu et al. 2019 ; Xu et al. 2019 ) combines IRL and meta-learning to address similar reward structures among demonstrations from different tasks. It introduces a latent probabilistic context variable that influences the reward function and performs context-conditioned reward inference without prior knowledge of the contexts. Although currently limited only to single-agent scenarios, this approach inspires the idea of assigning probabilistic context variables (representing types) externally to a family of MFGs instead of internally to agents within an MFG. By doing so, each MFG in this family retains its original theoretical properties, enabling the development of a meta-IRL method on top of it. Figure 1: An overview of the mechanism of PEMMFIRL. The context variable is unknown to the framework. This paper implements this idea into a novel IRL framework called Probabilistic Embedding for Meta Mean Field IRL (PEMMFIRL) that answers the question we raised above. It integrates meta-IRL, mean-field approximation, and latent variable models into a unified framework, as illustrated in Fig. 1 . Our contributions are threefold: 1. We extend the notion of MFG by introducing a probabilistic contextual variable, significantly enhancing its ability to handle heterogeneous agents without introducing additional assumptions to mean-field approximation. 2. We develop PEMMFIRL, an associated IRL framework for this generalised MFG model, capable of inferring reward functions from demonstrations with different context variables. Importantly, PEMMFIRL can infer rewards from structurally similar tasks without prior knowledge about underlying contexts. 3. Experimental results on simulated tasks and a real-world spatial taxi-ride pricing problem demonstrate the effectiveness of our approach, achieving an outstanding increase in drivers’ average profit when applied to real-world passenger demand data. Related Work Our work is closely related to the literature on MFGs, initially introduced in the continuous setting by ( Huang et al. 2006 ) and ( Lasry and Lions 2007 ) . Later, MFGs were extended to the discrete model by ( Gomes, Mohr, and Souza 2010 ) , which is commonly adopted in agent learning. Recently, learning MFGs has garnered significant attention ( Cardaliaguet and Hadikhanloo 2017 ) , with existing methods relying on reinforcement learning techniques ( Yang et al. 2018b ; Guo et al. 2019 ; Subramanian and Mahajan 2019 ; Cui and Koeppl 2021 ) . Our method stands out by directly recovering the reward function from observed behaviour, eliminating the need for manual reward design. IRL was introduced by ( Ng and Russell 2000 ) for single-agent settings. Early IRL methods based on margin optimization ( Ratliff, Bagnell, and Zinkevich 2006 ) were ill-defined. To resolve this, Maximum Entropy IRL (MaxEnt IRL) was proposed by ( Ziebart et al. 2008 ; Ziebart, Bagnell, and Dey 2010 ) . However, MaxEnt IRL is limited to small, discrete problems due to its iterative nature in reward function tuning. To extend MaxEnt IRL to high-dimensional or continuous domains, ( Fu, Luo, and Levine 2018 ) introduced Adversarial IRL (AIRL). It utilises a sampling-based approximation by relating MaxEnt IRL to generative adversarial networks ( Goodfellow et al. 2014 ) , enabling effective reward tuning for complex scenarios. Some recent work extends AIRL to the multi-agent setting ( Yu, Song, and Ermon 2019 ; Fu et al. 2021 ) and the mean-field setting ( Yang et al. 2018a ; Chen et al. 2023 ) . However, they are limited to either the

Read the complete text