Recurrent Structural Policy Gradient for Partially Observable Mean Field Games Clarisse Wibault, Johannes Forkel, Sebastian Towers, Tiphaine Wibault, Juan Duque, George Whittle, Andreas Schaab, Yucheng Yang, Chiyuan Wang, Michael Osborne, Benjamin Moll, Jakob Foerster Abstract Mean Field Games (MFGs) provide a principled framework for modeling interactions in large population models: at scale, population dynamics become deterministic, with uncertainty entering only through aggregate shocks— or common noise . However, algorithmic progress has been limited since model-free methods are too high variance and exact methods scale poorly. Recent Hybrid Structural Methods (HSMs) use Monte Carlo rollouts for the common noise in combination with exact estimation of the expected return, conditioned on those samples. However, HSMs have not been scaled to Partially Observable settings. We propose Recurrent Structural Policy Gradient (RSPG), the first history-aware HSM for settings involving shared aggregate observations. We also introduce MFAX, our JAX-based framework for MFGs. By leveraging known transition dynamics, RSPG achieves state-of-the-art performance as well as an order-of-magnitude faster convergence and solves, for the first time, a macroeconomics MFG with heterogeneous agents, common noise and history-aware policies. MFAX is publicly available at: https://github.com/CWibault/mfax . † † affiliationnotice: † Equal supervision 1 Introduction Training policies in large multi-agent systems is notoriously difficult: RL-based methods rely on high-variance, trajectory-based sampling and therefore scale poorly as the number of agents grows ( 8 ; 25 ; 9 ) . But, in many large population systems, such as financial markets, traffic control and communication networks ( 6 ; 41 ; 4 ) , individuals only respond to the aggregate behaviour of other agents. As a motivating example, an agent might care about the price of a stock, which is determined by the rest of the population, but not whether a certain individual has decided to buy or sell that stock. Moreover, in such large population systems, individual randomness or idiosyncratic noise marginalises out at the macroscopic level. At such a scale, the system’s evolution is well defined by deterministic population dynamics, with uncertainty entering only through common noise — aggregate shocks that affect the entire population simultaneously. Hence, assuming all agents have the same state-conditioned objective, reward, transition and observation models, the analysis of a large population game can be reduced to the interaction between a stand-in agent with policy \pi and the population distribution , or mean-field \mu_{t} , over all individual states \mathcal{S} . The Markov state of the system is the aggregate state (\mu_{t},z_{t}) , where z_{t} represents the common noise . Such Mean-Field Games (MFGs) ( 21 ; 15 ) have found applications in numerous areas such as epidemic control, communications, energy networks, finance, and macroeconomics, ( 10 ; 1 ; 40 ; 5 ; 26 ) . Unlike fully sample-based Reinforcement Learning (RL), Hybrid Structural Methods (HSMs) ( 13 ; 42 ) leverage the fact that individual transition dynamics \mathcal{T}(s_{t+1}\mid s_{t},a_{t},\mu_{t},z_{t}) are known, either by assumption or inferred at the population level, and that the individual state space is computationally tractable in many applications. The benefits are twofold. Firstly, this allows computing the analytic mean-field update — the mean-field update found by marginalising individual transition dynamics. Secondly, it allows policy optimisation to use exact expectations over individual dynamics, substantially reducing variance as now only the common noise is sampled. However, these HSMs, like RL based MFG algorithms, have been limited to fully-observable settings ( 7 ; 29 ; 3 ; 39 ; 42 ) . In many real-world situations involving public information, such as market prices in economics, agents only receive shared observations o_{t} of the aggregate state. We begin by introducing the more general Partially Observable Mean-Field Games with Common Noise . We then show how, by restricting memory to a history of shared aggregate observations, exact expectations can remain tractable, benefiting from the associated variance reduction, while still enabling history-dependent behaviour to be learned. To evaluate our resulting algorithm Recurrent Structural Policy Gradient (RSPG) against both HSM and RL based MFG algorithms, we introduce MFAX, our JAX-based MFG framework. MFAX makes three main improvements over current open-source MFG libraries. First, unlike current libraries, MFAX distinguishes between problem settings with white-box access to individual transition dynamics and those with only sample-based access. Second, MFAX accelerates analytic mean-field updates by using a functional representation of the update operator and leveraging GPU parallelism across both states and actions. For example, a single mean-field update of Linear Quadratic —a standard MFG environment— is respectively 10 and 1000 times faster than OpenSpiel and MFGLib 18 ; 11 . Being fully JAX-based, MFAX also enables parallel execution across environments, which we leverage in the HSM and RL algorithm implementations released alongside our framework. Third, MFAX supports more complex environments involving partial-observability, common-noise and multiple initial mean-field distributions; these features are missing from existing frameworks. We summarise our contributions as follows: 1. We provide a unified taxonomy of DP, RL and HSMs for MFGs that is missing in MFG literature. 2. We introduce Partially Observable Mean Field Games with Common Noise . 3. We identify the special case of shared observations of the aggregate state, and the policies’ memory being restricted to the history of shared observations, which makes things computationally tractable. We propose Recurrent Structural Policy Gradient (RSPG), the first history-aware HSM. 4. We introduce MFAX, our JAX-based MFG framework that, unlike existing libraries, also provides white-box access to individual transition dynamics, faster mean-field updates, and support for more complex environments. 5. Empirically, we show that RSPG achieves state-of-the-art performance and an order-of-magnitude faster convergence (in wall-clock time) than RL based methods, while history-awareness enables it to capture more realistic agent behaviour. We solve a partially observable version of a macroeconomics MFG with common noise for the first time ( 17 ) . 2 Preliminaries 2.1 Mean Field Games (MFGs) with Common Noise Definition 2.1 . ( 39 ) An infinite-horizon MFG with common noise is defined by a tuple, \displaystyle(p_{\mu_{0}},p_{z_{0}},\mathcal{S},\mathcal{A},\mathcal{Z},\mathcal{T},\Xi,R,\gamma). where \mathcal{S} and \mathcal{A} are the state and action spaces of the individual agents, which are both finite, and \mathcal{Z} is the space of values of the common noise, which is also finite. The initial common noise z_{0} is drawn from the distribution p_{z_{0}} , and the noise evolves according to z_{t+1}\sim\Xi(\cdot\mid z_{t}) . \mu_{t}\in\Delta_{\mathcal{S}} denotes the mean-field at time t , and \mu_{0}\sim p_{z_{0}} . \mathcal{T} denotes the transition function of the individual agents, i.e. s_{t+1}\sim\mathcal{T}(\cdot|s_{t},a_{t},\mu_{t},z_{t}) . The mean-field evolves according to the analytic mean-field update operator \Phi^{\pi} : \displaystyle\mu_{t+1}(s_{t+1}) \displaystyle=\Phi^{\pi}(\mu_{t},z_{t})(s_{t+1}):=\iint\mathcal{T}(s_{t+1}\mid s_{t},a_{t},\mu_{t},z_{t})\pi(a_{t}\mid s_{t},\mu_{t},z_{t})\mu_{t}(s_{t})\mathrm{d}a_{t}\mathrm{d}s_{t}, (1) where \pi(\cdot\mid s_{t},\mu_{t},z_{t}) denotes the action distribution of an agent conditioned on the individual state s_{t} and aggregate state (\mu_{t},z_{t}) . The reward r_{t}=R(s_{t},a_{t},\mu_{t},z_{t}) is a scalar function of the individual state s_{t} , the action a_{t} and the aggregate state (\mu_{t},z_{t}) . Since \mathcal{S} and \mathcal{A} are finite, ( 1 ) can be rewritten in vector-matrix form, \displaystyle\boldsymbol{\mu}_{t+1}={\mathbf{A}^{\pi}_{\mu_{t},z_{t}}}^{\top}\boldsymbol{\mu}_{t}, (2) where \mathbf{A}^{\pi}_{\mu_{t},z_{t}} is a matrix whose (s_{t},s_{t+1})^{\text{th}} entry is \mathbb{E}_{a_{t}\sim\pi(\cdot\mid s_{t},\mu_{t},z_{t})}\left[\mathcal{T}(s_{t+1}\mid s_{t},a_{t},\mu_{t},z_{t})\right] and \boldsymbol{\mu}_{t} is a vector of length \lvert\mathcal{S}\rvert such that \sum_{k=1}^{\lvert\mathcal{S}\rvert}\boldsymbol{\mu}_{t}(k)=1 . We let \text{M}_{\text{evol}}(\pi,z_{0:t}) denote the mean-field sequence \mu_{0:t}=(\mu_{0},...,\mu_{t}) which is generated from \pi and z_{0:t} 1 1 1 We assume w.l.o.g. that \mu_{0} is a deterministic function of z_{0} . This is notationally convenient as then \mu_{0:\infty} is a deterministic function of z_{0:\infty} through ( 1 ), and define \displaystyle J_{\text{evol}}(\pi,\pi^{\prime})= \displaystyle\mathbb{E}_{\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}R(s_{t},a_{t},\text{M}_{\text{evol}}(\pi^{\prime},z_{0:t})_{t},z_{t})\right], (3) where the expectation is taken with respect to z_{0}\sim p_{z_{0}} , z_{t+1}\sim\Xi(\cdot|z_{t}) , \mu_{t}=\text{M}_{\text{evol}}(\pi^{\prime},z_{0:t})_{t} , s_{0}\sim\mu_{0} , a_{t}\sim\pi(\cdot\mid s_{t},\mu_{t},z_{t}),s_{t+1}\sim\mathcal{T}(\cdot\mid s_{t},a_{t},\mu_{t},z_{t}) . We distinguish between the mean-field control problem and the mean-field equilibrium problem — or mean-field game (MFG). In the control problem, we condition on exogenous mean-field behaviour induced by a policy \pi^{\prime} , and try to learn \pi\in\text{argmax}_{\pi}J_{\text{evol}}(\pi,\pi^{\prime}) . Thus the control problem reduces to finding an optimal policy in a non-stationary MDP. In the equilibrium problem, the mean-field behaviour is endogenous and generated by the policy \pi itself. We aim to find a mean-field Nash equilibrium, at which a policy \pi^{*} is the best response to the population behaviour it induces, i.e.: Definition 2.2 . ( 39 ) A policy \pi^{*} (and its induced population behaviour) constitute a mean-field Nash equilibrium, if \pi^{*}\in\text{argmax}_{\pi}J_{\text{evol}}(\pi,\pi^{*}) . Solving MFGs generally involves a two-step process alternating between calculating the population behaviour of the current policy, and then letting the updated policy be the best response to that population behaviour. 3 Unified Taxonomy of DP, RL & HSMs for MFGs Information Box 1. Dynamic Programming based methods assume access to both the individual state dynamics, i.e. \mathcal{T}(s_{t+1}\mid s_{t},a_{t},\mu_{t},z_{t}) and the aggregate state dynamics, i.e. \Xi(z_{t+1}\mid z_{t}) and \displaystyle{\boldsymbol{\mu}_{t+1}=\Phi^{\pi}(\boldsymbol{\mu}_{t},z_{t})={{\color[rgb]{0,0.45,0}\mathbf{A}}^{\pi}_{\mu_{t},z_{t}}}^{\top}\boldsymbol{\mu}_{t}}. Moreover, DP based methods could theoretically compute the exact value functions as expectation over next states s_{t+1} , actions a_{t} , and common noise z_{t} : \displaystyle V_{\pi,\pi}(s_{t^{\prime}}=s) \displaystyle={\color[rgb]{0,0.45,0}\mathbb{E}_{\pi}}\left[\sum_{t=t^{\prime}}^{\infty}\gamma^{t-t^{\prime}}R(s_{t},a_{t},\boldsymbol{\mu}_{t},z_{t})\;\Big|\;s_{t^{\prime}}=s\right], (4) where the expectation is taken over z_{0}\sim p_{z_{0}},z_{t+1}\sim\Xi(\cdot\mid z_{t}),\boldsymbol{\mu}_{0:\infty}=\text{M}_{\text{Evol}}(\pi,z_{0:\infty}),a_{t}\sim\pi(\cdot\mid s_{t},\boldsymbol{\mu}_{t},z_{t}),s_{t+1}\sim\mathcal{T}(\cdot\mid s_{t},a_{t},\boldsymbol{\mu}_{t},z_{t}) . When \textbf{v}_{\pi,\pi} is an \lvert\mathcal{S}\rvert -length vector with the values V(s_{t^{\prime}}=s) of each individual state s\in\mathcal{S} , ( 4 ) becomes, \displaystyle\mathbf{v}_{\pi,\pi,t^{\prime}} \displaystyle={\color[rgb]{0,0.45,0}\mathbb{E}_{\boldsymbol{\mu}_{0}\sim p_{\boldsymbol{\mu}_{0}},z_{0}\sim p_{z_{0}},z_{t+1}\sim\Xi(\cdot\mid z_{t})}}\left[{\color[rgb]{0,0.45,0}\mathbf{\tilde{r}}}^{\pi}_{\boldsymbol{\mu}_{t^{\prime}},z_{t^{\prime}}}+\gamma{\color[rgb]{0,0.45,0}\mathbf{A}}^{\pi}_{\boldsymbol{\mu}_{t^{\prime}},z_{t^{\prime}}}{\color[rgb]{0,0.45,0}\mathbf{\tilde{r}}}^{\pi}_{{\boldsymbol{\mu}}_{t^{\prime}+1},z_{t^{\prime}+1}}+\gamma^{2}{\color[rgb]{0,0.45,0}\mathbf{A}}^{\pi}_{\boldsymbol{\mu}_{t^{\prime}},z_{t^{\prime}}}{\color[rgb]{0,0.45,0}\mathbf{A}}^{\pi}_{\boldsymbol{\mu}_{t^{\prime}+1},z_{t^{\prime}+1}}{\color[rgb]{0,0.45,0}{\color[rgb]{0,0.45,0}\mathbf{\tilde{r}}}}^{\pi}_{{\boldsymbol{\mu}}_{t^{\prime}+2},z_{t^{\prime}+2}}+\cdots\right], (5) where {\color[rgb]{0,0.45,0}\mathbf{\tilde{r}}}_{\boldsymbol{\mu}_{t},z_{t}}:=\left[{\color[rgb]{0,0.45,0}\boldsymbol{\Pi}}_{\boldsymbol{\mu}_{t},z_{t}}\odot\mathbf{R}_{\boldsymbol{\mu}_{t},z_{t}}\right]\mathbf{1} , where \boldsymbol{\Pi}_{\boldsymbol{\mu}_{t},z_{t}} and \mathbf{R}_{\boldsymbol{\mu}_{t},z_{t}} are |\mathcal{S}|\times|\mathcal{A}| matrices whose (s,a)^{\text{th}} elements are \pi(a\mid s,\boldsymbol{\mu}_{t},z_{t}) and R(s,a,\boldsymbol{\mu}_{t},z_{t}) respectively. Note that premultiplying by {\color[rgb]{0,0.45,0}\mathbf{A}}^{\pi}_{\boldsymbol{\mu}_{t},z_{t}} computes the expectation over next states. The value function V satisfies the infinite-dimensional “Master-Equation” in which the distribution \boldsymbol{\mu} is a state variable: V(s,\boldsymbol{\mu},z)={\color[rgb]{0,0.45,0}\mathbb{E}_{a\sim\pi(\cdot|s,\boldsymbol{\mu},z)}}\left[R(s,a,\boldsymbol{\mu},z)+\gamma{\color[rgb]{0,0.45,0}\mathbb{E}_{s^{\prime}\sim\mathcal{T}(\cdot\mid s,a,\boldsymbol{\mu},z),z^{\prime}\sim\Xi(\cdot|z)}}\left[V(s^{\prime},\Phi^{\pi}(\boldsymbol{\mu},z),z^{\prime})\right]\right] (6) Information Box 2. Reinforcement Learning based methods assume no access to transition dynamics, instead sampling next states, actions and common noise. Without access to individual dynamics, the analytic mean-field update operator cannot be applied, so the mean-field distribution is repeatedly re-approximated e.g. using j=1,..,M sample-based mean-fields: \displaystyle\boldsymbol{\mu}^{{\color[rgb]{1,0,0}{(j)}}}_{t}(s)\approx\frac{1}{N}\sum_{i=1}^{N}\delta(s-s^{{\color[rgb]{1,0,0}(i,j)}}_{t}). (7) The value V_{\pi,\pi} of a state is approximated using Monte Carlo rollouts: \displaystyle V_{\pi,\pi}(s^{{\color[rgb]{1,0,0}(i,j)}}_{t^{\prime}}) \displaystyle\approx\Bigg[\sum_{t=t^{\prime}}^{\infty}\gamma^{t-t^{\prime}}R(s^{{\color[rgb]{1,0,0}{{(i,j)}}}}_{t},a^{{\color[rgb]{1,0,0}{{(i,j)}}}}_{t},\boldsymbol{\mu}^{{\color[rgb]{1,0,0}{(j)}}}_{t},z^{{\color[rgb]{1,0,0}{(j)}}}_{t})\;\Big|\;s^{{\color[rgb]{1,0,0}(i,j)}}_{t^{\prime}}\Bigg] (8) where z_{0}^{{\color[rgb]{1,0,0}{(j)}}}\sim p_{z_{0}} , \boldsymbol{\mu}^{{\color[rgb]{1,0,0}{(j)}}}_{0}\sim p_{\boldsymbol{\mu}_{0}},s^{{\color[rgb]{1,0,0}(i,j)}}_{0}\sim\boldsymbol{\mu}_{0},a^{{\color[rgb]{1,0,0}(i,j)}}_{t}\sim\pi(\cdot\mid s_{t}^{{\color[rgb]{1,0,0}(i,j)}},\boldsymbol{\mu}^{{\color[rgb]{1,0,0}{(j)}}}_{t},z^{{\color[rgb]{1,0,0}{(j)}}}_{t}),s^{{\color[rgb]{1,0,0}(i,j)}}_{t+1}\sim\mathcal{T}(\cdot\mid s^{{\color[rgb]{1,0,0}(i,j)}}_{t},a^{{\color[rgb]{1,0,0}(i,j)}}_{t},\boldsymbol{\mu}^{{\color[rgb]{1,0,0}{(j)}}}_{t},z^{{\color[rgb]{1,0,0}{(j)}}}_{t}),z^{{\color[rgb]{1,0,0}{(j)}}}_{t+1}\sim\Xi(\cdot\mid z^{{\color[rgb]{1,0,0}{(j)}}}_{t}) and the mean-field is repeatedly re-approximated. Information Box 3. Hybrid Structural Methods leverage known individual dynamics \mathcal{T}(s_{t+1}\mid s_{t},a_{t},\mu_{t},z_{t}) to compute the expectation over next states and actions, but, unlike DP they sample common noise. Access to individual transitions means that the mean-field can be updated using the analytic mean-field update operator: \displaystyle{\boldsymbol{\boldsymbol{\mu}}^{{\color[rgb]{1,0,0}(j)}}_{t+1}=\Phi^{\pi}(\boldsymbol{\mu}^{{\color[rgb]{1,0,0}(j)}}_{t},z^{{\color[rgb]{1,0,0}(j)}}_{t})={{\color[rgb]{0,0.45,0}\mathbf{A}}^{\pi}_{\boldsymbol{\mu}^{{\color[rgb]{1,0,0}(j)}}_{t},z^{{\color[rgb]{1,0,0}(j)}}_{t}}}^{\top}\boldsymbol{\mu}^{{\color[rgb]{1,0,0}(j)}}_{t}}. The state values are approximated as: \displaystyle\mathbf{v}_{\pi,\pi,t^{\prime}} \displaystyle\approx\left[{\color[rgb]{0,0.45,0}\mathbf{\tilde{r}}}^{\pi}_{\boldsymbol{\mu}^{{\color[rgb]{1,0,0}(j)}}_{t^{\prime}},z^{{\color[rgb]{1,0,0}(j)}}_{t^{\prime}}}+\gamma{\color[rgb]{0,0.45,0}\mathbf{A}}^{\pi}_{\boldsymbol{\mu}^{{\color[rgb]{1,0,0}(j)}}_{t^{\prime}},z^{{\color[rgb]{1,0,0}(j)}}_{t^{\prime}}}{\color[rgb]{0,0.45,0}\mathbf{\tilde{r}}}^{\pi}_{\boldsymbol{\mu}^{{\color[rgb]{1,0,0}(j)}}_{t^{\prime}+1},z^{{\color[rgb]{1,0,0}(j)}}_{t^{\prime}+1}}+\gamma^{2}{\color[rgb]{0,0.45,0}\mathbf{A}}^{\pi}_{\boldsymbol{\mu}^{{\color[rgb]{1,0,0}(j)}}_{t^{\prime}},z^{{\color[rgb]{1,0,0}(j)}}_{t^{\prime}}}{\color[rgb]{0,0.45,0}\mathbf{A}}^{\pi}_{\boldsymbol{\mu}^{{\color[rgb]{1,0,0}(j)}}_{t^{\prime}+1},z^{{\color[rgb]{1,0,0}(j)}}_{t^{\prime}+1}}{\color[rgb]{0,0.45,0}\mathbf{\tilde{r}}}^{\pi}_{\boldsymbol{\mu}^{{\color[rgb]{1,0,0}(j)}}_{t^{\prime}+2},z^{{\color[rgb]{1,0,0}(j)}}_{t^{\prime}+2}}+\cdots\right] (9) where z^{{\color[rgb]{1,0,0}{(j)}}}_{0}\sim p_{z_{0}},\boldsymbol{\mu}^{{\color[rgb]{1,0,0}(j)}}_{0}\sim p_{\boldsymbol{\mu}_{0}},z^{{\color[rgb]{1,0,0}(j)}}_{t+1}\sim\Xi(\cdot\mid z^{{\color[rgb]{1,0,0}(j)}}_{t}),\boldsymbol{\mu}^{{\color[rgb]{1,0,0}(j)}}_{t+1}=\Phi^{\pi}(\boldsymbol{\mu}^{{\color[rgb]{1,0,0}(j)}}_{t},z^{{\color[rgb]{1,0,0}(j)}}_{t}) . As before, premultiplying by {\color[rgb]{0,0.45,0}\mathbf{A}}^{\pi}_{\boldsymbol{\mu}_{t},z_{t}} computes the expectation over next states and {\color[rgb]{0,0.45,0}\mathbf{\tilde{r}}}_{\boldsymbol{\mu}^{{\color[rgb]{1,0,0}{(j)}}}_{t},z^{{\color[rgb]{1,0,0}{(j)}}}_{t}}:=\left[{\color[rgb]{0,0.45,0}\boldsymbol{\Pi}}_{\boldsymbol{\mu}^{{\color[rgb]{1,0,0}{(j)}}}_{t},z^{{\color[rgb]{1,0,0}{(j)}}}_{t}}\odot\mathbf{R}_{\boldsymbol{\mu}^{{\color[rgb]{1,0,0}{(j)}}}_{t},z^{{\color[rgb]{1,0,0}{(j)}}}_{t}}\right]\mathbf{1} . Dynamic Programming (DP) based methods assume access to both the individual state transition dynamics \mathcal{T}(s_{t+1}\mid s_{t},a_{t},\mu_{t},z_{t}) , and the aggregate state transition dynamics \Xi(z_{t+1}\mid z_{t}) , \Phi^{\pi}(\mu_{t},z_{t}) (Information Box 1). By integrating over both the individual and aggregate states, the individual-state value function ( 4 ) eliminates sample variance ( 5 ) . In practice, however, DP becomes intractable because, as demonstrated by the “Master equation” ( 6 ), the resulting Bellman equation is set on (and hence requires integrating over) the space of all possible mean-field distributions induced by common-noise sequences. Reinforcement Learning (RL) based methods treat the transition model of both the individual and aggregate states as black-boxes (Information Box 2). Hence, RL based methods rely on Monte Carlo rollouts ( 8 ). Without access to individual dynamics, the analytic mean-field update operator cannot be applied and the mean-field distribution is repeatedly re-approximated using sample-based methods ( 7 ) or by learning a function \mu_{t+1}\approx\hat{\Phi}_{\theta}(\mu_{t},z_{t}) (where \theta represents the parameters of that function) that can approximate the analytic mean-field update operator ( 30 ; 16 ) , see Figure 1 . Since RL-based methods do not require the analytic mean-field update operator to be tractable, they can scale to larger individual state spaces. Hybrid Structural Methods (HSMs) are an intermediary between DP and RL based methods by leveraging access to known individual dynamics \mathcal{T}(s_{t+1}\mid s_{t},a_{t},\mu_{t},z_{t}) (Information Box 3) to achieve lower-variance updates than fully sample-based RL ( 9 ), but using Monte Carlo rollouts of the common-noise and its associated mean-field. Figure 1 : Top left : the analytic mean-field update computes the exact expectation over next states. Bottom left : a sample-based mean-field update re-approximates the mean-field at each step, by tracking individual agents. Right : network architecture for the reduced policy used in RSPG. The hidden state is independent of the individual state such that the analytic mean-field update has the same asymptotic computational cost as a memoryless policy. If the actions are continuous, an underlying continuous distribution is parameterised. The log-probabilities are evaluated at equal intervals along the action space, and used as logits for a categorical distribution. This structured prior induces ordinality in the action space. 4 Related Work 4.1 Problem Setting 43 study independent learning in a partially observable n -player MFG and prove almost-sure convergence of naive-single agent learning iterates under mild assumptions. By assuming that the observation of the aggregate state is independent of the individual state, their “Compressed Observability” setting is less general than our partially-observable problem formulation presented in Section 5 . 34 consider a more general observation structure by only specifying an observation kernel that depends on the individual state. Notably, neither work includes common noise in the problem formulation. 4.2 Algorithms Convergence has been established for several classes of MFG algorithms, but the underlying assumptions, such as monotonicity ( 14 ; 31 ; 28 ) or continuous reward and transition functions ( 7 ) , often fail to hold in practice, particularly in settings with threshold-based dynamics or multiple Nash equilibria. Among DP for MFGs, 31 compute best responses via backward induction in the presence of common noise. However, since policies are conditioned on the common-noise sequence, the algorithm requires enumerating all realisations, significantly limiting the scalability of the approach. By contrast, 13 and 42 , who study macroeconomic MFGs, propose HSMs that leverage known individual dynamics and accommodate continuous noise realisations; the latter introduce a Structural Policy Gradient (SPG) method, but their algorithm is limited to memoryless, tabular policies. RSPG is the first HSM to learn history-aware policies with continuous observations of the mean-field. Most RL algorithms assume full observability y_{t}=(s_{t},o_{t})=(s_{t},\mu_{t},z_{t}) ( 39 ; 30 ) or local observability y_{t}=s_{t} ( 32 ; 2 ; 14 ; 7 ) . (Note that local observability is sufficient for deterministic initial mean-field distributions and in the absence of common noise ( 22 ) ). 36 consider partial observability by only allowing observation of neighboring agents, and, although not formally presented as a partially observable setting, 3 consider a scenario where agents form a local estimate of the global mean-field. Since neither implement recurrent or history-dependent policies, agents are memoryless. Only few RL methods consider problem settings with common noise. Those that do ( 39 ; 3 ) are based on Q-Functions, which extend less naturally to continuous action spaces than policy-based algorithms. 2 and 12 use policy-based algorithms, but applications are restricted to fully-observable environments with no common noise. 5 Partially Observable Mean Field Games with Common Noise We formalise Partially Observable Mean Field Games with Common Noise (POMFGs-CN) as decision making under uncertainty in large population models, where agents receive only partial information about the aggregate state (\mu_{t},z_{t}) of the system, which, as previously defined, includes both the mean-field and external shocks that affect the entire population simultaneously. We note that in POMFGs-CN, the aggregate state (\mu_{t},z_{t}) is not in general Markovian, but for consistent naming, we still refer to it as state. A POMFG-CN is defined by the following tuple: \displaystyle(p_{\mu_{0}},p_{z_{0}},\mathcal{S},\mathcal{A},\mathcal{Z},\mathcal{O},\mathcal{U},\mathcal{T},\Xi,R,\gamma), where p_{\mu_{0}},p_{z_{0}},\mathcal{S},\mathcal{Z},\mathcal{A},\mathcal{T},\Xi,R,\gamma are the same as in MFGs with common noise as defined in Section 2.1 . While in MFGs with common noise the individual agents have knowledge of the entire mean field \mu_{t} and the common noise z_{t} , in POMFGs-CNs the agents only obtain a partial observation o_{t} in the finite observation space \mathcal{O} , which depends on s_{t} , \mu_{t} and z_{t} , i.e. o_{t}\sim\mathcal{U}(\cdot\mid s_{t},\mu_{t},z_{t}) . When o_{t} depends on the individual state s_{t} , there is relevant information in the Individual-Action-Observation History (IAOH) \tau_{t}:=(s_{0},o_{0},a_{0},s_{1},o_{1},a_{1},...,s_{t},o_{t})\in\mathcal{H}_{t}:=(\mathcal{S}\times\mathcal{O}\times\mathcal{A})^{t}\times\mathcal{S}\times\mathcal{O} , and thus policies condition on \tau_{t} , i.e. a_{t}\sim\pi(\cdot\mid\tau_{t}) . Defining the analytic mean-field update operator is more involved, as we need to keep track of not just the current mean-field \mu_{t} , which is a distribution over \mathcal{S} , but a distribution \tilde{\mu}_{t} over \mathcal{H}_{t} . \mu_{0:t}=(\mu_{0},\mu_{1},...,\mu_{t}) is then fully determined by \tilde{\mu}_{t} . Note though that the reverse it not true, i.e. \mu_{0:t} does not fully determine \tilde{\mu}_{t} . Given an individual policy \pi , the evolution of \tilde{\mu} is then defined as follows. Given \tilde{\mu}_{t} and z_{t} , we define \displaystyle\mu_{t+1}(s_{t+1}) \displaystyle=\Phi^{\pi}(\tilde{\mu}_{t},z_{t})(s_{t+1}):=\iint\mathcal{T}(s_{t+1}\mid s_{t},a_{t},\mu_{t},z_{t})\pi(a_{t}\mid\tau_{t})\tilde{\mu}_{t}(\tau_{t})\mathrm{d}a_{t}\mathrm{d}\tau_{t}, (10) or \boldsymbol{\mu}_{t+1}={\mathbf{A}^{\pi}_{\boldsymbol{\tilde{\mu}}_{t},z_{t}}}^{\top}\boldsymbol{\tilde{\mu}}_{t} in matrix-vector form. We write \tau_{t+1}=(\tau_{t},a_{t},s_{t+1},o_{t+1}) , and then, given \tilde{\mu}_{t} and z_{t:t+1} , define \tilde{\mu}_{t+1} through \displaystyle\begin{split}\tilde{\mu}_{t+1}(\tau_{t+1})&=\tilde{\Phi}^{\pi}(\tilde{\mu}_{t},z_{t:t+1})(\tau_{t+1})\\ &:=\mathcal{U}(o_{t+1}\mid s_{t+1},\Phi^{\pi}(\tilde{\mu}_{t},z_{0:\infty}),z_{t+1})\mathcal{T}(s_{t+1}\mid s_{t},a_{t},\mu_{t},z_{t})\pi(a_{t}\mid\tau_{t})\tilde{\mu}_{t}(\tau_{t}).\end{split} (11) Furthermore, we let \text{M}_{\text{evol}}(\pi,z_{0:t}) denote the sequence \mu_{0:t} that is generated from \pi and z_{0:t} through ( 11 ) 2 2 2 As for MFGs-CN, we assume w.l.o.g. that \mu_{0} is a deterministic function of z_{0} . This is notationally convenient as then \mu_{0:\infty} is a deterministic function of z_{0:\infty} . As in fully-observable MFGs defined in Section 2.1 , we distinguish between the control problem and the equilibrium problem. In the control problem in POMFGs-CN, we condition on exogeneous population behaviour induced by a fixed policy \pi^{\prime} , and we look to find a policy \pi which maximizes \displaystyle J_{\text{evol}}(\pi,\pi^{\prime}):= \displaystyle\mathbb{E}_{\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}R(s_{t},a_{t},\text{M}_{\text{evol}}(\pi^{\prime},z_{0:t})_{t},z_{t})\right], (12) where the expectation is taken with respect to z_{0}\sim p_{z_{0}} , z_{t+1}\sim\Xi(\cdot|z_{t}) , \mu_{t}=\text{M}_{\text{evol}}(\pi^{\prime},z_{0:t})_{t} , s_{0}\sim\mu_{0} , o_{t}\sim\mathcal{U}(\cdot|s_{t},\mu_{t},z_{t}) , a_{t}\sim\pi(\cdot\mid\tau_{t}),s_{t+1}\sim\mathcal{T}(\cdot\mid s_{t},a_{t},\mu_{t},z_{t}) . Thus the control problem reduces to finding an optimal policy in a standard single-agent POMDP. In the equilibrium problem the population behaviour is endogenous and generated by the policy itself, and the goal is to find a mean-field Nash equilibrium, defined as follows: Definition 5.1 . We say that a policy \pi^{*} (and its induced population behaviour) constitutes a mean-field Nash equilibrium if and only if \displaystyle\pi^{*}\in\text{argmax}_{\pi}J_{\text{evol}}(\pi,\pi^{*}). (13) Unfortunately, unlike the analytic mean-field update operator for policies conditioning on (\mu_{t},z_{t}) as in ( 1 ), the update operators \Phi^{\pi} and \tilde{\Phi}^{\pi} for policies conditioning on IAOHs are intractable, as they require maintaining a distribution \tilde{\mu}_{t} over \mathcal{H}_{t} , with \mathcal{H}_{t} growing exponentially with time. Even if the mean-field update were to be a queryable black-box, to compute the expected reward at the next timestep in HSMs ( 9 ) requires integrating over IAOHs. Thus we consider the following special case, in which things get a lot simpler. 5.1 The Special Case of Shared Observations In many real-world settings involving public information, such as financial markets, for example, agents receive shared observations of the aggregate state, i.e. o_{t}=\mathcal{U}(\mu_{t},z_{t}) 3 3 3 Note that o_{t} being a deterministic rather than a random function of \mu_{t} and z_{t} does not lose generality, as we can just put the randomness of o_{t} into z_{t} . In that case, in order to find mean-field Nash equilibria, we can restrict our search space to policies that only condition on the current individual state s_{t} and the history of shared observations o_{0:t} of the aggregate state, i.e. a_{t}\sim\pi(\cdot\mid s_{t},o_{0:t}) . The benefit is that if o_{t}=\mathcal{U}(\mu_{t},z_{t}) , and a policy \pi only conditions on s_{t} and o_{0:t} , then the dynamics described through \displaystyle\begin{split}\mu_{t+1}(s_{t+1})&=\Phi^{\pi}(\mu_{t},z_{t},o_{0:t})(s_{t+1}):=\iint\mathcal{T}(s_{t+1}\mid s_{t},a_{t},\mu_{t},z_{t})\pi(a_{t}\mid s_{t},o_{0:t})\mu_{t}(s_{t})\mathrm{d}a_{t}\mathrm{d}s_{t},\end{split} (14) are the same as the dynamics described by ( 11 ). Thus, rather than having to keep track of the distribution \tilde{\mu}_{t} over \mathcal{H}_{t} , with \mathcal{H}_{t} growing exponentially in time, memory can be restricted to one shared OH o_{0:t} , which is computationally tractable. 6 Methods 6.1 Recurrent Structural Policy Gradient (RSPG) With access to the transition model and using the reduced policy, we can compute and differentiate through the exact expected return given a sampled initial mean-field distribution and common-noise sequence. The pseudo-code for RSPG is provided in Algorithm 1 in Appendix A.1 . At each iteration, we sample E environments in parallel, rollout the endogenous mean-field sequence \mu^{\pi}_{0:T} , and compute the discounted return backwards through time using the reduced policy \pi(a_{t}\mid s_{t},o_{0:t}) and version of ( 9 ) conditioned on the history of shared aggregate observations. Gradients are taken with respect to the policy parameters and allowed to flow through the individual state transitions and expected rewards over actions. Gradients do not propagate through the mean-field transitions. Figure 1 (right) depicts the architecture of the policy network: only observations of the aggregate state are processed by the recurrent neural network (RNN), ensuring that the hidden state remains independent of the individual state. For continuous actions, rather than parameterising a categorical policy on the discretised action space, we parameterise an underlying continuous distribution and construct a categorical policy by evaluating log-probability-densities at fixed action intervals. This structured prior improves performance compared to directly parameterising the categorical policy by preserving ordinality in the action space. Ablations are provided in Appendix C.4.1 . 6.2 Advantages & Limitations of HSMs and RL HSMs and RL based methods offer complementary advantages and limitations for solving MFGs. By leveraging access to individual transition dynamics, HSMs benefit from greater sample efficiency and lower-variance gradient updates. However, HSMs require tractable analytic mean-field updates and discrete policy classes. In contrast, RL based methods do not require explicit knowledge of the transition model, enabling learning under unknown dynamics, large or intractable state spaces, and dense transition operators. Unlike HSMs, they naturally support continuous state and action spaces, avoiding approximation errors and sensitivities associated with discretisation. Moreover, since RL based methods are fully sample-based, they also extend to partially-observable settings where observations of the aggregate state depend on the individual states, allowing for greater environment complexity. However, they do not benefit from the same variance reduction as HSMs. 7 MFAX 7.1 Environment and Algorithmic Framework Table 1 : Comparison of Open-Source MFG libraries. MFAX is the only library to explicitly distinguish between white-box and black-box access to the individual transition dynamics, and support common-noise, partial observability and multiple initial mean-field distributions. Individual Transition Dynamics Features Library White Box Black Box Partially Observable Common Noise Multiple \mu_{0} MFAX Y Y Y Y Y OpenSpiel N Y N N N MFGLib N Y N N N MFAX is our JAX-based environment framework for MFGs. Unlike existing MFG libraries and as summarised in Table 1 , MFAX cleanly separates environments with white-box access to the individual state transition dynamics from those with black-box access, supporting both exact analytic mean-field updates and sample-based mean-field updates. Also unlike other MFG libraries, MFAX accommodates partial observability, common noise, and multiple initial mean-field distributions. MFAX is designed to combine computational efficiency with ease of use. MFAX environments are implemented with a base class with deterministic individual state transition and reward functions; MFAX then handles the mean-field updates using either the exact analytic mean-field update or a sample-based approximation. When computing the analytic mean-field update and expectations over next individual states, MFAX computes the associated matrix–vector products in functional form, avoiding materialising the \mathbf{A} matrix altogether. This translates memory requirements from O(|\mathcal{S}|^{2}) to O(|\mathcal{S}|+|\mathcal{S}|\times|\mathcal{A}|+|\mathcal{Z}|) , which is more efficient when the transition kernel \mathcal{T}(s_{t+1}\mid s_{t},\mu_{t},z_{t}) is sparse and the action space is much smaller than the state space. Such sparsity and imbalance |\mathcal{A}|\ll|\mathcal{S}| commonly arise in discretised MFGs derived from continuous-state models and where the action space is low-dimensional relative to the discretisation resolution. We refer the reader to Appendix B for more details on MFAX’s environment structure. By parallelising across both states and actions and using the functional representation of the expectation and update operators (i.e. premultiplying by the \mathbf{A} matrix and its transpose), one mean-field update in MFAX is more than 10 times faster than OpenSpiel and 1000 times faster than MFGLib ( 19 ; 11 ) . Being JAX-based, MFAX also allows for GPU accelerated parallelisation across environments, which we leverage in our algorithmic implementations. As well as environments, MFAX implements two HSM and three RL algorithms. We refer the reader to Appendix B.2 for more details. Table 2 : Time for one mean-field update of the Linear Quadratic environment (100 states and 7 actions, following MFG literature ( 39 ) ) for each library. All experiments were averaged over 90 updates after 10 warm-up steps (excluding compilation time) and conducted on NVIDIA L40S GPUs (48 GB). Library Language Mean-Field Update (s) MFAX Python Analytic: 2.98\text{\times}{10}^{-4} Sample-based: 4.35\text{\times}{10}^{-4} OpenSpiel C++ 5.44\text{\times}{10}^{-3} MFGLib Python 3.58\text{\times}{10}^{-1} 7.2 Specific Environment Implementations Table 3 in Appendix B.4 summarises the features of our test environments. The two toy environments, Linear Quadratic and Beach Bar , are partially observable adaptations of environments with common noise that are standard in MFG literature ( 31 ; 39 ) . In Linear Quadratic, the reward encourages agents to concentrate together. Beach Bar’s reward encourages agents to remain close to the bar while it is open, but move away from the bar at the potential closure time. Hence, Beach Bar tests agents’ ability to anticipate that the bar might close. In both environments, the shared observation includes the mean-state, but not the timestep, realisation of common noise, or full mean-field distribution. The Macroeconomics environment is a heterogeneous agent model with common noise, where prices (interest rates and wages) are endogenously determined from the mean-field distribution ( 17 ) . We implement the environment as a long but finite horizon game. The individual state space is 2D, representing wealth and income, while the action space 1D, representing the proportion of its budget constraint that an agent chooses to consume. Agents aim to maximise their present discounted value (PDV) of utility from consumption and thus must learn to balance instant reward with saving for the future. Following 26 , we consider a partially observable case where agents only observe the prices. The common noise is 1D and has the interpretation of the aggregate shock, or the productivity of a so-called representative firm. We refer the reader to Appendix B for more details regarding each environment. 8 Experiments Figure 2 : Exploitability versus training wall-clock time for partially observable Linear Quadratic , Beach Bar , and Macroeconomics environments. All experiments were conducted on NVIDIA L40S GPUs (48 GB). HSMs (SPG, RSPG) are an order of magnitude faster than RL methods, with history-aware RSPG consistently achieving among the lowest exploitability. Shaded regions indicate 95%ile CI for the mean over 10 seeds. Our experiments assess RSPG for quality of convergence (proximity to a Nash equilibrium); speed of convergence and quality of the learned policy, including its ability to capture history-dependent behaviour. 8.1 Ablations and Benchmarks We ablate RSPG with its history-independent counterpart, SPG, which we extend to use an MLP rather than a tabular policy. We also benchmark HSMs with purely sample-based RL algorithms: Independent PPO (IPPO) ( 35 ; 2 ; 37 ) , Recurrent IPPO ( 27 ) and M-OMD ( 23 ) . We refer the reader to Appendix B.2 for more details. 8.2 Evaluation Metrics 8.2.1 Exploitability 22 use the notion of exploitability, which quantifies the maximum increase in expected return that a player can achieve by deviating from the policy used by the population, to determine the proximity to a Nash equilibrium and evaluate convergence: \mathcal{X}(\pi)=\mathbb{E}\left[J^{*}_{\text{evol}}\left(\pi\right)-J_{\text{evol}}\left(\pi,\pi\right)\right] where J^{*}_{\text{evol}}\left(\pi\right)=\sup_{\pi^{\prime}\in\Pi}J_{\text{evol}}\left(\pi^{\prime},\pi\right) , \Pi is the set of policies conditioning on the same inputs as the learned policy \pi , and the expectation is taken with respect to \mu_{0}\sim p_{\mu_{0}},z_{0}\sim p_{z_{0}},z_{t+1}\sim\Xi(\cdot\mid z_{t}). A policy \pi forms a Nash Equilibrium with respect to \Pi if and only if its exploitability is exactly zero. For our experiments, the “exploitability” resembles regret, since rather than restricting \Pi for the best-response policy to the information set of each learned policy (which would render exploitability values incomparable between memoryless and history-aware algorithms), we compute J^{*}_{\text{evol}}(\pi) over a common, more expressive class of best-response policies with perfect information. Even though this might render it impossible to reach zero exploitability, it provides a conservative upper bound on the true distance to equilibrium and enables meaningful comparison across algorithms. We approximate exploitability by averaging over sampled common-noise sequences. Since all of our environments have exact analytic mean-field counterparts, we use backwards induction to compute J^{*}_{\text{evol}}(\pi) . 8.2.2 Wall-Clock Training Time Since the expectation in HSMs renders comparing the number of environment steps for HSMs and RL impossible, we compare algorithms based on wall-clock training time. To facilitate reproducibility and scrutiny, we release all algorithm implementations and hyperparameter configurations (see Appendix C.2 for more details). 8.2.3 Qualitative Analysis To identify interesting behaviour, we visualise the evolution of the mean-field and learned policy. 8.3 Results Figure 3 : Heatmaps: mean-field distribution ( income on y-axis and wealth on x-axis) at specific timesteps during the episode for the Macroeconomics environment (with total episode length of 128 steps). Interest rates ( first column ) and wages ( second column ) are determined by the mean-field distribution. The environment is implemented as a finite horizon: with RSPG, agents learn anticipatory behaviour, spending more wealth just before the end of the episode, pushing interest rates up and wages down. This is not the case for SPG, which is memoryless. RSPG is the only method that consistently achieves competitive exploitability across all three environments, converging to the lowest or second-lowest exploitability (Figure 2 ). SPG performs comparably to RSPG in the endogenous environment because the observation provides substantial information about the aggregate state, reducing the importance of memory. In contrast, M-OMD, SPG, and IPPO perform poorly in the Beach Bar and Linear Quadratic environments because they learn memoryless policies. Notably, despite the endogenous environment being nearly fully observable, M-OMD still underperforms. This could be due to the environment’s larger state-action space, where Q-function based methods struggle to exploit the underlying ordinal structure of the action space. This is not the case for the policy-based methods, which allow us to parameterise an underlying continuous distribution (Section 6.1 and Figure 10 ). As anticipated, SPG and RSPG achieve approximately an order-of-magnitude faster convergence than RL-based methods. HSMs learn directly from mean-field rollouts, while RL methods incur additional overhead from single-agent rollouts between successive mean-field iterations. RIPPO is slower than IPPO and M-OMD because maintaining temporal consistency for history-awareness requires sequential re-calculation in the loss function. This contrasts to memoryless algorithms, where loss function re-calculations can be fully parallelised. Visual inspection of the learned mean-field distributions shows that, unlike memoryless agents, history-dependent agents acquire anticipatory behaviour. This is particularly evident in the Beach Bar and Macroeconomics environments (Figures 4 and 3 ). With RSPG and RIPPO, agents learn to move away from the Beach Bar just before the potential closure time. Similarly, in the macroeconomics environment, agents learn to spend their wealth nearing the end of the episode, which pushes the interest rates upwards. Interest rates are determined by the mean-field distribution, and negatively correlated with percentage of agents in high wealth states. Even though SPG and RSPG attain similar levels of exploitability in the macroeconomics environment (Figure 2 ), only RSPG captures such anticipatory behaviour. We refer the reader to Appendix C.1 for more visualisations. Figure 4 : Mean-field distribution (y-axis) versus time (x-axis) for the Beach Bar environment. Agents are rewarded for being next to the bar when it is open, and penalised for being directly next to the bar when it is closed, or just before it closes, which can occur halfway through the episode (white-line). With RSPG and RIPPO, agents learn to apprehend the time at which the bar might close, moving away from the bar just before potential closure, and back towards it if it stays open (as above). 9 Conclusion In this work we present RSPG, the first history-aware HSM leveraging known transition dynamics for lower variance updates. Empirically, we show that when access to the individual dynamics is available, RSPG achieves state-of-the-art performance, an order-of-magnitude faster convergence, and more realistic agent behaviour in both toy environments and a macroeconomics MFG. We also present MFAX, the first JAX-based environment framework for MFGs. Unlike other MFG libraries, MFAX distinguishes between mean-field environments with white-box and black-box access to the transition model, as well as supporting partial observability, common noise and multiple initial distributions. In the future, HSMs could be applied to more complex situations involving games of major and minor players or several mean-fields. In such cases (where the “common-noise” would now include the behaviour of the major player or opposing mean-fields) rather than differentiating through the entire rollout, a value function could be learned, with the variance reduction being incorporated within the generalised advantage estimation. MFAX enables the implementation of much more complex environments, such as ones with individual agent observation functions or threshold-based reward functions, which could be used to simulate phenomena such as bank-runs, for example. More generally, the library could incorporate a third mean-field update wrapper that supports function approximation for the analytic mean-field update, thereby allowing extensions to higher-dimensional individual state spaces: the analytic mean-field update could be learned from real-world data, for example, with attention being paid only to volatile areas of the mean-field. Impact Statement This paper presents work whose goal is to advance the field of machine learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here. Appendix A Methods A.1 Recurrent Structural Policy Gradient Below, we provide the pseudo-code for RSPG. Crucially, gradients are allowed to propagate through the expected rewards over actions and the individual state transitions, but not through the mean-field updates or environment reward function: Algorithm 1 RSPG input: parameters \theta_{\pi} repeat for E environments in parallel do \mu_{0}\sim p_{\mu_{0}},\;z_{0}\sim p_{z_{0}},\;o_{0}\sim\mathcal{O}(\mu_{0},z_{0}),\;h_{0}\leftarrow\mathbf{0},\;d_{0}\leftarrow 0,\textbf{v}_{T}\leftarrow\mathbf{0} for t=0:T-1 do \triangleright Generate mean-field sequence using analytic mean-field update (\boldsymbol{\Pi}_{t},h_{t+1})\leftarrow\pi_{\theta}(\cdot\mid\mathbf{s},o_{t},h_{t},d_{t}) \boldsymbol{\mu}_{t+1}\leftarrow\text{stop-gradient}\left({\mathbf{A}^{\pi}_{\mu_{t},z_{t},o_{0:t}}}^{\top}\boldsymbol{\mu}_{t}\right) \triangleright Environment transition \mathbf{R}_{t}\leftarrow\text{stop-gradient}\left(R(\mathbf{s},\boldsymbol{\Pi}_{t},\boldsymbol{\mu}_{t},z_{t})\right),\quad z_{t+1}\sim\Xi(\cdot\mid z_{t}),\quad o_{t+1}\sim\mathcal{U}(\cdot\mid\mu_{t+1},z_{t+1}) end for \triangleright Return \{\mu_{t},z_{t},o_{0:t}\}_{0:T},\{\boldsymbol{\Pi}_{t},\mathbf{R}_{t}\}_{0:T-1} for T-1:0 steps do \triangleright Calculate discounted return backwards through time \mathbf{v}_{t}\leftarrow(\boldsymbol{\Pi}_{t}\odot\mathbf{R}_{t})\cdot\mathbf{1}+\gamma\cdot\mathbf{A}^{\pi}_{\mu_{t},z_{t},o_{0:t}}\mathbf{v}_{t+1} end for \triangleright Return \mathbf{v}_{0} end for \triangleright Return \{\boldsymbol{\mu}_{0},\mathbf{v}_{0}\}_{1:E} J\leftarrow\frac{1}{E}\sum\boldsymbol{\mu}_{0}\cdot\mathbf{v}_{0} \theta_{\pi}\leftarrow\theta_{\pi}+\alpha_{\pi}\nabla J \triangleright Update policy parameters until convergence Appendix B Benchmark Structure JAX-based open-source implementations inspiring MFAX include PureJaxRL ( 24 ) , Gymnax ( 20 ) and JaxMARL ( 33 ) , respectively implementing single file RL algorithms; single-agent RL environments; and multi-agent environments and algorithms. B.1 MFAX Environment Structure In this section, we provide a brief overview of MFAX’s modular environment structure. B.1.1 Base Environments Environments are required to have a base class that implements deterministic single state step and reward functions —which return the deterministic next individual states and rewards given the current individual state, aggregate state and a single action— and functions determining when the environment sequence has terminated or truncated. B.1.2 Analytic Mean-Field Wrapper Broadly speaking, analytic mean-field environments must implement: 1. a single state step function, which adds idiosyncratic noise to the deterministic next state and returns the next state indices as well as their probabilities; 2. an aggregate observation function, which returns the common observation of the aggregate state; 3. and a reset function, which determines the exact initial mean-field distribution. Given these functions, the generic environment analytic mean-field update wrapper handles updating the entire mean-field distribution (which involves computing \mathbf{A^{\top}\boldsymbol{\mu}} ), calculating expectations over next states (such as \mathbf{Av} , for example) and returning the reward matrix \mathbf{R} . Rather than constructing or storing \mathbf{A} , we implement the associated matrix–vector products —the analytic mean-field update and expectation operators— in functional form, with the mean-field distribution, policy, vector to be pre-multipled by, and common noise as inputs. This avoids materialising the transition operator altogether, reducing memory requirements from O(|\mathcal{S}|^{2}) to O(|\mathcal{S}|+|\mathcal{S}|\times|\mathcal{A}|+|\mathcal{Z}|) , which is linear in the number of states. This is much more efficient in practice, since in many applications (such as two-dimensional physical spaces and macroeconomic models) the transition kernel \mathcal{T}(s^{\prime}\mid s,\mu,z) is sparse, inducing a highly sparse transition operator \mathbf{A} , while the action space is much smaller than the state space ( |\mathcal{A}|\ll|\mathcal{S}| ). Mean-field sequence The pseudo-code for generating the mean-field sequence for a analytic mean-field environment is presented in Algorithm 2 ; “analytic mean-field” represents the functional form of pre-multiplying by the \mathbf{A^{\top}} matrix. Algorithm 2 Mean-field sequence generation for analytic mean-field environment input: (Recurrent) policy \pi \mu_{0}\sim\triangle_{\mathcal{S}_{0}},\;z_{0}\sim p_{z_{0}},\;o_{0}\sim\mathcal{O}(\mu_{0},z_{0}),\;(h_{0}\leftarrow\mathbf{0}),\;d_{0}\leftarrow 0 for t=0:T-1 do (\boldsymbol{\Pi}_{t},h_{t+1})\leftarrow\pi(\cdot\mid\mathbf{s},o_{t},h_{t},d_{t})\quad(\text{or}\quad\boldsymbol{\Pi}_{t}\leftarrow\pi(\cdot\mid\mathbf{s},o_{t})\quad\text{if non-recurrent)} \boldsymbol{\mu}_{t+1}\leftarrow{\text{analytic-mean-field-update}(\boldsymbol{\mu}_{t},z_{t},\boldsymbol{\Pi}_{t}}) \triangleright Environment transition \mathbf{R}_{t}\leftarrow R(\mathbf{s},\boldsymbol{\Pi}_{t},\boldsymbol{\mu}_{t},z_{t}),\quad z_{t+1}\sim\Xi(\cdot\mid z_{t}),\quad o_{t+1}\sim\mathcal{U}(\cdot\mid\mu_{t+1},z_{t+1}) end for \triangleright Return \{\mu_{t},z_{t}\}_{0:T} B.1.3 Sample-based Wrapper Broadly speaking, sample-based mean-field environments must implement: 1. a single agent step function, which steps a single agent forward given the current individual state, aggregate state and action by sampling idiosyncratic noise and adding that to the deterministic next state from the base environment; 2. a local observation function, which returns the individual observation of the aggregate state; 3. a reset function, which samples individual agents from an initial mean-field distribution. Given these functions, the generic environment sample-based wrapper handles updating the entire mean-field distribution by stepping a fixed number of agents forwards and re-computing the mean-field statistics based on the updated samples. Mean-field sequence The pseudo-code for generating the mean-field sequence for a sample-based environment is presented in Algorithm 3 . Algorithm 3 Mean-field sequence generation for sample-based environment input: (Recurrent) policy \pi \mu_{0}\sim\triangle_{\mathcal{S}_{0}},\;z_{0}\sim p_{z_{0}},\;o_{0}\sim\mathcal{O}(\mu_{0},z_{0}),\;(h_{0}\leftarrow\mathbf{0}),\;d_{0}\leftarrow 0 for N agents in parallel do s^{(i)}_{0}\sim\mu_{0} end for for t=0:T-1 do for N agents in parallel do (\pi_{t},h^{(i)}_{t+1})\leftarrow\pi(\cdot\mid s^{(i)}_{t},o^{(i)}_{t},h^{(i)}_{t},d_{t}))\quad\text{(or}\quad\pi_{t}\leftarrow\pi(\cdot\mid s^{(i)}_{t},o^{(i)}_{t})\quad\text{if non-recurrent}) a^{(i)}_{t}\sim\pi_{t},\quad s^{(i)}_{t+1}\sim\mathcal{T}(\cdot\mid s^{(i)}_{t},a^{(i)}_{t},\mu_{t},z_{t}),\quad r^{(i)}_{t+1}\sim\mathcal{R}(\cdot\mid s^{(i)}_{t},a^{(i)}_{t},\mu_{t},z_{t}) end for z_{t+1}\sim\Xi(\cdot\mid z_{t}),\quad\mu_{t+1}\leftarrow\frac{1}{N}\sum_{i}\delta_{s^{(i)}_{t+1}} for N agents in parallel do o^{(i)}_{t+1}\sim\mathcal{U}(\cdot\mid s^{(i)}_{t+1},\mu_{t+1},z_{t+1}) end for end for \triangleright Return \{\mu_{t},z_{t}\}_{0:T} B.2 MFAX Algorithms We implement the following algorithms in MFAX: SPG We implement SPG ( 42 ) using an MLP rather than a tabular policy. SPG is implemented in the same way as RSPG (pseudo-code given in Appendix A.1 ), but without history. RSPG We implement RSPG (pseudo-code given in Appendix A.1 ). IPPO We implement a mean-field version of Independent Proximal Policy Optimization ( 35 ; 37 ; 2 ) as, despite its theoretical shortcomings, it has been shown to outperform joint learning approaches in various Multi-Agent Reinforcement Learning tasks due to its robustness to environment non-stationarity ( 37 ) . We provide the pseudo-code for our implementation below. Algorithm 4 IPPO input: Policy \pi with parameters \theta_{\pi} repeat for E environments in parallel do \triangleright\text{ Generate mean-field sequence using sample-based mean-field distribution} end for \triangleright Return \{\mu_{0:T},z_{0:T}\}_{1:E} \triangleright\text{Update $\theta_{\pi}$ using single-agent PPO for N agents within each environment.} until convergence RIPPO We implement a mean-field version of Recurrent IPPO ( 27 ) . The pseudo-code is identical to that of IPPO, but the policy includes an RNN. M-OMD We implement M-OMD, an algorithm specifically designed for MFGs, which amends the DQN target to y=r(s,a)+\alpha\tau\log\pi(a\mid s)+\gamma\mathbb{E}_{a^{\prime}\sim\pi(\cdot\mid s^{\prime})}\left[Q(s^{\prime},a^{\prime})-\tau\log\pi(a^{\prime}\mid s^{\prime})\right] and uses a softmax rather than argmax policy to smoothen policy updates, while still allowing for the policy to become greedier with time. Modifying the reward rather than the optimisation objective amends the convergence trajectory of (\pi,\mu_{0:T}) without altering the final fixed-point converged to. M-OMD and its variations ( 38 ; 39 ; 3 ) are state-of-the-art for most MFG problem settings. The pseudocode is given in ( 39 ) . B.3 MFAX Evaluation Metrics To evaluate the algorithms, MFAX computes an approximate exploitability using backwards induction, if environments are implemented using the analytic mean-field update wrapper. For both analytic mean-field and sample-based environment wrappers, MFAX computes the per-agent mean-expected return for that environment. All algorithms allow the train-wall-clock time to be measured. B.4 MFAX Specific Environment Details Table 3 : Overview and comparison of the features and spatial dimensions of the Linear Quadratic, Beach Bar and Endogenous test environments. Features Spatial Dimensions Environment Partially Observable Common Noise Multiple \mu_{0} |\mathcal{S}| |\mathcal{A}| |\mathcal{Z}| Linear Quadratic Y Y N 100 7 2 Beach Bar Y Y Y 100 11 2 Macroeconomics Y Y N 1000 \infty \infty B.4.1 Linear Quadratic Environment The first environment we implement is a partially observable adaptation of the Linear Quadratic environment with common noise that is standard in MFG literature ( 31 ; 39 ) . The environment is finite horizon, with a fixed initial mean-field distribution. We define the following tuple: \displaystyle(p_{\mu_{0}},p_{z_{0}},\mathcal{S},\mathcal{Z},\mathcal{A},\mathcal{O},\mathcal{U},\mathcal{T},\Xi,R,T). (15) The individual state space, the state space of the common noise and the action space are all 1D and given by: \displaystyle\mathcal{S}:=\{0,1,2,\cdots,99\},\quad\mathcal{Z}:=\{-1,1\},\quad\mathcal{A}:=\{-3,-2,-1,0,1,2,3\}. (16) The initial mean-field and common noise are given by: \displaystyle p_{\mu_{0}}:=\delta_{\text{Unif}(\mathcal{S})},\quad p_{z_{0}}:=\text{Unif}(\{-1,1\}). (17) The transition dynamics are given by: \displaystyle\mathcal{T}(s_{t+1}\mid s_{t},a_{t},\boldsymbol{\mu}_{t},z_{t},t):=\text{clip}\left(\text{round}\left(s_{t}+a_{t}+\sigma\cdot\left(\xi_{t}\cdot\rho+\epsilon\cdot\sqrt{1-\rho^{2}}\right)\right),0,|\mathcal{S}|\right),\quad\Xi[z_{t+1}\mid z_{t}]:=\delta_{z_{t}}, (18) where \xi_{t} has a piecewise effect depending on the instantiation of the common noise z_{t} : \displaystyle\xi_{t}=\begin{cases}-10\,z_{t},&t<8,\\ \phantom{-}0\,z_{t},&8\leq t\leq 20,\\ \phantom{-}10\,z_{t},&t>20,\end{cases} (19) and the idiosyncratic noise \epsilon\sim\mathcal{N}(0,1) , is, in practice, discretised as \{-3\sigma,...,3\sigma\} . The reward is given by: \displaystyle R_{t}(s_{t},a_{t},\boldsymbol{\mu}_{t},z_{t},t):=-c_{a}\cdot a_{t}^{2}+q\cdot a_{t}\cdot\left[\boldsymbol{\mu}_{t}\cdot\boldsymbol{s}-s_{t}\right]-\frac{\kappa}{2}\left[\boldsymbol{\mu}_{t}\cdot\boldsymbol{s}-s_{t}\right]^{2},\quad R_{T}(s_{t},a_{t},\boldsymbol{\mu}_{t},z_{t},t):=-\frac{c_{term}}{2}\left[\boldsymbol{\mu}_{t}\cdot\boldsymbol{s}-s_{t}\right]^{2}, (20) effectively encouraging agents to concentrate together. The observation of the aggregate state g=(\boldsymbol{\mu},z,t) is 1D and given by: \displaystyle\mathcal{U}(o_{t}\mid\boldsymbol{\mu}_{t},z_{t},t)=\left(\delta\left(o_{t}-\boldsymbol{\mu}_{t}\cdot\boldsymbol{s}\right)\right), (21) where \boldsymbol{s} denotes a vector of length |\mathcal{S}| containing the index of state, such that the observation is just the mean state of the mean-field. We use the following parameters for the implementation: Table 4 : Parameters used for the Linear-Quadratic environment. Parameter Description Value \sigma Idiosyncratic Noise Std 1.0 \rho Noise Parameter 0.5 c_{a} Reward Action Cost 0.5 c_{\text{term}} Reward Terminal Cost 1.0 q Reward Action Direction Parameter 0.1 \kappa Reward Distance Penalty 0.5 T Time Horizon 30 |\mathcal{S}| Number of States 99 |\mathcal{A}| Number of Actions 7 |\mathcal{Z}| Number of Common Noise Realisations 2 B.4.2 Beach Bar 1D Environment The second environment we implement is a partially observable version of the beach-bar environment, again standard in MFG literature ( 31 ; 39 ) . The environment is finite horizon. The aggregate aggregate state is given by g=(\boldsymbol{\mu},z,l,t) , where l is the location of the bar that is set at the beginning of the episode and hence independent of time, and z , the common noise, determines whether the bar closes at t=T/2 . We define the following tuple: \displaystyle(p_{\mu_{0}},p_{z_{0}},\mathcal{L}_{0},\mathcal{S},\mathcal{Z},\mathcal{A},\mathcal{O},\mathcal{U},\mathcal{T},\Xi,R,T). (22) The individual state space, the state space of the common noise and the action space are all 1D and given by: \displaystyle\mathcal{S}:=\{0,1,2,\cdots,99\},\quad\mathcal{Z}:=\{0,1\},\quad\mathcal{A}:=\{-5,-4,-3,-2,-1,0,1,2,3,4,5\}. (23) The initial mean-field and common noise are given by: \displaystyle\mathcal{L}_{0}:=\text{Unif}(\mathcal{S}),\quad l_{0}\sim\mathcal{L}_{0},\quad p_{\mu_{0}}:=\delta_{\text{Unif}(\mathcal{S}\backslash l_{0})},\quad p_{z_{0}}:=\text{Unif}(\{0,1\}). (24) The transition dynamics are given by: \displaystyle\mathcal{T}(s_{t+1}\mid s_{t},a_{t},\boldsymbol{\mu}_{t},z_{t},t):=\text{legal}\left[s_{t}+a_{t}+\epsilon\right],\quad\Xi[z_{t+1}\mid z_{t}]:=\delta_{z_{t}}, (25) where \epsilon is an instantiation of the idiosynchratic noise and is sampled from \{-2,-1,0,1,2\} with probability \{0.05,0.1,0.7,0.1,0.05\} , and “ legal ” is a function that prevents agents from moving into illegal states. The instantiation of the common noise \xi is given by: \displaystyle\xi_{t}=\begin{cases}1,&t