Model-free policy gradient for discrete-time mean-field control
Matthieu Meunier, Huyên Pham, Christoph Reisinger · 2026
Read Model-free policy gradient for discrete-time mean-field control by Matthieu Meunier, Huyên Pham, Christoph Reisinger (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-11217.txt.
Introduction
Matthieu Meunier, Huyen Pham, and Christoph Reisinger learn control rules with no model. Their 2026 preprint deals with mean-field control in discrete time. Mean-field control steers a vast crowd as one system. Each agent's best move depends on the whole crowd's state.
Past work on this field leans on value-based methods. Policy-based ways are far less explored, for a clear reason. The crowd state drives both moves and rewards as time runs. That loop blocks the classic gradient tricks from single-agent learning. The team builds a policy gradient that works without a model. The state space is finite and the action space compact. The result opens a new path for crowd-scale control.
Catalog record
| CREATOR | Matthieu Meunier, Huyên Pham, Christoph Reisinger |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 138 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-11217
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Model-free policy gradient for discrete-time mean-field control Matthieu Meunier, Huyên Pham, Christoph Reisinger Abstract We study model-free policy learning for discrete-time mean-field control (MFC) problems with finite state space and compact action space. In contrast to the extensive literature on value-based methods for MFC, policy-based approaches remain largely unexplored due to the intrinsic dependence of transition kernels and rewards on the evolving population state distribution, which prevents the direct use of likelihood-ratio estimators of policy gradients from classical single-agent reinforcement learning. We introduce a novel perturbation scheme on the state-distribution flow and prove that the gradient of the resulting perturbed value function converges to the true policy gradient as the perturbation magnitude vanishes. This construction yields a fully model-free estimator based solely on simulated trajectories and an auxiliary estimate of the sensitivity of the state distribution. Building on this framework, we develop MF-REINFORCE, a model-free policy gradient algorithm for MFC, and establish explicit quantitative bounds on its bias and mean-squared error. Numerical experiments on representative mean-field control tasks demonstrate the effectiveness of the proposed approach. Key words: Mean-field control; Policy gradient methods; Model-free reinforcement learning; Markov decision process; Perturbation analysis. AMS subject classifications: 93E20; 60J10; 68T05; 60K35; 49N80. 1 Introduction Designing algorithms which optimally control large populations of cooperative agents has recently become an important problem due to the breadth and depth of applications. In the face of the complexity of modelling individual interactions between each pair of agents, mean-field control provides a rigorous model particularly suited for problems with many agents interacting in a symmetrical way: instead of considering all possible interactions between every pair of players, one lets the number of agents go to infinity and tries to solve a limiting problem for which it suffices to study the interaction between a typical agent and the probability distribution of its state. An important and practical research direction for such decision-making problems consists in model-free reinforcement learning (RL) algorithms. RL algorithms are based on two elementary approaches: value-based learning algorithms, such as Q -learning ( [ 42 ] ), where one tries to learn a map which allows one to compare the optimal expected reward for different actions and subsequently learn a policy from this map, and policy-based algorithms, also known as policy gradient methods, such as REINFORCE ( [ 44 ] ), which directly try to optimize the value function with respect to the parameters of a policy. These two approaches are of great importance and are combined in modern deep reinforcement learning algorithms. The study of such approaches tailored to mean-field control problems has recently received more attention, with many works developing value-based algorithms. However, there is a notable lack of study of model-free policy-based methods for this setting in the literature. This paper fills this gap by introducing the first policy gradient algorithm for mean-field control problems in discrete time. As in the classical REINFORCE algorithm for single-agent problems, our algorithm is entirely model-free and can be performed by simply observing trajectories of the system. 1.1 Outline of Main Results In this section, we provide an informal description of the problem at hand and present our main contributions. We consider a mean-field control problem in discrete time without common noise and time horizon T\in{\mathbb{N}} . We assume that the state space \mathcal{X} is finite and that the action space \mathcal{A} is compact and Polish. Given an admissible control \alpha=(\alpha_{t})_{t=0}^{T-1} , the \mathcal{X} -valued state follows the dynamics X_{0}=\xi,\quad X_{t+1}\sim P(\cdot\mid X_{t},\alpha_{t},\mathbb{P}_{X_{t}}), where \mathbb{P}_{X_{t}}\in\mathcal{P}(\mathcal{X}) denotes the law of X_{t} . We denote by P(\cdot\mid x,a,\mu) the underlying probability kernel. Consider a running reward r:\mathcal{X}\times\mathcal{A}\times\mathcal{P}(\mathcal{X})\rightarrow{\mathbb{R}} and a terminal reward g:\mathcal{X}\times\mathcal{P}(\mathcal{X})\rightarrow{\mathbb{R}} , which are both assumed to be measurable and bounded. The goal is to maximize the following objective, for a given initial condition \xi \sim \mu_{0} , \mathbb{E}_{\xi\sim\mu_{0}}\left[\sum_{t=0}^{T-1}r(t,X_{t},\alpha_{t},\mathbb{P}_{X_{t}})+g(X_{T},\mathbb{P}_{X_{T}})\right], over a set of admissible controls. For now, we consider randomized controls of feedback form, meaning that the set of admissible controls is parametrized by \Pi\coloneq\{\pi:\{0,\ldots,T-1\}\times\mathcal{X}\times\mathcal{P}(\mathcal{X})\rightarrow\mathcal{P}(\mathcal{A})\text{ measurable}\}. Hence a control \alpha is admissible if there exists \pi\in\Pi such that \alpha_{t}\sim\pi(\cdot\mid t,X_{t},\mathbb{P}_{X_{t}}) for all t\in\{0,\ldots,T-1\} . Therefore, we seek to maximize V^{\pi}(\mu_{0})\coloneq\mathbb{E}^{\pi}_{\xi\sim\mu_{0}}\left[\sum_{t=0}^{T-1}r(X_{t},\alpha_{t},\mathbb{P}_{X_{t}})+g(X_{T},\mathbb{P}_{X_{T}})\right], (1.1) where \mathbb{E}^{\pi} refers to \alpha=(\alpha_{t}) being generated by \pi . Our goal is to develop a model-free policy gradient algorithm to solve this mean-field control problem. Consider a set of parametrized policies \{\pi_{\theta}:\theta\in\Theta\} with \Theta\subseteq{\mathbb{R}}^{D} given, e.g., by a neural network with trainable parameters. Then we want to find an estimator for \nabla_{\theta}V^{\pi_{\theta}}(\mu_{0}) that can easily be computed in a reinforcement learning setting, e.g., by generating trajectories of X with a simulator. In continuous time, this problem has been studied by [ 11 ] , where the Itô formula along flows of measures is used to write a policy gradient of the value function that can be estimated in a model-free setting. Our main contributions are as follows: • We rigorously derive a policy gradient formula for the value function in Proposition 2.1 . This result highlights the computational challenges stemming from the mean-field dependence of the transition kernel and the reward functions. In particular, contrary to the single-agent problem, it is not possible to directly obtain a model-free estimator of the gradient using a likelihood ratio trick. • We introduce a perturbation scheme, along with a perturbed value function, and show in Theorem 2.4 that the gradient of the perturbed value function converges to the true value function as the size of the perturbation goes to 0 . In particular, the gradient of the perturbation is amenable to model-free estimation, but requires an auxiliary estimation procedure of the gradient (of the logits) of the state distribution. • We present a model-free algorithm to estimate the gradient of the value function, which we call MF-REINFORCE, based on the formula for the gradient of the perturbed value function in Algorithm 1 . We provide a detailed analysis and quantitative bounds on the bias and mean squared error of the newly introduced estimator in Theorems 3.3 and 3.5 respectively, based on the size of the perturbation and number of trajectories sampled. • We present numerical experiments of MF-REINFORCE on a few problems in Section 4 . 1.2 Related Works Policy Gradient Methods in Multi-Agent Problems Previous works tackle the challenge of learning optimal policies in problems with many agents, both in the mean-field and finite-player settings. The work of [ 5 ] investigates policy gradient methods for linear-quadratic MFC in discrete time. The recent work of [ 36 ] studies a (model-based) policy learning algorithm for MFC by simulating a corresponding N -particle system. Actor–critic algorithms for asymptotic mean-field games and control problems have been introduced and analyzed by [ 2 ] and [ 8 ] . Surprisingly enough, continuous-time problems have received more attention (see, e.g., [ 9 ] ), due to the explicit policy gradient formula one can obtain via Itô’s formula along flows of measures ( [ 11 ] ), where policy gradient algorithms have been developed using moment neural networks ( [ 31 ] ), and for linear-quadratic problems ( [ 41 , 12 ] ). From the perspective of multi-agent reinforcement learning (MARL), [ 45 ] use mean-field symmetry to design policy gradient methods, [ 29 ] propose a natural policy gradient method for heterogeneous