Maximum Principle of Optimal Probability Density Control
Nathan Gaby, Xiaojing Ye · 2025
Read Maximum Principle of Optimal Probability Density Control by Nathan Gaby, Xiaojing Ye (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18362.txt.
Introduction
Controlling a crowd of agents is easier as a distribution than as a list of individuals. Nathan Gaby and Xiaojing Ye build a theory of optimal probability density control on general measure spaces, aimed at large multi-agent problems. They prove a maximum principle for control problems posed on the infinite-dimensional space of probability distributions and control fields.
They also derive the Hamilton-Jacobi-Bellman equation for the value functional on that space. Both results come in compact form with rigorous proofs, built for numerical use. On top of the principle they design a scalable algorithm that uses deep neural networks for high-dimensional cases. Tests on multi-agent problems with obstacles and inter-agent interactions show the method working in practice.
Catalog record
| CREATOR | Nathan Gaby, Xiaojing Ye |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 83 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-18362
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Maximum Principle of Optimal Probability Density Control Nathan Gaby, Xiaojing Ye Abstract We develop a general theoretical framework for optimal probability density control on standard measure spaces, aimed at addressing large-scale multi-agent control problems. In particular, we establish a maximum principle (MP) for control problems posed on infinite-dimensional spaces of probability distributions and control vector fields. We further derive the Hamilton–Jacobi–Bellman equation for the associated value functional defined on the space of probability distributions. Both results are presented in a concise form and supported by rigorous mathematical analysis, enabling efficient numerical treatment of these problems. Building on the proposed MP, we introduce a scalable numerical algorithm that leverages deep neural networks to handle high-dimensional settings. The effectiveness of the approach is demonstrated through several multi-agent control examples involving domain obstacles and inter-agent interactions. Keywords: Control Hamiltonian dynamics, Hamilton–Jacobi–Bellman equation of value functional, Maximum principle, Optimal probability control. 1 Introduction Optimal probability density control has recently gained significant attention due to its increasing prevalence in multi-agent control systems managing swarms of agents, such as drones, robots, and autonomous vehicles Brambilla et al. (2013) ; Dorigo et al. (2021) ; Feng et al. (2020) . These problems are increasingly relevant in real-world applications, including large-scale surveillance, time-critical search-and-rescue operations, and infrastructure inspection. Given the large scale of these systems, a probabilistic description of the state is often the most effective approach Jimenez et al. (2020) . Consequently, a standard method for solving these multi-agent control problems is through mean-field modeling, which serves as a continuum approximation of the discrete multi-agent system. Here, the probability density of the agent states represents the entire system. Notably, the state of an agent may reside in a high-dimensional space, as it encompasses combined variables such as location, orientation, velocity, and acceleration. In this regard, optimal probability control and multi-agent control are unified into the same problem framework, and the goal is to find the control vector field that maximizes a given total reward function describing the probability density, or equivalently the collective behaviors of the agents, with consideration of inter-agent collaboration and competitions. Let \Omega be an open subset of \mathbb{R}^{d} , where d is the dimension of state vector (such as the concatenation of location, orientation, acceleration, and etc.) of an agent. Suppose there are N agents with states \{x_{i}\in\Omega\mathrel{\mathop{\mathchar 58\relax}}i=1,\dots,N\} which follow some initial distribution p on \Omega , i.e., x_{i}\sim p . Under a time-evolving control vector field u\mathrel{\mathop{\mathchar 58\relax}}\Omega\times[0,T]\to\mathbb{R}^{d} over a fixed time horizon [0,T] , the states of the agents change according to \dot{x}_{i}(t)=u(x_{i}(t),t),\hskip 9.24994pti=1,\dots,N. (1) Each agent accumulates some running rewards during [0,T] and receives a terminal reward at time T . Unlike classical optimal control, these rewards depend not only this specific agent but also the collective behaviors of all agents. We provide a few examples of running and terminal reward functions below. We use \rho(\cdot,t) to denote the time marginal of the states x_{i}(t) for every t , namely, x_{i}(t)\sim\rho(\cdot,t) . Then ( 1 ) corresponds to the continuity equation \partial_{t}\rho(x,t)+\nabla\cdot\bigr(\rho(x,t)u(x,t)\bigr)=0 (2) in \Omega\times[0,T] . For notation simplicity, we write \rho(\cdot,t) and u(\cdot,t) as \rho_{t} and u_{t} respectively, and often omit (x) in the integrals hereafter unless it is needed for clarification. Consider for example that the agents are required to keep distances from each other to avoid collision. In this case, we can set the running reward received by the i th agent at time t as -\frac{1}{2}|u(x_{i}(t),t)|^{2}-\sum_{j\neq i}\frac{\gamma}{|x_{j}(t)-x_{i}(t)|^{2}+\epsilon}, (3) where \gamma,\epsilon>0 are user-selected hyperparameters. In ( 3 ), the first term penalizes the cost of control energy by the i th agent, whereas the second term penalizes the closeness to other agents, with \gamma as the penalty weight and \epsilon a tiny threshold value ensuring the denominator is positive. Taking sum of ( 3 ) over i=1,\dots,N , dividing by N , and letting N\to\infty , we find the continuum limit to be a functional of \rho_{t} and u_{t} : -\frac{1}{2}\int_{\Omega}|u_{t}|^{2}\rho_{t}\,dx-\int_{\Omega}\int_{\Omega}\frac{\gamma}{|y-x|^{2}+\epsilon}\rho_{t}(y)\rho_{t}(x)\,dydx. (4) This is an example of running reward functional R(\rho_{t},u_{t}) . For the terminal reward, a simple example is that we want all agents to gather at a specified state x^{*}\in\Omega at the terminal time T . In this case, we can set the terminal reward received by the i th agent at T as -\frac{1}{2}|x_{i}(T)-x^{*}|^{2}, (5) which encourages the i th agent to get close to the target x^{*} . Again, taking sum of i=1,\dots,N , dividing by N , and letting N\to\infty , we obtain the total terminal reward functional of \rho_{T} : -\frac{1}{2}\int_{\Omega}|x-x^{*}|^{2}\rho_{T}(x)\,dx, (6) which is an example of terminal reward functional G(\rho_{T}) . The running and terminal rewards ( 4 ) and ( 6 ) are typical examples in multi-agent control, whereas other suitable rewards can be designed according to specific applications. Now we have the optimal probability control problem of interest in its general form as \displaystyle\max_{u\in U_{T}}\hskip 9.24994pt \displaystyle I[u]\mathrel{\mathop{\mathchar 58\relax}}=\int_{0}^{T}R(\rho_{t},u_{t})\,dt+G(\rho_{T}), (7a) \displaystyle\mathrm{s.t.}\hskip 9.24994pt \displaystyle\partial_{t}\rho_{t}+\nabla\cdot(\rho_{t}u_{t})=0,\hskip 9.24994pt\forall\,(t,x)\in[0,T]\times\Omega,\phantom{\frac{}{}} (7b) \displaystyle\rho_{0}=p,\hskip 9.24994pt\forall\,x\in\Omega, (7c) where I is the total reward functional, U_{T} is the admissible set of control vector fields, and p is a given initial probability distribution, i.e., x_{i}(0)\sim p . (We assume I^{*}\mathrel{\mathop{\mathchar 58\relax}}=\max_{u\in U_{T}}I[u] to be finite in this work.) We will give the definitions of the terms in ( 7 ) in detail in Section 3 . The main goal of this work is to establish a maximum principle of the optimal probability control problem ( 7 ) in Section 3.1 , along with newly defined adjoint partial differential equation (adjoint PDE) and control Hamiltonian functional H . In addition to the maximum principle, we establish the Hamilton–Jacobi–Bellman equation of the value functional associated with ( 7 ) in Section 3.2 . Moreover, we develop a numerical algorithm aligned with the proposed maximum principle and demonstrate its performance on several probability (multi-agent) control examples in Section 4 . The novelty and significance of the results presented in this work are in two phases: (i) These results are rigorously established on the infinite-dimensional space of probability density functions in contrast to finite-dimensional Euclidean space in classic control theory; and (ii) Our theory is built on standard measure space, yielding concise presentation, simple interpretation, and developments of fast numerical algorithms for applications. 2 Related Work Characterizing behaviors of large multi-agent systems using probability distributions is typically called a mean-field model. The approach of mean-field model to approximating multi-agent system can be traced back to McKean (1967) . Theoretical justifications are provided in Fornasier and Solombrino (2014) ; Fornasier et al. (2019) . Mean-field models are also used to establish Nash equilibrium in the case where each agent try to maximal its individual reward depending on the configurations of all the other agents in large multi-agent systems. This is an important research topic, known as the mean field game (MFG) Caines et al. (2006) ; Nourian and Caines (2013) ; Lasry and Lions (2007) , which has been intensively studied in the past two decades. In this section, we present an overview of optimality conditions and numerical algorithms for general probability control problem ( 7 ). Due to space limitation, we omit results for special cases of ( 7 ), such as those with control-affine drifts and quadratic running cost. A substantial