Batch Sample-wise Stochastic Optimal Control via Stochastic Maximum Principle

Hui Sun, Feng Bao · 2025

Read Batch Sample-wise Stochastic Optimal Control via Stochastic Maximum Principle by Hui Sun, Feng Bao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-02688.txt.

Introduction

Hui Sun and Feng Bao speed up a probabilistic method for steering noisy systems toward a goal, in a 2025 paper. Stochastic optimal control weighs future noise against present action, and the maximum principle gives necessary conditions for the best control. Solving those conditions by gradient descent on simulated paths was accurate but slow to converge.

Their batch scheme simulates many sample paths at once and uses a higher-order solver for the forward equation. The convergence rate improves markedly: one error term that shrank like one over root N now shrinks like one over N squared. Careful batch simulation of the backward equation also yields a sharper estimate of the control that minimizes the Hamiltonian. A damped contraction algorithm follows, with convergence proved for a special case. Tests confirm the predicted rates. The authors sketch use with randomized neural networks, where updates turn into pure algebra.

Catalog record

CREATORHui Sun, Feng Bao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE86 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-02688

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Batch Sample-wise Stochastic Optimal Control via Stochastic Maximum Principle Hui Sun, Feng Bao Abstract In this work, we study the stochastic optimal control problem (SOC) mainly from the probabilistic view point, i.e. via the Stochastic Maximum principle (SMP) [ 25 ] . We adopt the sample-wise backpropagation scheme proposed in [ 1 ] to solve the SOC problem under the strong convexity assumption. Importantly, in the Stochastic Gradient Descent (SGD) procedure, we use batch samples with higher order scheme in the forward SDE to improve the convergence rate in [ 1 ] from ∼ 𝒪 ⁡ ( N K + 1 N ) \sim\mathcal{O}(\sqrt{\frac{N}{K}+\frac{1}{N}}) to ∼ 𝒪 ⁡ ( 1 K + 1 N 2 ) \sim\mathcal{O}(\sqrt{\frac{1}{K}+\frac{1}{N^{2}}}) and note that the main source of uncertainty originates from the scheme for the simulation of Z Z term in the BSDE. In the meantime, we note the SGD procedure uses only the necessary condition of the SMP, while the batch simulation of the approximating solution of BSDEs allows one to obtain a more accurate estimate of the control u u that minimizes the Hamiltonian. We then propose a damped contraction algorithm to solve the SOC problem whose proof of convergence for a special case is attained under some appropriate assumption. We then show numerical results to check the first order convergence rate of the projection algorithm and analyze the convergence behavior of the damped contraction algorithm. We note that comparisons between the numerical results from these two algorithms (projection and contraction) show the projection approach is still favored in terms of stability and efficiency. Lastly, we briefly discuss how to incorporate the proposed scheme in solving practical problems especially when the Randomized Neural Networks are used. We note that in this special case, the error backward propagation can be avoided and parameter update can be achieved via purely algebraic computation (vector algebra) which will potentially improve the efficiency of the whole training procedure. Such idea will require further exploration and we will leave it as our future work. Keywords: Stochastic Maximum Principle, batch gradient descent, contraction with damping, convergence analysis. Neural networks. MSC codes: 60H10, 60H30, 65C20. 1 Introduction In recent years, stochastic optimal control problems have been studied extensively especially with the dawning of machine learning/deep learning assisted mathematical approaches. A large body of both theoretical [ 3 , 31 , 23 , 17 , 26 ] and applied numerical aspect of the subject [ 14 , 9 , 33 , 16 , 29 , 20 ] can be found. Recently, an extensive amount of work has been dedicated in leveraging the deep learning techniques to solve the control problems in a supervised learning framework for both the classical and the mean-field settings [ 18 , 27 , 28 , 4 , 5 , 6 , 22 ] . All those frameworks rely on directly approximating the control with a deep neural network whose trainable weights are then updated iteratively by a general gradient descent type method via minimizing the expected loss. The current work instead focuses on using one of the more classical probabilistic approach in solving the stochastic optimal control problem, namely the stochastic maximum principle. In this approach, one will first find the numerical approximation of the variational derivative J^{\prime}_{u}(u) of the loss and then update the control via gradient descent method. The benefit of such approach is that when one works with a deterministic control under some strong convexity like condition, and when J^{\prime}_{u}(u) is appropriately approximated, the first order error convergence can be shown [ 14 ] . The downside however, is that a good approximation of J^{\prime}_{u}(u) requires solving a system of FBSDE at each gradient descent iteration which is time consuming. More specifically, finding the numerical approximation J^{{}^{\prime},N}_{u}(u) typically requires solving the following FBSDEs (discretized): \displaystyle\begin{cases}X^{N}_{n}&=b(t_{n},X^{N}_{n},u_{t_{n}})h+\sigma(t_{n},X^{N}_{n},u_{t_{n}})\Delta W_{n}\\ Y^{N}_{t_{n}}&=\mathbb{E}_{t_{n}}[Y^{N}_{t_{n+1}}]+h\mathbb{E}_{t_{n}}[b^{\prime}_{x}(t_{n},X_{n},u_{t_{n}})Y^{N}_{{t_{n+1}}}+\sigma^{\prime}_{x}(t_{n},X_{n},u_{t_{n}})Z^{N}_{t_{n}}+f^{\prime}_{x}(t_{n},X_{n},u_{t_{n}})]\\ Z^{N}_{t_{n}}&=\mathbb{E}_{t_{n}}[Y^{N}_{t_{n+1}}\Delta W_{t_{n+1}}]/h.\end{cases} (1.1) Common methods used in solving the system of such FBSDE involves i) Monte Carlo methods ii) using regressors to approximate (Y^{N}_{t_{n}},Z^{N}_{t_{n}}) . iii) Find (Y^{N}_{t_{n}},Z^{N}_{t_{n}}) on preselected mesh-points and then use interpolation to find approximation for the entire target function. iv) use deep learning to find approximation for both (Y^{N}_{n},Z^{N}_{n}) and solve them backward iteratively. However all of these approaches requires finding the solution of the BSDEs iteratively for each projection iteration. And some of the methods are not very effective when the underlying dimension of the control of state is high. On the other hand, it is noted in [ 2 ] and [ 1 ] that the goal of finding the optimal control is not about solving the BSDEs but to find an effective approximation of the projection J^{\prime}_{u}(u) . As such, a sample-wise approximation of the ( 1.1 ) is proposed: \displaystyle\begin{cases}X_{n+1}=X_{n}+b(t_{n},X_{n},u_{n})h+\sigma(t_{n},u_{n})\Delta W_{n}\\ Y_{n}=Y_{n+1}+h\big(b^{\prime}_{x}(t_{n},X_{n},u_{n})Y_{n+1}+\sigma^{\prime}_{x}(t_{n},X_{n},u_{n})Z_{n}+f^{\prime}_{x}(t_{n},X_{n},u_{n})\big),\ \ Y_{T}=g_{x}(X_{T})\\ Z_{n}=\frac{Y_{n+1}\Delta W_{n}}{h}\end{cases} (1.2) and the original gradient (under numerical scheme) J^{{}^{\prime},N}_{u}(u)|_{t=t_{n}}=\mathbb{E}[b^{\prime}_{u}(t_{n},X_{t},u_{t})Y^{N,u}_{t_{n}}+\sigma^{\prime}_{u}(t_{n},X^{N,u}_{t_{n}},u_{t_{n}})Z^{N,u}_{t_{n}}+f^{\prime}_{u}(t_{n},X^{N,u}_{t_{n}},u_{t_{n}})] is approximated by the sample gradient j^{\prime}_{u}(u)=b^{\prime}_{u}(t,X_{n},u_{n})Y^{u}_{n}+\sigma^{\prime}_{u}(t,X_{n},u_{n})Z^{u}_{n}+f^{\prime}_{u}(t,X_{n},u_{n}) . This sample-wise approach together with stochastic gradient descent as the update procedure is shown to be very effective. A convergence proof under the strong convexity assumption is also given in [ 1 ] and the rate of convergence is found to be \sim\mathcal{O}(\sqrt{\frac{N}{K}+\frac{1}{N}}) . We still adopt the sample-wise backward simulation in this work and we seek to improve the efficiency of the algorithm by using batch sampling and provide rigorous proofs for the rate of convergence. We will also explore a new optimization approach based on batch sampling and study its effectiveness. The main results of the paper are as follows: 1. we adopt batch samples with higher order scheme in the forward SDE to improve the convergence rate in [ 1 ] from \sim\mathcal{O}(\sqrt{\frac{N}{K}+\frac{1}{N}}) to \sim\mathcal{O}(\sqrt{\frac{1}{K}+\frac{1}{N^{2}}}) and note that the main source of uncertainty originates from the scheme for the simulation of Z term in the BSDE. Thus, the batch simulation in the BSDE and the SGD procedure significantly reduces the noise in the original estimator j^{\prime}_{u}(u) . We also note that the first order convergence of the control is the optimal rate given the Euler type approximation of the true control. 2. In the meantime, we note the SGD procedure uses only the necessary condition of the SMP, while the batch simulation of the approximating solution of BSDEs allows one to obtain a more accurate estimate of the control u that minimizes the Hamiltonian. We then propose a damped contraction algorithm to solve the SOC problem whose proof of convergence is attained under some appropriate assumption. We then show numerical results to check the first order convergence rate of the projection algorithm and convergence behavior of the damped contraction algorithm. We comment that the convergence analysis for the contraction approach, though needs to be further refined, still sheds some light about how to chose the batch size M against N We note that comparisons between the numerical results from these two new algorithms (batch SGD projection and contraction) and the original SGD shows that the two new approaches based on batch sampling is much more time-efficient and the batch SGD approach stands out to be most efficient and whose experimental results are more interpretable. 3. Lastly, we discussed and showed how to incorporate the proposed scheme in training neural network especially when the Randomized Neural Networks are used. We note that in

Read the complete text