Learning-based primal-dual optimal control of discrete-time stochastic systems with multiplicative noise

Xiushan Jiang, Weihai Zhang · 2025

Read Learning-based primal-dual optimal control of discrete-time stochastic systems with multiplicative noise by Xiushan Jiang, Weihai Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-02613.txt.

Introduction

Xiushan Jiang and Weihai Zhang present a study from twenty twenty-five on control of systems that face random noise. They use learning methods that do not need a full model of the system. Classic Q-learning has gaps: its theory is loose, and it needs added noise while it runs. The pair take a fresh view of a known control task with noise that scales with the state.

They recast the task as a paired problem with a best and a worst side, and prove the two sides meet. From set rules for the best point, they draw both model based and model free methods. A test case shows the model free method works. They link it to how the brain guides arm moves.

Catalog record

CREATORXiushan Jiang, Weihai Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-02613

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learning-based primal-dual optimal control of discrete-time stochastic systems with multiplicative noise Xiushan Jiang, Weihai Zhang Abstract Abstract- Reinforcement learning (RL) is an effective approach for solving optimal control problems without knowing the exact information of the system model. However, the classical Q-learning method, a model-free RL algorithm, has its limitations, such as lack of strict theoretical analysis and the need for artificial disturbances during implementation. This paper explores the partially model-free stochastic linear quadratic regular (SLQR) problem for a system with multiplicative noise from the primal-dual perspective to address these challenges. This approach lays a strong theoretical foundation for understanding the intrinsic mechanisms of classical RL algorithms. We reformulate the SLQR into a non-convex primal-dual optimization problem and derive a strong duality result, which enables us to provide model-based and model-free algorithms for SLQR optimal policy design based on the Karush-Kuhn-Tucker (KKT) conditions. An illustrative example demonstrates the proposed model-free algorithm’s validity, showcasing the central nervous system’s learning mechanism in human arm movement. K eywords: Stochastic linear quadratic problem; multiplicative noise; reinforcement learning; primal-dual method. 1 Introduction Linear quadratic regular (LQR) was initiated by Kalman [ 14 ] , and further developed in [ 1 , 17 ] . It is well-known that LQR is one of the most important optimal controls, which is very elegant in theory and has more applications in engineering practice [ 8 , 35 ] . Stochastic linear quadratic regular (SLQR) seems to be first studied by [ 31 ] , in particular, since [ 7 ] established the indefinite SLQR theory, SLQR has gained a lot of scholars’ attention, and has been extensively studied; see [ 11 , 25 , 32 , 37 ] . Generally speaking, SLQR will lead to solving a generalized algebraic Riccati equation (GARE), which requires us to know complete system information including the system structure and exact parameter information. However, the exact model structure and system parameters are commonly unknown in the process of practical modeling, in this case, all traditional model-based methods become invalid. The model-free reinforcement learning (RL) approach provides a solution to unknown dynamics by exploring poorly structured systems through state-input data analysis. RL is a branch of machine learning that iteratively achieves an optimal policy through interactions with the environment. Pioneering studies [ 2 , 30 ] initiated the development of RL within the optimal control framework, which has since garnered significant attention [ 3 , 5 , 22 , 42 , 28 , 12 ] . For example, the reference [ 16 ] used the off-policy RL to study the H_{\infty} control of linear discrete-time systems, while [ 27 ] researched the adaptive optimal control of linear continuous-time systems based on the policy iteration (PI). In [ 23 ] , the authors used a novel off-policy RL method named optimistic least squares-based PI to find directly near-optimal controllers from input/state data for adaptive optimal stationary control of linear Itô systems with additive and multiplicative noises. Q-learning is one of the most important RL approaches, which has been studied as an effective model-free RL algorithm [ 9 , 10 , 15 , 19 , 26 , 33 ] for LQR optimal policy design. This is because LQR is one class of the most important and simplest optimal controls, which captures the main characteristics of Q-learning. Particularly, in [ 10 ] , a stochastic policy gradient algorithm was presented with a shortcoming of large variance. Modified approximate PIs for LQR were given in [ 15 , 33 ] . In [ 19 ] , the model-free RL algorithm based on Q-function was proposed for discrete-time systems with multiplicative and additive noises. The authors of [ 26 ] discussed the gap between model-based and model-free model algorithms on LQR. In our recent work [ 40 ] , we applied an off-policy RL method to study the stochastic H_{\infty} control problem with unknown system model. As said in [ 29 ] , Q-function learning-based optimal policy is only guaranteed for finite Markov decision process, and this limitation poses significant data storage requirements for complex systems and makes its practical applications face challenges. It can be found that, most of Q-learning algorithms lack of a solid theoretical analysis or scalability and are dependent on the persistent excitation assumption. To address these shortcomings, a novel primal-dual Q-learning framework for the LQR problem was recently introduced [ 18 ] and further developed [ 20 ] to investigate the SLQR problem with additive Gaussian white noises. In [ 20 ] , the random variables are assumed to be Gaussian white noises, and the cost functions are quadratic with a discount factor \gamma belonging to 0<\gamma<1 . Primal-dual RL method clarifies the essential relations among Q-learning algorithms, off-line PI algorithm and LQR optimization based on semidefinite programming [ 32 ] . Primal-dual RL algorithms [ 18 , 20 , 34 ] possess the advantages of a fast convergence and convenience for handling higher dimensional systems. However, to the best of our knowledge, up to now, no literature has succeeded in solving the model-free optimal policy design for the SLQR of discrete-time multiplicative noise systems from the primal-dual perspective. This paper aims to explores the partially model-free RL algorithms of SLQR in linear discrete-time stochastic systems with multiplicative noises by employing the primal-dual approach. This paper can be viewed as a non-trivial extension of [ 18 ] to stochastic multiplicative noise systems due to that there have essential differences between deterministic and stochastic systems. In fact, in order to develop a parallel frame to deterministic primal-dual model-free algorithms, we have to apply our previously introduced new definitions and theorems such as exact observability, exact detectability, Popov-Belevith -Hautus (PBH) criteria for eigenvector test of exact observability and exact detectability, as well as generalized Lyapunov theorems for the asymptotical stability in mean-square (ASMS) sense [ 11 , 37 ] , while related definitions and results of continuous-time Itô systems can be found in [ 36 , 38 , 39 ] . The main contributions of this paper are as follows: (1) We propose a novel off-line PI to solve the GARE from the concerned SLQR, and a strict convergence proof is also presented. More importantly, we point out that this PI algorithm has quadratic convergence speed; see Remark 3 . Our off-line PI algorithm can be viewed as a discretized version of [ 39 ] , which can also be viewed as an extension of classical Kleinman iteration algorithm [ 13 ] . (2) When the drift term coefficients are unknown, we propose a novel primal-dual optimization algorithm to obtain a partially model-free SLQR optimal policy design. As corollaries, all results of the deterministic system [ 18 ] can be obtained. When the diffusion term coefficients are also unknown, the fully primal-dual model-free SLQR optimal policy design remails unsolved. (3) Compared with the persistent excitation condition-based Q-learning algorithm [ 23 ] , our primal-dual-based algorithm can quickly converge to the optimal solution. Moreover, the designed algorithm is obtained by solving the Karush-Kuhn-Tucker (KKT) condition, which not only demonstrates the equivalence with respect to classical PI and Q-learning algorithms [ 19 ] , but also provides a rigorous convergence analysis for RL design of SLQR. The organization

Read the complete text