Actor–Critic Learning for Risk-Constrained Linear Quadratic Regulation
Weijian Li, Andreas A. Malikopoulos · 2025
Read Actor–Critic Learning for Risk-Constrained Linear Quadratic Regulation by Weijian Li, Andreas A. Malikopoulos (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22267.txt.
Introduction
Weijian Li and Andreas A. Malikopoulos add a conscience about risk to the workhorse of control theory. The linear quadratic regulator balances state error against effort with elegant mathematics. It says nothing about how wildly the state may swing, and in safety-critical systems the swings are the danger.
Their risk-constrained version appends a statistical limit on state variability to the classical problem over an infinite horizon. Reformulated as a max-min game, the problem yields an actor-critic learning scheme that needs no model of the system and learns online. The critic prices actions by temporal-difference learning, the actor improves the policy by gradient steps, and a dual variable climbs to enforce the risk cap. The three updates run on separate time scales, the standard recipe for keeping such a trio stable while it learns.
Catalog record
| CREATOR | Weijian Li, Andreas A. Malikopoulos |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-22267
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Actor–Critic Learning for Risk-Constrained Linear Quadratic Regulation Weijian Li, Andreas A. Malikopoulos Abstract In this paper, we investigate the infinite-horizon risk-constrained linear quadratic regulator problem (RC-LQR), which augments the classical LQR formulation with a statistical constraint on the variability of the system state to incorporate risk awareness, a key requirement in safety-critical control applications. We propose an actor–critic learning algorithm that jointly performs policy evaluation and policy improvement in a model-free and online manner. The RC-LQR problem is first reformulated as a max–min optimization problem, from which we develop a multi–time–scale stochastic approximation scheme. The critic employs temporal-difference learning to estimate the action–value function, the actor updates the policy parameters via a policy gradient step, and the dual variable is adapted through gradient ascent to enforce the risk constraint. I Introduction Reinforcement learning (RL), aiming to learn optimal policies directly from data in the absence of explicit model knowledge, has gained significant attention in recent years. Its versatility has enabled progress across a wide range of sequential decision-making problems, including poker games [ 1 ] , robotic control [ 2 ] , and cyber–physical systems [ 3 ] . Recent studies have investigated the connection between learning and control by introducing information–state representations that decouple estimation and control [ 4 , 5 ] , and by developing integrated learning–and–control frameworks for systems with unknown dynamics [ 6 ] . Within the control community, there has been an increased interest in establishing a rigorous theoretical foundation for RL in linear systems—most notably through the linear quadratic regulator (LQR) problem, which seeks to regulate an unknown linear system to minimize a quadratic cost function [ 7 , 8 ] . The LQR problem provides a mathematically tractable framework for studying the interplay between learning and control and serves as a canonical benchmark for analyzing the stability, optimality, and convergence of data-driven control algorithms. However, the classical formulation is risk-neutral , as performance is evaluated solely through the expected cost, overlooking low-probability but high-impact disturbances that can cause unsafe behaviors in safety-critical systems [ 9 ] . This shortcoming has motivated the development of risk-aware formulations that explicitly regulate the variability of the closed-loop behavior. Early work introduced an exponential cost transformation to capture risk sensitivity [ 9 , 10 , 11 ] , though such problems can become ill-posed under general noise distributions. Conditional value-at-risk formulations [ 12 , 13 ] offer a more coherent treatment of tail events but often rely on tractable approximations. The risk-constrained LQR (RC-LQR) framework [ 14 ] augments the standard LQR objective with a constraint on the predictive variance of the state penalty, yielding a closed-form optimal policy that accommodates arbitrary noise models. Recent extensions established global convergence guarantees for policy optimization methods [ 15 , 16 ] . Advances in reinforcement learning (RL) have been largely driven by policy optimization methods, such as policy gradient and actor–critic algorithms [ 17 , 18 , 19 , 20 ] , which directly parameterize and optimize control policies. These approaches efficiently handle high-dimensional, continuous spaces and are simple to implement. Theoretical analyses have also been developed for linear control systems [ 21 , 22 , 23 , 24 ] , while primal–dual policy gradient algorithms have been proposed for the RC-LQR problem [ 16 ] . Among these methods, the actor–critic framework, which performs policy evaluation and improvement simultaneously, has received extensive attention [ 19 , 25 , 26 , 27 ] . Compared with sample-based policy gradient schemes, actor–critic algorithms typically converge faster due to variance reduction in gradient estimation. However, their extension to constrained RL problems remains relatively unexplored. In this paper, we develop an actor–critic algorithm for the RC-LQR problem and characterize its theoretical properties. The main contributions of this work are summarized as follows: (1) We study a model-free LQR problem subject to a risk constraint. Unlike the classical LQR formulation [ 8 , 21 , 27 ] , the proposed setting minimizes a quadratic cost and also regulates the statistical variability of the system state. While similar formulations appear in [ 14 , 15 , 28 ] , our focus is on the model-free regime. (2) We propose a multi–time–scale actor–critic algorithm under a primal–dual framework. The critic employs temporal-difference learning, the actor performs policy-gradient updates, and the dual variable evolves via gradient ascent. The proposed scheme generalizes two–time–scale actor–critic algorithms [ 19 , 25 , 26 ] to constrained RL and, unlike the sample-based policy gradient method of [ 16 ] , integrates policy evaluation and improvement in an online and model-free manner. (3) Using stochastic approximation theory, we establish almost sure asymptotic convergence of all iterates. This result extends classical actor–critic convergence guarantees [ 19 , 25 , 26 ] by incorporating a dual update in the learning loop and strengthens prior results such as [ 16 ] , which established convergence only with high probability. The remainder of the paper is structured as follows. In Section II, we review the necessary preliminaries that form the basis of our analysis. In Section III, we present the formulation of the RC-LQR problem. In Section IV, we develop the proposed actor–critic algorithm under a primal–dual framework, and in Section V, we establish its asymptotic convergence using stochastic approximation theory. In Section VI, we provide simulation results that illustrate the performance of the algorithm, and in Section VII, we draw concluding remarks. Notation: Let \mathbb{N}:=\{1,2,\dots\} denote the set of positive integers, and fix dimensions n,m\in\mathbb{N} . Let \mathbb{R}_{+}:=[0,\infty) , \mathbb{R}^{n} be the set of n -dimensional real column vectors, \mathbb{R}^{m\times n} be the set of m\times n real matrices, and I_{n} the n\times n identity. Let \mathbb{S}^{n}\subset\mathbb{R}^{n\times n} denote the set of real n\times n symmetric matrices, and write \mathbb{S}_{++}^{n} ( \mathbb{S}_{+}^{n} ) for the positive (semi-)definite cones in \mathbb{S}^{n} . We use X\succ(\succeq)\,0 to indicate X\in\mathbb{S}_{++}^{n} ( \mathbb{S}_{+}^{n} ). The transpose and trace operators are (\cdot)^{\top} and \mathrm{tr}(\cdot) , respectively. For A,B\in\mathbb{R}^{m\times n} , the Frobenius inner product is \langle A,B\rangle_{F}:=\mathrm{tr}(A^{\top}B) , with Frobenius norm \|A\|_{F}:=\sqrt{\langle A,A\rangle_{F}} . For x\in\mathbb{R}^{n} , \|x\| denotes the Euclidean ( \ell_{2} ) norm. For X\in\mathbb{S}^{n} , \mathrm{svec}(X) denotes the vectorization of the upper triangular part of X , with off-diagonal entries weighted by \sqrt{2} . Given a nonempty closed convex set \mathcal{X}\subset\mathbb{R}^{n} , the Euclidean projection is \Gamma_{\mathcal{X}}(y):=\arg\min_{x\in\mathcal{X}}\|y-x\|, for all y\in\mathbb{R}^{n}. II Preliminaries In this section, we review fundamental results from stochastic approximation theory that will be used in the convergence analysis. Consider a sequence \{x_{t}\}\subset\mathbb{R}^{n} generated by x_{t+1}=x_{t}+\gamma_{t}[h(x_{t})+M_{t+1}],\qquad t\geq 0,~x_{0}\in\mathbb{R}^{n}, (1) where h(\cdot) is a deterministic map, \{M_{t}\} is a noise sequence, and \{\gamma_{t}\} denotes the step size. Assumption 1 The recursion ( 1 ) satisfies: 1. h:\mathbb{R}^{n}\!\to\!\mathbb{R}^{n} is Lipschitz continuous. 2. The step sizes obey \sum_{t}\gamma_{t}=\infty and \sum_{t}\gamma_{t}^{2}<\infty . 3. \{M_{t}\} is a martingale difference sequence with M_{0}=0 and \mathbb{E}[M_{t+1}\mid x_{s},M_{s},s\!\leq\!t]=0 , and there exists K>0 such that \mathbb{E}[\|M_{t+1}\|^{2}\mid x_{s},M_{s},s\!\leq\!t]\leq K(1+\|x_{t}\|^{2}). The asymptotic behavior of ( 1 ) is characterized by the limiting ordinary differential equation (ODE) \dot{x}=h(x). (2) If ( 2 ) admits a unique globally asymptotically stable equilibrium x^{*} , the following results hold [ 29 ] . Lemma 1 Under Assumption 1 , if \sup_{t}\|x_{t}\|<\infty a.s., then x_{t}\to x^{*} a.s. Lemma 2