Ergodic-Risk Constrained Policy Optimization: The Linear Quadratic Case

Shahriar Talebi, Na Li · 2025

Read Ergodic-Risk Constrained Policy Optimization: The Linear Quadratic Case by Shahriar Talebi, Na Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-05878.txt.

Introduction

Talebi and Li add long-run risk limits to a classic control problem, in a 2025 paper. Controllers are usually judged on average performance, but rare large disturbances can matter as much. Their new criterion captures cumulative risk over time using limit theorems from probability, and it stays well behaved even for heavy-tailed noise, as long as the noise has a finite fourth moment.

They fold this risk measure into the linear quadratic regulator as a constraint, then solve the problem with a primal-dual method that optimizes average performance while respecting the risk cap. Simulations show the resulting controller also tames the long-run variance, shrugging off sporadic large shocks.

Catalog record

CREATORShahriar Talebi, Na Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-05878

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Ergodic-Risk Constrained Policy Optimization: The Linear Quadratic Case Thanks: The authors are with the School of Engineering and Applied Sciences, Harvard University, Cambridge, USA. Emails: [email protected] and [email protected] . This work is supported by NSF AI Institute 2112085. Shahriar Talebi, Na Li Abstract Risk-sensitive control balances performance with resilience to unlikely events in uncertain systems. This paper introduces ergodic-risk criteria , which capture long-term cumulative risks through probabilistic limit theorems. By ensuring the dynamics exhibit strong ergodicity, we demonstrate that the time-correlated terms in these limiting criteria converge even with potentially heavy-tailed process noises as long as the noise has a finite fourth moment. Building upon this, we proposed the ergodic-risk constrained policy optimization which incorporates an ergodic-risk constraint to the classical Linear Quadratic Regulation (LQR) framework. We then propose a primal-dual policy optimization method that optimizes the average performance while satisfying the ergodic-risk constraints. Numerical results demonstrate that the new risk-constrained LQR not only optimizes average performance but also limits the asymptotic variance associated with the ergodic-risk criterion, making the closed-loop system more robust against sporadic large fluctuations in process noise. I Introduction Optimizing average performance, as is typical in standard stochastic optimal control, often fails to yield effective policies for decision making in stochastic environments where deviations from expected outcomes carry significant risk; e.g. in financial markets [ 1 ] , safe robotics and autonomous systems [ 2 ] , and healthcare [ 3 ] . As such, incorporating risk measures become vital in such decision making problems for balancing the performance with resilience to rare events. While robust control frameworks (e.g. the mixed \mathcal{H}_{2} - \mathcal{H}_{\infty} in [ 4 ] ), focus on incorporating the worst-case scenario performance (e.g. \mathcal{H}_{\infty} -norm) as a constraint, they can be overly conservative (or occasionally unfeasible) when those unlikely events are (possibly) unbounded. Risk-aware approaches, on the other hand, offer a (probabilistic) compromise by building on available stochastic priors to manage both risk and performance, simultaneously. Consequently, there have been significant efforts [ 5 , 6 , 7 , 8 ] in developing risk-aware decision making frameworks using tools like Conditional Value at Risk (CVaR) [ 1 ] , Markov Risk Measure [ 9 ] , and Entropic Value at Risk (EVaR) [ 10 ] , offering a better balance by considering both risk and average performance. These measures are often deployed on finite-horizon variables [ 8 ] , under finite first-hitting time [ 7 ] , and/or in finite state-space models [ 6 ] . While these settings avoid complications regarding limiting probabilities, they may not fully capture long-term risk associated with the stochastic behaviors, especially in unbounded general-state Markov processes–see the recent survey [ 11 ] . Also at the stationary limits of the process, the optimal policy that minimizes the worst-case CVaR of the quadratic cost is shown to be equivalent to that of the lqr ( lqr ) optimal [ 12 ] . This is yet another evidence suggesting that risk-sensitive design is particularly critical in nonstationary processes, where their statistical properties are still changing over time. Among these, the folklore risk-sensitive framework by Whittle [ 5 ] , aka Linear Exponential Quadratic Gaussian (LEQG), handles the general unbounded, nonstationary setting–which (in certain parameter regimes) can be interpreted as optimizing a specific mixture of the average performance and its higher moments (e.g. variance). However, the Gaussian noise (with finite moments of all orders) is critical for the exponentiation to be well-defined, and thus does not capture cases with heavy-tailed noise distributions modeling rare events. This motivated [ 13 ] to introduce a framework for constraining the uncertainty in the “state-related portion” of the finite-horizon lqr cost, which is then extended to infinite-horizon through policy optimization techniques [ 14 ] . This paper considers the stochastic Linear Time Invariant (LTI) model with (unbounded) heavy-tailed process noise, providing a framework for risk-aware decision-making in unbounded , nonstationary Markov processes. We introduce ergodic-risk criteria to address risks in the long-term stochastic behavior, accounting for extreme deviations beyond mean performance ( Section II ). Built upon this, we propose the ergodic-risk constrained policy optimization which incorporates the ergodic-risk constraints in the classical LQR framework. By ensuring a strong ergodicity of the process [ 15 ] , we handle system’s state correlations and characterize quadratic ergodic-risk criteria as long as the process noise has finite fourth moment ( Section III ). This enables us to address long-term risk in non-stationary processes, previously excluded from the literature (see e.g. [ 12 ] ). We then consider a primal-dual policy optimization method based on strong duality that optimizes the average performance while meeting the risk constraints ( Section IV ), with convergence guarantee in Section IV-D . Finally, we demonstrate the numerical performance of the algorithm over randomly sampled problem instances and contrast the quality of the ergodic-risk optimal policy against the LQR optimal policy ( Section IV-E ). Finally, our ergodic-risk analysis is fundamentally different than that of [ 13 , 14 ] and in fact, the proposed Ergodic-risk framework ( Theorem 4 ) generalizes to provably capture the long-term risk associated with heavy-tailed noise for any quadratic functional of both the states and the inputs. However similar to [ 13 , 14 ] , the strong duality ( Section IV-B ) and the resulting primal-dual algorithm ( Section IV-C ) are established only for the case in which the risk functional does not explicitly depend on the input signal. II Ergodic-Risk Criteria and Problem Setup Consider the discrete-time stochastic linear system, {X}_{t+1}=A{X}_{t}+B{U}_{t}+H{W}_{t+1},\quad t\geq 0, (1) where A\in\mathbb{R}^{n\times n} , B\in\mathbb{R}^{n\times m} , and H\in\mathbb{R}^{n\times d} are system parameter matrices, {X}_{t} and {U}_{t} denote the stochastic state and input (vectors), respectively. Also, {W}_{t} is denoting the process noise and {X}_{0} is the initial state vectors. {W}_{t} and {X}_{0} are independently sampled from zero-mean probability distributions \mathbb{P}_{{W}} and \mathbb{P}_{0} with covariances \Sigma_{W} and \Sigma_{0} , respectively. The history of state-input trajectory up to time t is denoted by {\mathcal{H}}_{t}=\{{X}_{j},{U}_{j}\}_{j=0}^{t} , and \mathcal{F}_{t}=\sigma({\mathcal{H}}_{t}) denotes the \sigma -algebra generated by {\mathcal{H}}_{t} . Each {W}_{t} is measurable with respect to \mathcal{F}_{t} , denoted by {W}_{t}\in\mathcal{F}_{t} . Let \mathcal{F}_{-1} denote the trivial \sigma -algebra, and \mathcal{F}_{t-1}\vee\sigma\{{X}_{t}\} denote the smallest \sigma -algebra containing both \mathcal{F}_{t-1} and \sigma\{{X}_{t}\} . At each time t\geq 0 , we can apply an admissible input {U}_{t}\in\mathcal{L}_{2}(\mathcal{F}_{t-1}\vee\sigma\{{X}_{t}\}) , i.e. a square-integrable, measurable function with respect to \mathcal{F}_{t-1}\vee\sigma\{{X}_{t}\} , and then measure the next state {X}_{t+1} . We restrict ourselves to stationary Markov policies \pi , measurable mappings independent of time t that generate an input sequence \{{U}_{t}=\pi({\mathcal{H}}_{t-1},{X}_{t})\}_{t} such that {U}_{t} is admissible for all time t and does not depend on the history {\mathcal{H}}_{t-1} –see [ 15 , 16 ] for further details regarding probabilistic notions. We require the process noise and initial states to be uncorrelated across time, i.e., {\mathbb{E}\,}[{X}_{0}{W}_{t}^{\intercal}]=0 , and {\mathbb{E}\,}[{W}_{t+\tau}{W}_{t}^{\intercal}]=0,\;\forall t,\tau\geq 1 , so that {W}_{t} is independent of \mathcal{F}_{t-1} for all t\geq 1 . For simplicity, we pose the following assumption: Assumption 1 . The sequence \{{W}_{t}\} is i.i.d. samples of a common zero-mean probability measure \mathbb{P}_{W} that is non-singular with respect to Lebesgue measure on \mathbb{R}^{d} and has a non-trivial density, with a finite covariance \Sigma_{W}\succ 0 . ∎

Read the complete text