Quadratic Truncated Random Return in Distributional LQR: Positive Definiteness, Density, and Log-Concavity
Ruyi Teng, Dan Wang, Wei Chen, Yulong Gao · 2025
Read Quadratic Truncated Random Return in Distributional LQR: Positive Definiteness, Density, and Log-Concavity by Ruyi Teng, Dan Wang, Wei Chen, Yulong Gao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-23332.txt.
Introduction
Ruyi Teng, Dan Wang, Wei Chen, and Yulong Gao study the random return in distributional linear quadratic control, where the whole distribution of outcomes matters, not just the average cost. In this framework the random return has an exact closed form over infinitely many random variables. Prior work approximated it with finitely many variables, called truncation.
The authors analyze a quadratic truncated version and prove positive definiteness, a density result, and log-concavity of the return. Those properties make the truncated model tractable for risk-aware control. The work firms up theory for controllers that care about spread and downside, not only means.
Catalog record
| CREATOR | Ruyi Teng, Dan Wang, Wei Chen, Yulong Gao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-23332
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Quadratic Truncated Random Return in Distributional LQR: Positive Definiteness, Density, and Log-Concavity Ruyi Teng, Dan Wang, Wei Chen, Yulong Gao Abstract Distributional linear quadratic regulator (LQR) is a new framework that integrates the distributional reinforcement learning and classical LQR, which offers a new way to study the random return instead of the expected cost. Unlike iterative approximation using dynamic programming in the DRL, a closed-form expression for the random return can be exactly characterized in the distributional LQR, which is defined over infinitely many random variables. In recent work [ 1 , 2 ] , it has been shown that this random return can be well approximated by a finite number of random variables, which we call truncated random return. In this paper, we study the truncated random return in the distributional LQR. We show that the truncated random return can be naturally expressed in the quadratic form. We develop a sufficient condition for the positive definiteness of the block symmetric matrix in the quadratic form and provide the lower and upper bounds on the eigenvalues of this matrix. We further show that in this case, the truncated random return follows a positively weighted non-central chi-square distribution if the random disturbances admits Gaussian, and its cumulative distribution function is log-concave if the probability density function of the random disturbances is log-concave. I Introduction Linear Quadratic Regulator (LQR) [ 3 ] is one of the most fundamental problems in systems and control. Its importance can be reflected in a large number of practical applications, e.g., [ 4 , 5 ] . When the additive disturbance is stochastic, the optimal controller that minimizes the expected quadratic cost is a linear feedback controller, where the optimal feedback gain can be obtained by solving the Riccati equation. Inspired by distributional reinforcement learning (DRL) [ 6 ] , distributional LQR was developed in the recent work [ 1 , 2 ] to take into account the randomness of total return, beyond its expectation. The motivation of developing distributional LQR is the benefits of evaluating and exploiting the return distribution in the decision making that accounts for uncertainty, risk, and multi-modal outcomes [ 7 , 2 ] . In the DRL, it has been shown or demonstrated that the use of return distribution not only improves resilience against unexpected deviations but also improves the efficiency of exploration during learning, guiding policies toward reduced variance and stabilized behavior [ 8 ] . A key result of distributional LQR is the closed-form expression of the random return [ 1 , 2 ] , which is intractable for other stochastic systems, including finite Markov decision processes. The objective of this paper is to study the truncated random return in the distributional LQR. It was shown in [ 1 , 2 ] that truncated random return converges pointwise in distribution to the un-truncated random return while providing improved computational tractability. We show that the truncated random return can be naturally expressed as a quadratic form in random variables, which is a key concept in statistics and applied mathematics [ 9 , 10 ] . To the best of our knowledge, it is the first time to explore the positive definiteness of the block matrix in the quadratic truncated random return. This fundamental property reveals some important results: explicit from of density function and log-concavity. Our main contributions are summarized as follows. (1) We prove that the symmetric block matrix in the quadratic form of the truncated random return is positive definite under a sufficient condition. (2) We show that if the random disturbance is zero-mean Gaussian, the truncated random return follows a positively weighted non-central chi-square distribution. (3) We prove that if the probability density function (PDF) of the random disturbances is log-concave, the cumulative distribution function of the truncated random return is also log-concave. II Preliminaries and Problem Statement II-A Notation We denote by \mathbb{R} the set of real numbers and \mathbb{N} the set of natural numbers. For a symmetric matrix P , the notation P\succ 0 means that P is positive definite. For a matrix Q\in\mathbb{R}^{n\times n} , we denote by \left\|Q\right\| the spectral norm. To indicate that two random variables Z_{1} and Z_{2} are equal in distribution, we use the notation Z_{1}\mathop{=}\limits^{D}Z_{2} . For a random variable Z , \mathbb{E}[Z] denotes its expectation. For a real symmetric matrix H , \lambda_{\min}(H) denotes its minimal eigenvalue. Lemma II.1 [ 11 , Theorem 2] Let H=[H_{ij}] be a block matrix where H_{ij}\in\mathbb{R}^{n\times n} , i,j=1,\ldots,N . For each eigenvalue \lambda of H , there exists at least one 1\leq i\leq N such that (\left\|(H_{ii}-\lambda I)^{-1}\right\|)^{-1}\leq\sum_{j=1,j\neq i}^{N}\left\|H_{ij}\right\|. (1) Definition II.1 Let H=[H_{ij}] be a block matrix where H_{ij}\in\mathbb{R}^{n\times n} , i,j=1,\ldots,N . If the diagonal submatrices H_{ii} are nonsingular and \left(\left\|H^{-1}_{ii}\right\|\right)^{-1}\geq\sum_{j=1,j\neq i}^{N}\left\|H_{ij}\right\|,\forall i=1,\ldots,N, (2) then H is block diagonally dominant. If the strict inequality is valid, then H is block strictly diagonally dominant. II-B Canonical discounted LQR Consider a discrete time LTI system: x_{t+1}=Ax_{t}+Bu_{t}+v_{t} where x_{t}\in\mathbb{R}^{n} , u_{t}\in\mathbb{R}^{p} , and v_{t}\in\mathbb{R}^{n} are the state, control input, and exogenous disturbance, respectively. The exogenous disturbances v_{t} are assumed to be independent and identically distributed (i.i.d.) with bounded moments, sampled from an arbitrary distribution \mathcal{D} . The discounted LQR is to find a feedback controller u_{t}=\pi(x_{t}) by minimizing J(u)=\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^{t}\left(x_{t}^{\top}Qx_{t}+u_{t}^{\top}Ru_{t}\right)\right], (3) where Q\in\mathbb{R}^{n\times n} and R\in\mathbb{R}^{p\times p} are symmetric positive definite matrices, \gamma\in(0,1) is the discount factor. Given an initial state x_{0}=x and a linear feedback controller u_{t}=Kx_{t} , the value function is defined as V_{\pi}(x):=\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^{t}\left(x_{t}^{\top}(Q+K^{\top}RK)x_{t}\right)\right] , which fulfills the Bellman equation: V^{\pi}(x)=x^{\top}(Q+K^{\top}RK)x+\gamma\mathbb{E}_{x^{\prime}=(A+BK)x+v_{0}}\left[V^{\pi}(x^{\prime})\right], (4) where x^{\prime}=(A+BK)x+v_{0} is the next random state from x . In this case, the solution to the Bellman equation can be explicitly characterized in a quadratic form: V_{\pi}(x)=x^{\top}Px+q, where q is a constant and P is the solution to the Lyapunov equation P=Q+K^{\top}RK+\gamma(A+BK)^{\top}P(A+BK). (5) II-C Distributional LQR Unlike the traditional discounted LQR which primarily focuses on the expected return, the distributional LQR emphasizes the random return G^{\pi}(x) with a given feedback controller u_{t}=\pi(x_{t}) and an initial state x_{0}=x : \displaystyle G^{\pi}(x)=\sum_{t=0}^{\infty}\gamma^{t}\left(x_{t}^{\top}Qx_{t}+u_{t}^{\top}Ru_{t}\right). (6) It has been shown in [ 1 ] , if under the linear feedback controller \pi(x_{t})=Kx_{t} that K is stabilizing and satisfies \|A_{K}\|<1 with A_{K}=A+BK , the random return G^{K}(x) that fulfills the random-variable Bellman equation G^{K}(x)\mathop{=}^{D}x^{\top}Q_{K}x+\gamma G^{K}(x^{\prime}),\quad x^{\prime}=A_{K}x+v_{0} can be exactly characterized as follows: \displaystyle G^{K}(x)=x^{\top}Px+2\sum_{k=0}^{\infty}\gamma^{k+1}w_{k}^{\top}PA_{K}^{k+1}x (7) \displaystyle+\sum_{k=0}^{\infty}\gamma^{k+1}w_{k}^{\top}Pw_{k}+2\sum_{k=1}^{\infty}\gamma^{k+1}w_{k}^{\top}P\sum_{\tau=0}^{k-1}A_{K}^{k-\tau}w_{\tau}, where w_{k}\sim\mathcal{D} are independent from each other for all k\in\mathbb{N} and P is obtained from the Lyapunov equation ( 5 ). Since G^{K}(x) involves an infinite number of random variables w_{k} , one can approximate its distribution using a finite number of random variables: \displaystyle{G}^{K}_{N}(x)=x^{\top}Px+2\sum_{k=0}^{N-1}\gamma^{k+1}w_{k}^{\top}PA_{K}^{k+1}x \displaystyle+\sum_{k=0}^{N-1}\gamma^{k+1}w_{k}^{\top}Pw_{k}+2\sum_{k=1}^{N-1}\gamma^{k+1}w_{k}^{\top}P\sum_{\tau=0}^{k-1}A_{K}^{k-\tau}w_{\tau}, (8) where N\in\mathbb{N} is the number of random variables. It has shown in [ 1 ] that the sequence \{{G}^{K}_{N}(x)\}_{N\in\mathbb{N}} converges pointwise in distribution to G^{K}(x) , \forall x\in\mathbb{R}^{n} . More specifically, if the PDF of w_{k} is bounded, \mathbb{E}[w_{k}^{\top}w_{k}]\leq\sigma^{2} , for all k\in\mathbb{N} , and the feedback gain K satisfies \left\|A_{K}\right\|<1 , the sup difference between the CDF F of {G}^{K}(x) and the CDF {F}_{N} of {G}_{N}^{K}(x) is bounded by \displaystyle\sup_{g}|{F}(g)-{F}_{N}(g)|\leq c_{0}\gamma^{N}, (9) where c_{0} is a constant that depends on the system parameters and the initial state value x . In this paper, we will focus on the truncated random return