Exploratory Randomization for Discrete-Time Linear Exponential Quadratic Gaussian (LEQG) Problem

Sébastien Lleo, Wolfgang Runggaldier · 2025

Read Exploratory Randomization for Discrete-Time Linear Exponential Quadratic Gaussian (LEQG) Problem by Sébastien Lleo, Wolfgang Runggaldier (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-06275.txt.

Introduction

Risk-sensitive investing bends the classic control problem: losses hurt more than gains please, in a precise exponential way. Sébastien Lleo and Wolfgang Runggaldier explore in 2025 a randomized version of that problem in discrete time, where the controller deliberately randomizes its actions to learn. A duality links free energy and relative entropy, two faces of the same coin.

Through that lens, the risk-sensitive problem becomes an ordinary risk-neutral one carrying an entropy penalty for wandering too far. Dynamic programming then supplies the solution. The result also justifies, from first principles, the entropy regularization that practitioners already add when they randomize control.

Catalog record

CREATORSébastien Lleo, Wolfgang Runggaldier
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE85 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-06275

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Exploratory Randomization for Discrete-Time Linear Exponential Quadratic Gaussian (LEQG) Problem Sébastien Lleo, Wolfgang Runggaldier Abstract We investigate exploratory randomization for an extended linear-exponential-quadratic-Gaussian (LEQG) control problem in discrete time. This extended control problem is related to the structure of risk-sensitive investment management applications. We introduce exploration through a randomization of the control. Next, we apply the duality between free energy and relative entropy to reduce the LEQG problem to an equivalent risk-neutral LQG control problem with an entropy regularization term, see, e.g. Dai Pra et al. (1996) , for which we present a solution approach based on Dynamic Programming. Our approach, based on the energy-entropy duality may also be considered as leading to a justification for the use, in the literature, of an entropy regularization when applying a randomized control. keywords : risk-sensitive control, free energy-entropy duality, stochastic differential games, Linear Exponential Quadratic Gaussian control, Linear Quadratic Gaussian games, exploratory controls. 1 Introduction Real-world stochastic control problems are often characterized by incomplete information about the state or the model. Traditionally, filtering has been used to address incomplete information ( Bensoussan and Van Schuppen, 1984 ; Bensoussan and Runggaldier, 1987 , see) . More recently, reinforcement learning has emerged as a promising alternative ( Wang et al., 2020 ; Jia and Zhou, 2023 ; Jia and Zhou, 2022a ; Jia and Zhou, 2022b ; Hambly et al., 2021 , see) . These techniques rely on iterative interactions with the environment, balancing exploration—randomized actions to acquire information—and exploitation—actions that maximize a reward function based on available data. Although reinforcement learning evolved from Markov Decision Processes, integrating it with stochastic control remains challenging. Wang et al. (2020) made a key advance in this direction by showing how relaxed controls create exploratory policies inside standard (risk-neutral) control problems formulated in continuous time. Adding an entropy penalization/regularization to the control criterion further led to a trade-off between exploration and exploitation. A particularly important class of stochastic control problems is that of risk-sensitive control ( Whittle, 1990 ; Bensoussan et al., 1998 ; Bielecki et al., 2022 ; Bäuerle and Rieder, 2014 ; Bäuerle and Jaśkiewicz, 2024 , see) , with linear exponential of quadratic Gaussian (LEQG) problems as a notable subclass ( Jacobson, 1973 ; Whittle, 1981 ; Bensoussan and van Schuppen, 1985 ; Bensoussan, 1992 , see) . LEQG problems are especially relevant in financial applications (see e.g. Bielecki and Pliska, 1999 ; Kuroda and Nagai, 2002 ; Davis and Lleo, 2008 ; Davis and Lleo, 2014 ; Pitera and Stettner, 2023 ; Davis and Lleo, 2020 ; Davis and Lleo, 2021 ; Lleo and Runggaldier, 2024 , and references therein) . In this paper, we consider a discrete-time risk-sensitive control problem of the LEQG type drawing inspiration from the results of Wang et al. (2020) . Specifically, we introduce randomized controls to emulate exploration. We generate these randomized controls by perturbing a deterministic control with an additive Gaussian distribution. Here, the mean is the deterministic control that captures exploitation, while the variance, which provides an additional (antagonizing) control, models exploration. While implementing additive randomization in continuous-time settings poses significant technical challenges 1 1 1 It is worth noting that alternative continuous-time approaches, such as those developed by Huyên Pham and coauthors (for a recent contribution see e.g. Denkert et al. (2024) ), provide valuable insights. , the discrete-time framework allows us to focus more directly on the exploration aspect. Our main contribution is to solve the randomized LEQG problem using a duality between free energy and entropy Dai Pra et al. (1996) . This approach reduces the risk-sensitive randomized stochastic control problem to a risk-neutral stochastic game problem with an entropy penalization. This penalization comes from two sources: (i) the randomized controls and (ii) the transition from risk-sensitive to risk-neutral. While Wang et al. (2020) introduce an entropy penalization term as a regularisation, the entropy penalization in our approach arises naturally from the energy-entropy duality, providing a conceptual justification for its use. Unlike Wang et al., who remain in the domain of standard risk-neutral stochastic control problems, our formulation yields a stochastic game. In this game, the decision-maker selects controls to exploit available information, while an antagonistic player determines the variance of the control perturbations, governing the degree of exploration. The introduction of randomization into stochastic control problems is primarily motivated by its applicability in reinforcement learning. A possible methodology to implement reinforcement learning in our setup is via a policy gradient approach, whereby learning is applied to a parametrized optimal policy ( Hambly et al., 2021 , see for instance) . However, our paper focuses on the foundational modeling steps that precede the development of the reinforcement learning algorithm. Specifically, we demonstrate how to reduce the randomized LEQG problem to a randomized stochastic LQG game via the energy-entropy duality. We also derive the optimal policy and value function. Both turn out to be parametrized by the initial data of the model via a backward recursion, as stated in Theorem 3.5 . If the model parameters are unknown and are updated recursively by some method that we only sketch briefly, the backward recursion has to be recomputed. In this sense, our approach retains conceptual similarities with the policy gradient framework, and in particular, actor-critic methods as further illustrated in Section 4.3 below. 2 Setup Consider a discrete-time grid with times t=0,1,\ldots,T , where T<\infty , and let \left(\Omega,\mathcal{F},\left(\mathcal{F}\right)_{t=0,\ldots,T},\mathbb{P}\right) be a filtered complete probability space, where \mathcal{F} is the natural filtration and \mathbb{P} is a reference probability measure. 2.1 Linear Exponential Quadratic (LEQG) Control Problem We start with a classical LEQG problem with standard noise under the measure \mathbb{P} . The dynamics of the controlled state x_{t}\in\mathbb{R}^{d_{x}},d_{x}\in\mathbb{N} is \displaystyle x_{t+1}=a+Ax_{t}+Bu_{t}+w_{t},\quad t=0,\ldots,T-1, (2.1) where a\in\mathbb{R}^{d_{x}} , A\in\mathbb{R}^{d_{x}\times d_{x}} , B\in\mathbb{R}^{d_{x}\times d_{u}} . Here, u_{t}\in\mathbb{R}^{d_{u}},d_{u}\in\mathbb{N} is the control and w_{t} is a d_{x} -dimensional Gaussian system noise, that is w_{t}\sim\mathcal{N}\left(0,\Lambda_{t}\right) where \Lambda_{t}\in\mathbb{R}^{d_{x}\times d_{x}},t=0,\ldots,T-1 . The system noises are serially independent so \mathbf{E}\left[(w_{t})^{\prime}w_{s}\right]=0,t\neq s . The risk-sensitive criterion J to be maximized is defined as \displaystyle J(u;T,\theta)=-\frac{1}{\theta}\ln\mathbf{E}\left[e^{\theta G_{T}}\right], (2.2) where \theta\in(-1,0)\cup(0,\infty) is the risk-sensitivity, G_{T} is the cost defined as \displaystyle G_{T}:=\sum_{t=0}^{T-1}\left(x_{t}^{\prime}Mx_{t}+u_{t}^{\prime}N_{t}u_{t}+u_{t}^{\prime}Qx_{t}+x_{t}^{\prime}m+u_{t}^{\prime}n\right)+x_{T}^{\prime}M_{T}x_{T}+x_{T}^{\prime}m_{T}, (2.3) where M\in\mathbb{R}^{d_{x}\times d_{x}} is semipositive definite and symmetric, N_{t}\in\mathbb{R}^{d_{u}\times d_{u}},t=1,\ldots,T-1 is also semipositive definite and symmetric, Q\in\mathbb{R}^{d_{u}\times d_{x}} , m\in\mathbb{R}^{d_{x}} , n\in\mathbb{R}^{d_{u}} , M_{T}\in\mathbb{R}^{d_{x}\times d_{x}} , and m_{T}\in\mathbb{R}^{d_{x}} . For convenience, we also introduce the exponentially transformed criterion I defined as \displaystyle I(u;T,\theta)=e^{-\theta J(u;T,\theta)}=\mathbf{E}\left[e^{\theta G_{T}}\right], (2.4) which is to be minimized. Remark 1 . The standard formulation of risk-sensitive control problems does not contain a constant a in the state dynamics or a cross-term u_{t}Qx_{t} in the running cost. These additions are required in investment management applications of continuous-time risk-sensitive control, see for example Davis and Lleo (2014) . Including them ensures that our model can be used to solve discretized versions of the risk-sensitive asset management model by Bielecki and Pliska (1999) ; Kuroda and Nagai (2002) and of the risk-sensitive benchmarked asset management model by Davis and Lleo (2008) . 2.2 Exploration via Randomized Controls The classical LEQG model may not provide a perfect description of reality, and we may not have sufficient data to estimate our model parameters with a high degree of accuracy. We employ exploratory randomized controls to address

Read the complete text