Data-Driven Exploration for a Class of Continuous-Time Indefinite Linear–Quadratic Reinforcement Learning Problems

Yilie Huang Xun Yu Zhou · 2025

Read Data-Driven Exploration for a Class of Continuous-Time Indefinite Linear–Quadratic Reinforcement Learning Problems by Yilie Huang Xun Yu Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-00358.txt.

Introduction

Yilie Huang and Xun Yu Zhou study reinforcement learning for control problems in twenty twenty-five. Their class of problems runs in continuous time with linear dynamics and quadratic costs. The costs here are indefinite, and the noise depends on both state and control. States are scalar, and there are no running control rewards.

Earlier work on this class used fixed exploration schedules. Those need heavy tuning and ignore how learning is going. The authors propose a model-free method that adapts exploration as it runs. The critic adjusts entropy regularization while the actor adjusts policy variance. Even with that freedom, the method keeps a sublinear regret bound. It matches the best known model-free results for this class.

Catalog record

CREATORYilie Huang Xun Yu Zhou
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE77 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-00358

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Data-Driven Exploration for a Class of Continuous-Time Indefinite Linear–Quadratic Reinforcement Learning Problems Yilie Huang Xun Yu Zhou Abstract We study reinforcement learning (RL) for the same class of continuous-time stochastic linear–quadratic (LQ) control problems as in [ 1 ] , where volatilities depend on both states and controls while states are scalar-valued and there are no running control rewards, the last feature also leading to the so-called indefinite LQ controls. We propose a model-free, data-driven exploration mechanism that adaptively adjusts entropy regularization by the critic and policy variance by the actor. Unlike the constant or deterministic exploration schedules employed in [ 1 ] , which require extensive tuning for implementations and ignore learning progresses during iterations, our adaptive exploratory approach boosts learning efficiency with minimal tuning. Despite its flexibility, our method achieves a sublinear regret bound that matches the best-known model-free results for this class of LQ problems, which were previously derived only with fixed exploration schedules. Numerical experiments demonstrate that adaptive explorations accelerate convergence and improve regret performance compared to the non-adaptive model-free and model-based counterparts. Keywords: S tochastic Indefinite Linear–-Quadratic Control, Continuous-Time Reinforcement Learning, Actor, Critic, Data-Driven Exploration, Model-Free Policy Gradient, Regret Bounds. 1 Introduction Linear–quadratic (LQ) control is a cornerstone of optimal control theory, widely applied in engineering, economics, and robotics among many others due to its analytical tractability and practical relevance. Its theory has been extensively developed in the classical model-based setting where all the LQ coefficients are known and given [ 2 , 3 ] . Assuming the objective is to minimize the cost functional, a key assumption in the LQ theory is that the control cost weighting matrix must be positive definite. Intuitively, this is because if increasing the level of control is costless or even beneficial, then the optimal control would just be infinitely large leading to an ill-posed problem. [ 4 ] finds, however, that certain stochastic control LQ control problems with indefinite cost matrices—including those associated with control—can still be well-posed if the diffusion coefficients depend on the control variable. This seemingly surprising result actually has a simple explanation: while benefiting from an indefinite control cost, a larger control is disadvantageous in terms of increasing the level of uncertainty because the volatility term depends on control multiplicatively. Therefore, one needs to carefully choose the optimal level of control, rendering a well-posed optimization problem. [ 4 ] coins the term “indefinite stochastic LQ control” for such a problem, and establishes conditions for well-posedness and solvability via newly introduced generalized Riccati equations. Subsequent works extend the theory in several directions, including infinite-time horizons [ 5 ] and connections with linear matrix inequalities [ 6 ] , asymptotic properties of the Riccati solutions [ 7 ] , and semidefinite programming for numerical computations [ 8 ] . Research on indefinite LQ controls and the related generalized Riccati equations has been active to this day; see, e.g., [ 9 , 10 , 11 , 12 ] . All the research on indefinite LQ controls, and indeed most studies on stochastic controls, are undertaken in a model-based paradigm where the system coefficients and objective/cost functionals are fully known. In real-world applications, however, complete knowledge of model parameters is rarely available. Many environments exhibit LQ-like structure yet key parameters in dynamics and objectives may be partially known or entirely unknown. While one can use historical data to estimate those model parameters as commonly done in the so-called “plug-in” approach, estimating some of the parameters to a required accuracy may not be possible. For instance, it is statistically impossible to estimate a stock return rate for a very small period – a phenomenon termed “mean-blur” [ 13 , 14 ] . On the other hand, objective function values can be highly sensitive to estimated model parameters, rendering inferior performances or even complete irrelevance of model-based solutions [ 15 ] . Given the limitations of model-based controls, particularly the challenges associated with unknown model coefficients, reinforcement learning (RL) rises to provide a promising remedy. Simply put, RL is stochastic control with unknown model parameters. Yet it never attempts to estimate any model coefficients, and solves the problem in a fundamentally different way compared to the classical, model-based stochastic control. A centerpiece of RL is exploration, with which the agent interacts with the unknown environment and improves her policies. The simplest exploration strategy is perhaps the so-called \epsilon -greedy for bandits problem [ 16 ] . More sophisticated and efficient exploration strategies include intrinsic motivation methods that reward exploring novel or unpredictable states [ 17 , 18 , 19 , 20 , 21 , 22 , 23 ] , count-based exploration that encourages the agent to explore less frequently visited states [ 24 , 25 ] , go-explore that explicitly stores and revisits promising past states to enhance sample efficiency [ 26 , 27 ] , imitation-based exploration which leverages expert demonstrations [ 28 ] , and safe exploration methods that rely on predefined safety constraints or risk models [ 29 , 30 ] . While these methods are shown to be successful in their respective contexts, they are all for discrete-time problems and seem difficult to be extended to the continuous-time counterparts. However, most real-life applications are inherently continuous-time with continuous state spaces and possibly continuous control spaces (e.g., autonomous driving, robot navigation, and video game playing). On the other hand, transforming a continuous-time problem into a discrete-time one upfront by discretization has shown to be problematic when the time step size becomes very small [ 31 , 32 , 33 ] . Thus, there is a pressing need for developing exploration strategies tailored to continuous-time RL, particularly and foremost in the LQ setting, for achieving both stability and learning efficiency. Recent works, such as [ 1 , 34 ] , have employed constant or deterministic exploration schedules to regulate exploration parameters and proved to achieve sublinear regret bounds. However, these approaches come with notable drawbacks. They require extensive manual tuning of the exploration hyperparameters, which considerably increases computational costs, yet these tuning efforts are typically not reflected in theoretical analyses and results including those concerning regret bounds. Moreover, pre-determined exploration strategies lack adaptability by nature, as they do not react to the incoming data nor to the current learning states, typically resulting in slower convergence and increased variance from excessive exploration or premature convergence to suboptimal policies from insufficient exploration. This paper presents a companion research of [ 1 ] . We continue to study the same class of stochastic LQ problems in [ 1 ] , which is also of an indefinite LQ type because control is absent in the objective functional. We develop a data-driven adaptive exploration framework for both critic (the value functions) and actor (the control policies). Specifically, we propose an adaptive exploration strategy that dynamically adjusts the entropy regularization parameter for the critic and the stochastic policy variance for the actor based on the agent’s learning progress. Moreover, we provide theoretical guarantees by proving the almost sure convergence of both the adaptive exploration and policy parameters, along with a sublinear regret bound of O(N^{\frac{3}{4}}) . This bound matches the best-known model-free result from [ 1 ] , which assumes a nonzero initial state

Read the complete text