Tractable Representations for Convergent Approximation of Distributional HJB Equations
Julie Alhosh, Harley Wiltzer, David Meger · 2025
Read Tractable Representations for Convergent Approximation of Distributional HJB Equations by Julie Alhosh, Harley Wiltzer, David Meger (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-05563.txt.
Introduction
Julie Alhosh, Harley Wiltzer and David Meger make a hard equation of continuous-time reinforcement learning solvable in practice. Beyond average rewards, agents may care about the whole spread of returns, for risk-aware choices. A distributional Hamilton-Jacobi-Bellman equation describes that spread over continuous time, but its solutions cannot be stored or computed exactly.
The team finds a condition on how return distributions are summarized: if the summary map has the right topological property, then fitting the summaries fits the true solution closely. The quantile summary, common in the field, passes that test. That certifies an efficient approximation method for risk-aware learning in continuous time.
Catalog record
| CREATOR | Julie Alhosh, Harley Wiltzer, David Meger |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-05563
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Tractable Representations for Convergent Approximation of Distributional HJB Equations Julie Alhosh, Harley Wiltzer, David Meger Abstract In RL , the long-term behavior of decision-making policies is evaluated based on their average returns. Distributional RL has emerged, presenting techniques for learning return distributions , which provide additional statistics for evaluating policies, incorporating risk-sensitive considerations. When the passage of time cannot naturally be divided into discrete time increments, researchers have studied the CTRL ( CTRL ) problem, where agent states and decisions evolve continuously. In this setting, the HJB equation is well established as the characterization of the expected return, and many solution methods exist. However, the study of distributional RL in the continuous-time setting is in its infancy. Recent work has established a DHJB ( DHJB ) equation, providing the first characterization of return distributions in CTRL . These equations and their solutions are intractable to solve and represent exactly, requiring novel approximation techniques. This work takes strides towards this end, establishing conditions on the method of parameterizing return distributions under which the DHJB equation can be approximately solved. Particularly, we show that under a certain topological property of the mapping between statistics learned by a distributional RL algorithm and corresponding distributions, approximation of these statistics leads to close approximations of the solution of the DHJB equation. Concretely, we demonstrate that the quantile representation common in distributional RL satisfies this topological property, certifying an efficient approximation algorithm for continuous-time distributional RL . Keywords: Distributional RL, Continuous-Time RL 1 Introduction Reinforcement Learning (RL) methods classically focus on evaluating policies by the returns they earn in expectation [ 7 ] . The field of distributional RL [ 1 , 2 ] provides techniques for evaluating policies on the basis of their return distributions . Such techniques have demonstrated impressive empirical performance in various deep RL benchmarks [ 1 , 11 ] , and additionally present new possibilities for designing risk-sensitive RL agents. Research in RL has traditionally focused on discrete-time problems, where time stops until a decision is made at fixed discrete timesteps. In contrast, CTRL ( CTRL ) is concerned with settings where decisions and state transitions evolve continuously in time. In CTRL , the expected return of a policy is governed by a differential equation known as the HJB ( HJB ) equation, which has been studied extensively in the RL [ 6 ] and optimal control [ 4 ] literature. Recent works [ 8 , 9 , 5 , 10 ] have studied distributional RL in the continuous-time setting. Particularly, a DHJB ( DHJB ) equation was introduced [ 8 , 9 ] , which characterizes the CDF of return distributions in the CTRL setting. However, methods for solving the DHJB equation are not as well understood. In this work, we establish conditions under which solutions to the DHJB equation can be approximated tractably and efficiently using familiar gradient-based iterative update schemes. Since return distributions have infinitely-many degrees of freedom, they cannot be represented exactly on a computer, and therefore solutions to the DHJB equation are intractable. In response to this, Wiltzer et al. [ 9 ] introduced a SHJB ( SHJB ) loss as an objective for approximately solving the DHJB equation by optimizing a finite set of return distribution statistics. However, until this work, it has not been known whether minimization of the SHJB loss truly yields close approximations to ground truth return distributions. We make the following contributions: 1. We prove that, as long as the imputation strategy —that is, the mapping from statistics to return distributions—satisfies a certain topological property, minimizing the SHJB loss yields convergent approximations of the return distributions; 2. We demonstrate that the quantile distribution representation [ 3 ] and its corresponding imputation strategy satisfy this topological property, providing a principled loss function for continuous-time distributional RL. 2 Background In this section, we briefly summarize the relevant prior results of [ 9 ] , in the continuous-time infinite-horizon discounted return setting. The state space is denoted \mathcal{X} and is assumed to be bounded and Euclidean, and the discount factor is \gamma\in(0,1) . We assume the reward function r is bounded, so that returns are confined to a bounded set \mathcal{R}=[V_{\min},V_{\max}] . We will consider a fixed policy \pi , and our goal is to estimate its return distribution function {\eta^{\pi}:\mathcal{X}\to\mathscr{P}(\mathcal{R}),} which is the distribution of \int_{0}^{\infty}\gamma^{t}r(X_{t})\mathrm{d}t=:G^{\pi}(x)\sim\eta^{\pi}(x),\quad X_{0}=x,\quad\mathrm{d}X_{t}=\mu_{\pi}(X_{t})\mathrm{d}t+\sigma_{\pi}(X_{t})\mathrm{d}B_{t}. (1) Here, (B_{t})_{t\geq 0} is a Brownian motion, and \mu_{\pi},\sigma_{\pi} describe the stochastic differential equation governing the agent’s state under the policy \pi . Recall that the value function V^{\pi} given by V^{\pi}:x\mapsto\mathbf{E}[G^{\pi}(x)] (2) is characterized by the HJB ( HJB ) equation [ 4 ] , \langle\nabla_{x}V^{\pi}(x),\mu_{\pi}(x)\rangle+\log\gamma V^{\pi}(x)+r(x)+\frac{1}{2}\mathrm{Tr}\left(\sigma_{\pi}(x)^{\top}(\nabla_{x}^{2}V^{\pi}(x))\sigma_{\pi}(x)\right)=0. (3) Henceforth, we will write F_{\eta^{\pi}}(x,\cdot) to denote the CDF of \eta^{\pi}(x) . To begin, we formalize how imputation strategies encode distribution approximations. Definition 2.1 (Imputation Strategy) . An imputation strategy is a map \Phi_{N}:\mathbf{R}^{N}\to\mathscr{P}(\mathcal{R}) which maps a set of N statistics into a probability measure with those statistics. Additionally, for every imputation strategy \Phi_{N} , we define a map \Phi_{N}^{\mathsf{cdf}}:\mathbf{R}^{N}\times\mathcal{R}\to[0,1] given by \Phi_{N}^{\mathsf{cdf}}(s,z)=[\Phi_{N}(s)]([V_{\min},z]). (4) As an example, we may consider a Gaussian imputation strategy, such as \Phi_{2}:(\mu,\sigma^{2})\mapsto\mathcal{N}(\mu,\sigma^{2}), (5) which is a valid imputation strategy when \mu,\sigma^{2} are interpreted as mean and variance statistics. Alternatively, for a fixed subset \{\xi_{i}\}_{i=1}^{N}\subset\mathcal{R} , the map \Phi_{N}:p\mapsto\sum_{i=1}^{N}p_{i}\delta_{\xi_{i}} (6) is an imputation strategy for statistics p in the simplex \Delta_{N} , which maps a set of probabilities to a categorical distribution on the support \{\xi_{i}\}_{i=1}^{N} . Generally, N controls the resolution of our distribution approximations; as N increases, we hope to achieve higher fidelity approximations, and to more closely solve the DHJB equation. Next, we recall some useful definitions regarding generalized functions. Definition 2.2 (Schwartz Class) . Let X be a normed space. A Schwartz class is a class \mathcal{S} of rapidly decaying-smooth functions \mathcal{S}=\{f\in C^{\infty}(X;\mathbf{R}):\ \sup_{x\in X}\big[(1+\|x\|^{k})|f^{(m)}(x)|\big]<\infty\quad\forall k,m\in\mathbf{N}\}. (7) Definition 2.3 (Tempered Distribution) . A tempered distribution is an element of the topological dual \mathcal{S}^{\prime} of the Schwartz class \mathcal{S} . That is, a tempered distribution is a linear map \varrho:\mathcal{S}\to\mathbf{R}. It is important to note that “Distribution”, in the context of a tempered distribution, refers to a type of generalized function, and not a probability distribution. This formalism allows us to generalize notions of solutions to differential equations, permitting solutions where derivatives are tempered distributions. For a differential operator \mathscr{L} , we say a differential equation \mathscr{L}f=0 in the distributional sense if \int\phi(y)(\mathscr{L}f)(y)\mathrm{d}y=0\quad\forall\phi\in\mathcal{S}. (8) We are now able to state the main regularity conditions assumed on the imputation strategy. Definition 2.4 (Statistical Smoothness [ 9 ] ) . An imputation strategy \Phi_{N} is said to be statistically smooth if \Phi_{N}(s) is a tempered distribution for each s\in\mathbf{R}^{N} . Likewise, a return distribution function {\eta:\mathcal{X}\to\mathscr{P}(\mathcal{R})} is said to be statistically smooth if its state-conditioned CDF, F_{\eta^{\pi}}(x,\cdot) is a tempered distribution for each x\in\mathcal{X} and F_{\eta^{\pi}}(\cdot,z) is twice continuously differentiable almost everywhere for each z\in\mathcal{R} . Assumption 1 . At every state x\in\mathcal{X} , \eta^{\pi}(x) is absolutely continuous with respect to the Lebesgue measure. Assumption 2 . The mapping (x,z)\mapsto F_{\eta^{\pi}}(x,z) is twice differentiable over \mathcal{X}\times\mathcal{R} almost everywhere, and its second partial derivatives are continuous almost everywhere. Having established these technical assumptions, we can state the DHJB equation, which characterizes return