An Actor-Critic Framework for Continuous-Time Jump-Diffusion Controls with Normalizing Flows

Liya Guo Ruimeng Hu Xu Yang Yi Zhu · 2026

Read An Actor-Critic Framework for Continuous-Time Jump-Diffusion Controls with Normalizing Flows by Liya Guo Ruimeng Hu Xu Yang Yi Zhu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-05398.txt.

Introduction

Guo, Hu, Yang, and Zhu build a mesh-free actor-critic solver for continuous-time control with jumps, where time-varying drift, volatility, and shocks defeat standard policy methods. Their method rests on a time-inhomogeneous action-value function and an occupation measure, while conditional normalizing flows express flexible non-Gaussian policies with exact likelihoods. Tests on linear-quadratic control, Merton portfolio choice, and a multi-agent portfolio game show stable learning and accurate policies.

Catalog record

CREATORLiya Guo Ruimeng Hu Xu Yang Yi Zhu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE100183
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.05398

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

An Actor-Critic Framework for Continuous-Time Jump-Diffusion Controls with Normalizing Flows Liya Guo Ruimeng Hu Xu Yang Yi Zhu Abstract Continuous-time stochastic control with time-inhomogeneous jump–diffusion dynamics is central in finance and economics, but computing optimal policies is difficult under explicit time dependence, discontinuous shocks, and high dimensionality. We propose an actor–critic framework that serves as a mesh-free solver for entropy-regularized control problems and stochastic games with jumps. The approach is built on a time-inhomogeneous “little” q q -function and an appropriate occupation measure, yielding a policy-gradient representation that accommodates time-dependent drift, volatility, and jump terms. To represent expressive stochastic policies in continuous-action spaces, we parameterize the actor using conditional normalizing flows, enabling flexible non-Gaussian policies while retaining exact likelihood evaluation for entropy regularization and policy optimization. We validate the method on time-inhomogeneous linear–quadratic control, Merton portfolio optimization, and a multi-agent portfolio game, using explicit solutions or high-accuracy benchmarks. Numerical results demonstrate stable learning under jump discontinuities, accurate approximation of optimal stochastic policies, and favorable scaling with respect to dimension and number of agents. 1 Introduction Continuous-time stochastic control provides a mathematical framework for dynamical decision making in finance and economics [ 33 ] . Many problems such as portfolio selection [ 12 , 29 ] can be formulated as controlling stochastic differential equations to maximize (or minimizing) an expected discounted objective. From a computational standpoint, however, classical approaches based on dynamic programming or stochastic maximum principles become difficult to implement when the state dimension is large [ 10 ] , and when the underlying dynamics are unknown or only partially specified. These challenges have motivated the development of continuous-time reinforcement learning (RL) methods [ 38 , 39 , 23 , 24 ] that combined with neural networks, aiming to learn near-optimal control directly from interaction with the environment without requiring explicit model structure and with improved scalability to higher dimensions. A growing literature has developed continuous-time analogs of policy evaluation and policy improvement. For policy evaluation, temporal-difference type schemes are derived in [ 14 , 23 ] , providing practical methods for approximating value function directly in continuous time. For policy improvement, [ 24 ] exploits martingale structure to rewrite policy-gradient objectives as policy-evaluation problems, yielding implementable update rules. From an action-value viewpoint, [ 25 ] studies continuous-time q learning and introduces a first-order surrogate, the “little” q -function, to avoid the degeneracy of the conventional “big” Q -function in the continuous-time limit [ 37 ] . Most of these developments are focused on finite-horizon criteria, with related extensions to mean-field control in [ 40 ] . Infinite-horizon continuous-time policy-gradient formulas have appeared more recently, for example in [ 41 ] . In many financial settings, pure diffusion models are inadequate because asset prices and economic factors may exhibit abrupt movements driven by liquidity shocks or macroeconomic events [ 31 , 3 ] . Incorporating jumps is therefore essential for capturing heavy tails, discontinuities, and jump risk premia for markets [ 4 , 6 ] . This has motivated a growing body of work on deep learning and RL for jump-diffusion dynamics [ 11 , 8 , 17 , 28 , 13 , 18 , 27 ] . Motivated by the little- q methodology, [ 8 , 17 ] extend continuous-time q-learning ideas to stochastic policies and entropy regularization in jump-diffusion settings, while [ 28 ] develops an actor-critic method for deterministic controls in finite-horizon jump-diffusion games, and [ 13 ] considers optimal switching problems under jump dynamics. Most existing formulations, however, remain tied to finite-horizon objectives and often adopt Gaussian policy parameterizations in practice. In many situations, the optimal stochastic policy is non-Gaussian; see, for example, [ 8 ] . This paper addresses these gaps by developing a learning framework for discounted infinite-horizon control of time-inhomogeneous jump-diffusions under general stochastic (possibly non-Gaussian) policies. Our first contribution is the introduction of a continuous-time little q -function and a time-dependent discounted occupation measure, and the establishment of structural properties that connect these objects to policy improvement. These results lead to a policy-gradient representation valid for general time-inhomogeneous jump-diffusions on an infinite horizon. To clarify the relationship with prior work, we provide a comparative summary of continuous-time q -function formulations in Table 1 . Table 1: Comparison continuous-time Q / q -function formulations. Work Time setting State dynamics Learning objects Role of q function Policy class [ 25 ] Continuous, finite horizon Diffusion V_{\psi}(t,\bm{x}) , q_{\theta}(t,\bm{x},\bm{u}) \pi^{*}(\bm{u}\mid t,\bm{x})\propto\exp\left\{\frac{1}{\gamma}q^{*}(t,\bm{x},\bm{u})\right\} q Learned via martingale orthogonality Gaussian [ 8 , 17 ] Jump-diffusion V_{\psi}(t,\bm{x}) , q_{\theta}(t,\bm{x},\bm{u}) [ 41 ] Continuous, time-homog., infinite horizon Diffusion V_{\psi}(\bm{x}) , \pi_{\theta}(\bm{x}) Policy gradient thm for \pi^{\ast} q approximated by GAE Gaussian This work Continuous, time-inhomog., infinite horizon Jump-diffusion V_{\psi}(t,\bm{x}) , \pi_{\theta}(t,\bm{x}) Policy gradient thm for \pi^{\ast} q approximated by GAE General (normalizing flow) Our second contribution is to provide tractable benchmarks by deriving explicit solutions in several canonical specifications, including linear-quadratic control, the Merton problem with jumps, and multi-agent games with jump-driven CARA utilities, together with representative time-inhomogeneous variants. These closed-form policies serve as ground truth for assessing numerical accuracy. Finally, we propose an implementable actor–critic algorithm that combines the derived policy-gradient representation with a conditional normalizing-flow parameterization of stochastic policies. The flow-based construction enables expressive, non-Gaussian distributions for controls while preserving tractable likelihoods and gradients, which is essential in the presence of entropy regularization and policy-gradient-type theorems. Numerical experiments in both low- and high-dimensional regimes demonstrate stable learning behavior across a range of time-dependent jump-diffusion models. The rest of the paper is organized as follows. Section 2 introduces the problem setting, including the classical jump-diffusion stochastic control problem and its entropy-regularized formulation. Section 3 presents the proposed actor-critic framework: Section 3.1 develops policy evaluation for the critic, Section 3.2 introduces the “little” q -function and the occupation measure, and develops policy improvement for the actor with its theoretical justification, and Section 3.3 details the conditional normalizing flow parameterization for the actor. Section 4 states explicit solutions for several canonical problems and reports numerical experiments. We conclude in Section 5 . 2 Problem Setup Let (\Omega,\mathcal{F},\mathbb{F}:=(\mathcal{F}_{t})_{t\geq 0},\mathbb{P}) be a filtered probability space satisfying the usual conditions. Let \bm{W}=(\bm{W}_{t})_{t\geq 0} be a d -dimensional Brownian motion, N(\mathrm{d}t,\mathrm{d}\bm{z}) be a Poisson random measure corresponding to a Lévy process (\bm{L}_{t})_{t\geq 0} , and \nu be the Lévy measure on \mathbb{R}^{d} satisfying the integrability condition \int_{\mathbb{R}^{d}}\min\{|\bm{z}|^{2},1\}\nu(\mathrm{d}\bm{z})<\infty [ 15 ] . The associated compensated Poisson random measure is defined as \tilde{N}(\mathrm{d}t,\mathrm{d}\bm{z}):=N(\mathrm{d}t,\mathrm{d}\bm{z})-\nu(\mathrm{d}\bm{z})\,\mathrm{d}t and we assume that \bm{W} and N are independent. We are interested in finding an optimal control policy \pi(\cdot\mid t,\bm{x})\in\mathcal{P}(\mathcal{A}) that maximizes an infinite-horizon discounted reward based on the controlled state process \bm{X}=(\bm{X}_{t}^{\pi})_{t\geq 0}\in\mathbb{R}^{d} , which is formally described by the Itô-Lévy process \displaystyle\mathrm{d}\bm{X}_{t}^{\pi}=\bm{b}(t,\bm{X}_{t-}^{\pi},\bm{u}_{t})\,\mathrm{d}t+\bm{\sigma}(t,\bm{X}_{t-}^{\pi},\bm{u}_{t})\,\mathrm{d}\bm{W}_{t}+\int_{\mathbb{R}^{d}}\bm{\alpha}(t,\bm{X}_{t-}^{\pi},\bm{u}_{t},\bm{z})\,\tilde{N}(\mathrm{d}t,\mathrm{d}\bm{z})\,, (2.1) where the coefficients are measurable maps (\bm{b},\bm{\sigma}):[0,\infty)\times\mathbb{R}^{d}\times\mathcal{A}\to(\mathbb{R}^{d},\mathbb{R}^{d\times d}) , \bm{\alpha}:[0,\infty)\times\mathbb{R}^{d}\times\mathcal{A}\times\mathbb{R}^{d}\to\mathbb{R}^{d} , and the control \bm{u}_{t} is intended to follow the randomized feedback law \pi(\cdot\mid t,\bm{x}) . We assume standard Lipschitz and linear growth conditions on (\bm{b},\bm{\sigma},\bm{\alpha}) , so that the corresponding SDE admits a unique strong solution for every admissible control process (cf. [ 31 ] ). With this state dynamics, we consider the following entropy-regularized reward: \tilde{f}(s,\bm{y};\pi):=\int_{\mathcal{A}}\big(f(s,\bm{y},\bm{u})-\gamma\log\pi(\bm{u}\mid s,\bm{y})\big)\,\pi(\bm{u}\mid s,\bm{y})\mathrm{d}\bm{u}, (2.2) where f is the standard running cost and we consider \mathcal{S}\big(\pi(\cdot\mid s,\bm{y})\big):=-\int_{\mathcal{A}}\pi(\bm{u}\mid s,\bm{y})\log\pi(\bm{u}\mid s,\bm{y})\,\mathrm{d}\bm{u} , the Shannon

Read the complete text