Bridging Discrete and Continuous RL: Stable Deterministic Policy Gradient with Martingale Characterization
Ziheng Cheng, Xin Guo, Yufei Zhang · 2025
Read Bridging Discrete and Continuous RL: Stable Deterministic Policy Gradient with Martingale Characterization by Ziheng Cheng, Xin Guo, Yufei Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-23711.txt.
Introduction
Most continuous-time reinforcement learning uses random policies, which demand frequent action sampling and costly averages. Ziheng Cheng and 2 co-authors develop deterministic policy gradient methods for that setting instead. They derive a gradient formula built on an advantage rate function, a running score of how much better an action is than average. Martingale characterizations of the value function and the advantage rate make the gradients estimable in practice.
On that base they build CT-DDPG, a model-free deep algorithm for problems with continuous time and state. In experiments it learned more stably and converged faster than stochastic-policy rivals, across varied time steps and noise levels.
Catalog record
| CREATOR | Ziheng Cheng, Xin Guo, Yufei Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-23711
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Bridging Discrete and Continuous RL: Stable Deterministic Policy Gradient with Martingale Characterization Ziheng Cheng, Xin Guo, Yufei Zhang Abstract The theory of discrete-time reinforcement learning (RL) has advanced rapidly over the past decades. Although primarily designed for discrete environments, many real-world RL applications are inherently continuous and complex. A major challenge in extending discrete-time algorithms to continuous-time settings is their sensitivity to time discretization, often leading to poor stability and slow convergence. In this paper, we investigate deterministic policy gradient methods for continuous-time RL. We derive a continuous-time policy gradient formula based on an analogue of the advantage function and establish its martingale characterization. This theoretical foundation leads to our proposed algorithm, CT-DDPG, which enables stable learning with deterministic policies in continuous-time environments. Numerical experiments show that the proposed CT-DDPG algorithm offers improved stability and faster convergence compared to existing discrete-time and continuous-time methods, across a wide range of control tasks with varying time discretizations and noise levels. 1 Introduction Deep Reinforcement learning (RL) has achieved remarkable success over the past decade, powered by theoretical advances and the success of algorithms in discrete-time systems such as Atari, Go, and Large Language Models [ Mnih et al. 2013 ; Silver et al. 2016 ; Guo et al. 2025 ] . However, many real-world problems, such as robotic control, autonomous driving, and financial trading, are inherently continuous in time. In these domains, agents need to interact with the environment at an ultra-high frequency, underscoring the need for continuous-time RL approaches [ Wang et al. 2020 ] . One major challenge in applying discrete-time RL to continuous-time environments is the sensitivity to the discretization step size. As the step size decreases, standard algorithms often degrade, resulting in exploding variance, poor stability, and slow convergence. While several works have attempted to resolve this issue with discretization-invariant algorithms [ Tallec et al. 2019 ; Park et al. 2021 ] , their underlying design principles are rooted in discrete-time RL. As a result, these methods are not robust when applied to complex, stochastic , and continuous real-world environments. Recently there is a fast growing body of research on continuous-time RL [ Yildiz et al. 2021 ; Jia and Zhou 2022a ; Jia and Zhou 2022b ; Jia and Zhou 2023 ; Zhao et al. 2023 ; Giegrich et al. 2024 ] , including rigorous mathematical formulations and various algorithmic designs. However, most existing methods either rely on model-based assumptions, or consider stochastic policy, which is difficult to sample in continuous time, state and action spaces [ Jia et al. 2025 ] , and imposes Bellman equation constraints which are not feasible for implementation within deep RL frameworks. These challenges hinder the application of continuous-time RL framework in practice, leading to an important research question: Can we develop a theoretically grounded algorithm that achieves stability and efficiency for deep RL in continuous-time environments? In this paper, we address this question by investigating deterministic policy gradient (DPG) methods. We consider general continuous-time dynamics driven by a stochastic differential equation over a finite horizon. our main contributions are summarized as follows: • In Section 3 , we develop a rigorous mathematical framework for model-free DPG methods in continuous-time RL. Specifically, 3.1 derives the DPG formula based on the advantage rate function . 3.2 further utilizes a martingale criterion to characterize the advantage rate function, laying the foundation for subsequent algorithm design. We also provide detailed comparisons against existing continuous-time RL algorithms with stochastic policy and discuss their major flaws and impracticality in deep RL frameworks. • In Section 4 , we propose CT-DDPG, a novel and practical actor-critic algorithm with provable stability and efficiency in continuous-time environments. Notably, we utilize a multi-step TD objective, and prove its robustness to time discretization and stochastic noises in Section 4.2 . For the first time, we provide the theoretical insights of the failure of standard discrete-time deep RL algorithms in continuous and stochastic settings. • Through extensive experiments in Section 5 , we verify that existing discrete/continuous time algorithms lack robustness to time discretization and dynamic noise, while our method exhibits consistently stable performance. 2 Problem Formulation This section formulates the continuous RL problem, where the agent learns an optimal parametrized policy to control an unknown continuous-time stochastic system to maximize a reward functional over a finite time horizon. Let the state space be \displaystyle\mathbb{R}^{n} and the action space be an open set \displaystyle\mathcal{A}\subseteq\mathbb{R}^{d} . For each non-anticipative \displaystyle\mathcal{A} -valued control (action) process \displaystyle\bm{a}=(a_{t})_{t\geq 0} , consider the associated state process governed by the following dynamics: \mathrm{d}X_{t}^{\bm{a}}=b(t,X_{t}^{\bm{a}},a_{t})\mathrm{d}t+\sigma(t,X_{t}^{\bm{a}},a_{t})\mathrm{d}W_{t},\;t\in[0,T];\quad X_{0}^{\bm{a}}=x_{0}\sim\nu, (2.1) where \displaystyle\nu is the initial distribution, \displaystyle(W_{t})_{t\geq 0} is an \displaystyle m -dimensional Brownian motion on a filtered probability space \displaystyle(\Omega,\mathcal{F},\mathbb{F}=(\mathcal{F}_{t})_{t\geq 0},\mathbb{P}) , and \displaystyle b:[0,T]\times\mathbb{R}^{n}\times\mathcal{A}\to\mathbb{R}^{n} , \displaystyle\sigma:[0,T]\times\mathbb{R}^{n}\times\mathcal{A}\to\mathbb{R}^{n\times m} are continuous functions. The reward functional of \displaystyle\bm{a} is given by \mathbb{E}\left[\int_{0}^{T}e^{-\beta t}r(t,X_{t}^{\bm{a}},a_{t})\mathrm{d}t+e^{-\beta T}g(X_{T}^{\bm{a}})\right], (2.2) where \displaystyle\beta\geq 0 is a discount factor, and \displaystyle r:[0,T]\times\mathbb{R}^{n}\times\mathcal{A}\to\mathbb{R} and \displaystyle g:\mathbb{R}^{n}\to\mathbb{R} are continuous functions, representing the running and terminal rewards, respectively. It is well-known that under mild regularity conditions, it suffices to optimize ( 2.2 ) over control processes generated by Markov policies [ Kurtz and Stockbridge 1998 ] . Given a Markov policy \displaystyle\mu:[0,T]\times{\mathbb{R}}^{n}\to\mathcal{A} , the associated state process \displaystyle(X^{\mu}_{t})_{t\geq 0} evolves according to the dynamics: \mathrm{d}X_{t}^{\mu}=b(t,X_{t}^{\mu},\mu(t,X_{t}^{\mu}))\mathrm{d}t+\sigma(t,X_{t}^{\mu},\mu(t,X_{t}^{\mu}))\mathrm{d}W_{t},\;t\in[0,T];\quad X^{\mu}_{0}=x_{0}\sim\nu. (2.3) The agent aims to maximize the following reward \mathbb{E}\left[\int_{0}^{T}e^{-\beta t}r(t,X_{t}^{\mu},\mu(t,X^{\mu}_{t}))\mathrm{d}t+e^{-\beta T}g(X_{T}^{\mu})\right] (2.4) over all admissible policies \displaystyle\mu . Importantly, the agent does not have access to the coefficients \displaystyle b , \displaystyle\sigma , \displaystyle r and \displaystyle g . Instead, the agent directly interacts with Equation 2.3 with different actions, and refines her strategy based on observed state and reward trajectories. We emphasize that in this paper, we directly optimize ( 2.4 ) over deterministic policies, which map the state space directly to the action space, rather than over stochastic policies as studied in Jia and Zhou 2022b ; Jia and Zhou 2023 ; Zhao et al. 2023 , which map the state space to probability measures over the action space (see Section 3.3 ). To solve Equation 2.4 , a practical approach is to restrict the optimization problem over a sufficiently rich class of parameterized policies. More precisely, given a class of policies \displaystyle\{\mu_{\phi}:[0,T]\times{\mathbb{R}}^{n}\to\mathcal{A}\mid\phi\in\mathbb{R}^{k}\} parameterized by \displaystyle\phi , we consider the following maximization problem: \max_{\phi\in\mathbb{R}^{k}}J(\phi),\quad\textnormal{with}\quad J(\phi)\coloneqq\mathbb{E}\left[\int_{0}^{T}e^{-\beta t}r(t,X_{t}^{\phi},\mu_{\phi}(t,X^{\phi}_{t}))\mathrm{d}t+e^{-\beta T}g(X_{T}^{\phi})\right], (2.5) where \displaystyle X^{\phi} denotes the state process controlled by \displaystyle\mu_{\phi} . Throughout this paper, we assume the initial state distribution \displaystyle\nu has a second moment, and impose the following regularity conditions on the policy class and model coefficients. Assumption 1 . There exists \displaystyle C\geq 0 such that for all \displaystyle t\in[0,T] , \displaystyle a,a^{\prime}\in\mathcal{A} and and \displaystyle x,x^{\prime}\in{\mathbb{R}}^{n} , \displaystyle\displaystyle|b(t,x,a)-b(t,x^{\prime},a^{\prime})|+|\sigma(t,x,a)-\sigma(t,x^{\prime},a^{\prime})| \displaystyle\displaystyle\leq C(|x-x^{\prime}|+|a-a^{\prime}|), \displaystyle\displaystyle|b(t,0,0)|+|\sigma(t,0,0)|\leq C,\quad|r(t,x,a)|+|g(x)| \displaystyle\displaystyle\leq C(1+|x|^{2}+|a|^{2}), and there exists a locally bounded function \displaystyle\rho_{1}:[0,\infty)\to[0,\infty) such that for all \displaystyle\phi\in{\mathbb{R}}^{k} , \displaystyle t\in[0,T] , and \displaystyle x,x^{\prime}\in{\mathbb{R}}^{n} , \displaystyle|\mu_{\phi}(t,x)-\mu_{\phi}(t,x^{\prime})|\leq\rho_{1}(|\phi|)|x-x^{\prime}| and \displaystyle|\mu_{\phi}(t,0)|\leq\rho_{1}(|\phi|) . 1 holds for all policies parameterized by feedforward neural networks with Lipschitz activations. It ensures that the state dynamics and the objective function are well defined for any \displaystyle\phi\in\mathbb{R}^{k} . 3 Main Theoretical Results We will first characterize the gradient of the objective functional Equation 2.5 with respect to the policy parameter \displaystyle\phi , using a continuous-time analogue of the discrete-time advantage function. We will