Comparing statistical and deep learning techniques for parameter estimation of continuous-time stochastic differentiable equations.

1 st Aroon Sankoh, 2 nd Mladen Victor Wickerhauser · 2025

Read Comparing statistical and deep learning techniques for parameter estimation of continuous-time stochastic differentiable equations. by 1 st Aroon Sankoh, 2 nd Mladen Victor Wickerhauser (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-03980.txt.

Introduction

Aroon Sankoh and Victor Wickerhauser compare classic statistics against deep learning for fitting stochastic equations in a 2025 paper. Processes like the Ornstein-Uhlenbeck model describe prices and temperatures that wander with a pull back to center. Knowing their parameters makes forecasts and risk estimates possible.

Maximum likelihood, the old workhorse, is pitted against a recurrent neural network across a series of experiments. Both accuracy and computing cost are scored. The contest asks whether the neural approach's flexibility beats the estimator's theory-backed efficiency on this classic problem. The setup frames a fair head-to-head for a workhorse model of continuous-time randomness.

Catalog record

CREATOR1 st Aroon Sankoh, 2 nd Mladen Victor Wickerhauser
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-03980

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Comparing statistical and deep learning techniques for parameter estimation of continuous-time stochastic differentiable equations. 1 st Aroon Sankoh, 2 nd Mladen Victor Wickerhauser Abstract Stochastic differential equations such as the Ornstein-Uhlenbeck process have long been used to model real-world probablistic events such as stock prices and temperature fluctuations. While statistical methods such as Maximum Likelihood Estimation (MLE), Kalman Filtering, Inverse Variable Method, and more have historically been used to estimate the parameters of stochastic differential equations, the recent explosion of deep learning technology suggests that models such as a Recurrent Neural Network (RNN) could produce more precise estimators. We present a series of experiments that compare the estimation accuracy and computational expensiveness of a statistical method (MLE) with a deep learning model (RNN) for the parameters of the Ornstein-Uhlenbeck process. I Introduction In section I, we will define the Ornstein-Uhlenbeck (OU) stochastic process and explore some of the theory behind its solution. After introducing useful properties of the OU process, we can define the likelihood function to optimize for MLE estimation and search algorithm(s) we will use to obtain estimates. Section II discusses the network architecture for our recurrent neural network and experimental settings used in training and testing. Section III describes the results of both parameter estimation techniques and interprets them in the context of relevant works. Section IV summarizes and concludes our case study. I-A Processes of Ornstein-Uhlenbeck Type A process of X=\{X_{t}\} is Ornstein-Uhlenbeck (OU) if it satisfies the following stochastic differential equation: \displaystyle dX_{t}=-\theta X_{t}dt+\sigma dW_{t} (1) \displaystyle X_{0}=X_{0},t\geq 0 with (\theta,\sigma)\in(0,\infty)\times(0,\infty) and t_{i}=i\Delta for i=1,2,... and fixed increments \Delta>0 . W_{t} is standard Brownian Motion. We can find a unique, strong solution to (1) using Itô’s Formula. Consider the integrating factor: I_{t}=e^{\theta t} Multiplying both sides of (1) by I_{t} leads to: e^{\theta t}dX_{t}=-e^{\theta t}X_{t}dt+\sigma e^{\theta t}dW_{t} Rewriting the left-hand-side using the product rule: d(e^{\theta t}X_{t})=\sigma e^{{\theta t}}dW_{t} Integrating from 0 to t : e^{\theta t}X_{t}=X_{0}+\sigma\int_{o}^{t}e^{{\theta s}}dW_{s} Leaves us with the solution: X_{t}=X_{0}e^{-\theta t}+\sigma\int_{o}^{t}e^{{-\theta(t-s)}}dW_{s} (2) The existence of (2) also means that (1) is well-defined. See [1] for more details. It is well-known that if X_{0}\sim N(0,\frac{\sigma^{2}}{2\theta}) then (2) is a time-stationary Gauss-Markov process with continuous sample paths [2]; we can easily confirm these properties below: 1. Gaussian. X_{0}e^{-\theta t} is deterministic (X_{0}=X_{0}) and \sigma\int_{o}^{t}e^{{-\theta(t-s)}}dW_{s} is Gaussian as it is a Weiner integral with respect to Brownian Motion W_{t} , which is a Gaussian process itself. Any linear combination of a deterministic and Gaussian process is Gaussian. 2. Time-Stationary. (2) has a bounded constant mean and its’ auto-covariance function only depends on time increments |t-s| , not on t or s seperatedly [1]. 3. Markovian. A markov process M_{t} is one where the future state of the process only depends on its’ present state and not any past states: P(M_{t}|M_{s}\cap M_{r})=P(M_{t}|M_{s}) where r<s . By examining the explicit solution: X_{t}=X_{s}e^{-\theta(t-s)}+\sigma\int_{s}^{t}e^{{-\theta(t-r)}}dW_{r} We see that the first term only depends on s (the previous state) and the second term is independent of X_{<s} given X_{s} since it depends on future increments of W_{t} , and Brownian motion has independent increments. Note that X_{t} has the following mean, variance, and covariance functions: E(X_{t})=E(X_{0})e^{-\theta t} Var(X_{t})=\frac{\sigma^{2}}{2\theta}+e^{-2\theta t}\left(Var(X_{0})-\frac{\sigma^{2}}{2\theta}\right) C(t,s)=\frac{\sigma^{2}}{2\theta}e^{-\theta|t-s|} We can use the Gaussian and Markovian properties, along with the known transition density of the Ornstein Uhlenbeck process to derive the likelihood function for the parameter vector \eta=(\sigma^{2},\theta) . Doob [2] first dervied the transition density, or the probability of a process being in state x at time t given the process was in state x^{\prime} at time s , using the Markov property. \displaystyle P(X_{t}=x\mid X_{s}=x^{\prime}) \displaystyle=\frac{1}{\sqrt{2\pi V(t-s)}} \displaystyle\quad\times\exp\left(-\frac{\left(x-x^{\prime}e^{-\theta(t-s)}\right)^{2}}{2V(t-s)}\right) where V(t-s)=\frac{\sigma^{2}}{2\theta}\left(1-e^{-2\theta(t-s)}\right) , this term accounts for the dispersion of the process over time, dependent on the relationship between \theta and \sigma^{2} . Suppose X=\{X_{t_{o}},X_{t_{1}},X_{t_{2}},...X_{t_{n}}\} is a sequence of n+1 observations of our process described by (2) . The likelihood function for the parameter \eta is constructed as the product of the transition densities between consecutive observations: \displaystyle L(\eta) \displaystyle=\prod_{i=1}^{n}P(X_{t_{i}}|X_{t_{i-1}}) \displaystyle=\prod_{i=1}^{n}\frac{1}{\sqrt{2\pi V(t_{i}-t_{i-1})}} \displaystyle\times\exp\left(\frac{\left(X_{t_{i}}-X_{t_{i-1}}e^{-\theta(t_{i}-t_{i-1})}\right)^{2}}{2V(t_{i}-t_{i-1})}\right) Now if we let \Delta t=t_{i}-t_{i-1} , the log-likelihood becomes: \displaystyle\log(L(\eta)) \displaystyle=-\frac{1}{2}\sum_{i=1}^{n}\left[\log(2\pi V(\Delta t_{i}))\right. \displaystyle\quad\left.+\frac{\left(X_{t_{i}}-X_{t_{i-1}}e^{-\theta\Delta t_{i}}\right)^{2}}{V(\Delta t_{i})}\right] Plugging V(\Delta t_{i})=\frac{\sigma^{2}}{2\theta}(1-e^{-2\theta(\Delta t_{i})}) solves to our final log-likelihood function: \displaystyle\log(L(\eta)) \displaystyle=\frac{n}{2}\log(2\pi) (3) \displaystyle-\frac{1}{2}\sum_{i=1}^{n}\log\left(\frac{\sigma^{2}}{2\theta}(1-e^{-2\theta\Delta t_{i}})\right) \displaystyle-\frac{1}{2}\sum_{i=1}^{n}\frac{\left(X_{t_{i}}-X_{t_{i-1}}e^{-\theta\Delta t_{i}}\right)^{2}}{\frac{\sigma^{2}}{2\theta}(1-e^{-2\theta(\Delta t_{i})})} Iacus [3] showed that if we take the maximum of (3) we can find a closed form solution for \sigma^{2} . But since \theta must be estimated numerically, we will estimate \eta numerically. I-B Optimization Procedure Many optimization techniques for parameter estimation of stochastic differential equations have been proposed. Valdivieso, Schoutens, and Tuerlinckx [4] used a modified global differential evolution algorithm developed by [5] to estimate parameters of one-dimensional Ornstein-Uhlenbeck processes driven by a background Levy process. Markov Chain Monte Carlo methods have been proposed by [6] to estimate parameters for diffusion processes, specifically for latent variable models. Durham and Gallant [7] propose simulation based MLE techniques that use Monte Carlo techniques to improve on [8]’s seminal simulation scheme. Various method of moments (MOM) estimators have also been proposed, Gallant and Tauchen [9] introduced an efficient MOM estimator to generate ideal generalized MOM conditions for MLE. We propose a hybrid approach that I. Uses generalized method of moments for parameter initialization. II. Refines the parameter estimates using the Broyden-Fletcher-Goldfarb-Shanno (BFGS) local optimization algorithm. III. Employs the basin-hopping global optimization algorithm if the likelihood surface remains extremely non-convex after parameter refinement. Pseudo-code for the generalized method of moments and BFGS algorithms can be found below. In this paper, we used Python version 3.12.9 for all algorithm implementations, simulations, and statistical analysis. Algorithm 1 Generalized Method of Moments 1: procedure GMM (X,\Delta t) 2: X\leftarrow\text{array}(X) 3: \hat{\rho}\leftarrow\text{corrcoef}(X_{1:T-1},X_{2:T})_{1,2} 4: \hat{\rho}\leftarrow\min(\max(\hat{\rho},10^{-4}),\ 0.999) 5: \hat{\theta}\leftarrow\max\left(\frac{-\log(\hat{\rho})}{\Delta t},\ 0.5\right) 6: \hat{\sigma}^{2}\leftarrow 2\cdot\hat{\theta}\cdot\text{Var}(X) 7: return (\hat{\theta},\hat{\sigma}^{2}) Algorithm 2 Broyden–Fletcher–Goldfarb–Shanno 1: procedure BFGS (f,\nabla f,x_{0},H_{0},\epsilon) 2: x\leftarrow x_{0} 3: H\leftarrow H_{0} 4: repeat 5: g\leftarrow\nabla f(x) 6: if \|g\|<\epsilon then 7: return x 8: end if 9: p\leftarrow-Hg 10: \alpha\leftarrow\text{LineSearch}(f,x,p) 11: x_{\text{new}}\leftarrow x+\alpha p 12: s\leftarrow x_{\text{new}}-x 13: y\leftarrow\nabla f(x_{\text{new}})-g 14: \rho\leftarrow\frac{1}{y^{\top}s} 15: H\leftarrow(I-\rho sy^{\top})H(I-\rho ys^{\top})+\rho ss^{\top} 16: x\leftarrow x_{\text{new}} 17: until convergence 18: return x II Deep Learning Parameter Estimators The exponential increase in computing power has catalzyed the wide-spread application of deep learning methods to the natural sciences, medical fields, and finance over the past twenty years. Stochastic processes are used to model random events within all these areas, suggesting that deep learning models can accurately estimate the parameters of such processes. Previous efforts at parameter estimation via deep learning focus on partial differential equations [10-11], or investigating trajectories in anomalous diffusion [12-13]. Works that explore parameter estimation of specific stochastic processes via deep learning [14] often do so with trajectory-rich training sets and high-performance GPUs. Since typically only research labs and companies have access to the most powerful graphics cards, a lightweight neural network that can be trained and tested using minimal resources will provide the best alternative to traditional estimation techniques for independent researchers. We propose a Long-Short-Term-Memory (LSTM) neural network that can be trained on as few as 20,000 sample paths for the estimation of the drift and volatility parameters of the Ornstein-Uhlenbeck process. II-A Network Architecture Long-Short-Term-Memory models are a type of recurrent neural network (RNN) specifically proposed [15] to address the vanishing gradient problem

Read the complete text