Are Statistical Methods Obsolete in the Era of Deep Learning? A Study of ODE Inverse Problems

Skyler Wu, Department of Statistics, Stanford University · 2025

Read Are Statistical Methods Obsolete in the Era of Deep Learning? A Study of ODE Inverse Problems by Skyler Wu, Department of Statistics, Stanford University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21723.txt.

Introduction

Are old-school stats dead now that deep nets can fit near any curve? Skyler Wu, Shihao Yang, and Samuel Kou put the claim to a test in 2025 on back-solving rate equations. Two test worlds served: a spread-of-disease model and a chaos model. A physics-fed net stood for deep learning; a curve method bound to the true paths stood for stats.

The catalog title drags grant thanks into the line, and its author field is garbled; the head gives the clean title and byline. On sparse, noisy data the stats method won on bias and spread, with far fewer knobs to tune. It also called the future better, where the big net strayed with no data to hold it. Small math slips hurt the net more as runs grew long. The team close: lean stats still earn their place.

Catalog record

CREATORSkyler Wu, Department of Statistics, Stanford University
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE85 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-21723

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Are Statistical Methods Obsolete in the Era of Deep Learning? A Study of ODE Inverse Problems Thanks: S.Y. is partially supported by a National Science Foundation grant DMS-2318883. Thanks: S.K. is partially supported by an award from Harvard Data Science Initiative. Skyler Wu, Department of Statistics, Stanford University Abstract In the era of AI, neural networks have become increasingly popular for modeling, inference, and prediction, largely due to their potential for universal approximation. With the proliferation of such deep learning models, a question arises: are leaner statistical methods still relevant? To shed insight on this question, we employ the mechanistic nonlinear ordinary differential equation (ODE) inverse problem as a testbed, using the physics-informed neural network (PINN) as a representative of the deep learning paradigm and manifold-constrained Gaussian process inference (MAGI) as a representative of statistically principled methods. Through case studies involving the SEIR model from epidemiology and the Lorenz model from chaotic dynamics, we demonstrate that statistical methods are far from obsolete, especially when working with sparse and noisy observations. On tasks such as parameter inference and trajectory reconstruction, statistically principled methods consistently achieve lower bias and variance, while using far fewer parameters and requiring less hyperparameter tuning. Statistical methods can also decisively outperform deep learning models on out-of-sample future prediction, where the absence of relevant data often leads overparameterized models astray. Additionally, we find that statistically principled approaches are more robust to accumulation of numerical imprecision and can represent the underlying system more faithfully to the true governing ODEs. Keywords: Bayesian Methods, Neural Networks, Dynamical Systems, ML for Science. 1 Introduction The advancement of deep neural network models in the last fifteen years has profoundly altered the scientific landscape of estimation, prediction and decision making, from the early success of image recognition ( Krizhevsky et al. 2012 , He et al. 2016 ) , to the success of self-learning of board games ( Silver et al. 2017 ) , to machine translation ( Wu et al. 2016 ) , to generative AI ( Ho et al. 2020 ) , and to the success of protein structure prediction ( Jumper et al. 2021 ) , among many other developments. In many of these successes, there are no well-established mechanistic models to describe the underlying problem — for example, we do not fully understand how human brains translate from one language to another. As such, it is conceivable that such successes are attributable to deep neural networks’ remarkable capabilities for universal function approximation. In contrast, the hand-crafted models that existed before deep neural networks (such as n-gram models ( Katz 1987 , Brown et al. 1992 , Bengio et al. 2000 ) ) were too restricted to offer satisfactory approximation. How well do deep neural network models work when there are well-established mechanistic models (as in physical sciences, where decades of theoretical and experimental endeavors have yielded highly accurate mechanistic models in many cases) — in particular, how do the inference and prediction results of deep neural network models compare to more statistical approaches in the presence of reliable mechanistic models — is an interesting question. The current article focuses on this comparison. In physical and natural sciences, the mechanistic models are often described by ordinary differential equations (ODEs), which are often referred to as dynamical systems. Statistical inference of dynamical systems, also known as the inverse problem ( Kaipio and Somersalo 2006 , Mueller and Siltanen 2012 ) , arises in many scientific studies, where advances in experimental and data collection techniques have allowed researchers to track dynamical systems closer to real-time ( Xie 2015 ) . Although ODE models are parametric, their inference has long challenged researchers, as most nonlinear ODEs require numerical solutions, which can be expensive for many practical applications ( Mueller and Siltanen 2012 ) . In this article, we compare the neural network approach to the statistical approach on their inference of dynamical systems. Both approaches were proposed to address the numerical challenge of (nonlinear) dynamical systems. Specifically, we compare the performance of the recently developed physics-informed neural network (PINN) ( Raissi et al. (2017b) , Raissi et al. (2019) , Lu et al. (2021) ), which represents the neural network approach to the ODE inference problem, to that of manifold-constrained Gaussian process inference (MAGI) ( Yang et al. 2021 ) , which represents a principled Bayesian statistical approach to ODE inference. In this comparison, we investigate three different aspects of statistical inference from noisy and sparse observations: (i) how well can the methods recover the model parameters (including quantities of scientific interest defined by the parameters); (ii) how well can the methods recover the system’s time-trajectories; and (iii) how well can the methods predict the future state of the system given the observations from a short time window. In the comparison, for a comprehensive understanding we also consider the case of missing system components, where some system components are entirely unobserved—such cases often arise in real experiments due to various (technical or experimental) limitations of data collection. We consider two dynamical systems in this article: the SEIR model, a widely used model in epidemiology, and the Lorenz model, a model famous for generating chaotic behavior. Together, these two models represent a spectrum of dynamical systems: one with significant scientific implications, and the other characterized by inherent challenges in long-term predictability. These two dynamical systems thus serve as illustrative and meaningful test cases to assess the different inference methods. Our study demonstrates that MAGI is generally more robust and reliable than PINN, particularly in sparse and noisy data environments that are common in laboratory sciences. In these environments, researchers often face fewer than 100 observation data points with noise levels ranging from 5%, 15% to higher. We found that for the SEIR model, when all system components are observed, the performance of PINN and MAGI is comparable. However, when missing components are present, MAGI clearly outperforms PINN. The contrast becomes even more pronounced with the chaotic Lorenz system, where PINN routinely struggles to handle the complexity of the system, while MAGI continues to consistently produce accurate inference results. Our results indicate that in the era of deep learning, statistically principled methods are still very relevant, especially when working with sparse and noisy data. Our study underscores the importance of proper probabilistic modeling in capturing the underlying dynamics for inference and prediction when one has limited data. The rest of the article is organized as follows. Section 1.1 briefly discusses the problem of statistical inference of dynamical models. Section 1.2 reviews other approaches. Section 2 provides the architectural details on PINN and MAGI. Section 3 compares PINN and MAGI on the SEIR model. Section 4 compares PINN and MAGI on the Lorenz model. Finally, Section 5 concludes the article with discussion. 1.1 Statistical inference of dynamical systems In physical and natural sciences, the mechanistic models are often governed by dynamical systems.: \dot{\bm{x}}(t)=\frac{d\bm{x}(t)}{dt}=f(\bm{x}(t),\bm{\theta},t),\quad t\in[0,T], (1) where the vector \bm{x}(t) represents the system’s state/status at time t , and \bm{\theta} is the model parameters. For example, in neuroscience, ODEs underpin models of neuron signal transmission ( FitzHugh 1961 , Nagumo et al. 1962 ) . In ecology, predator-prey population dynamics uses ODEs ( Lotka 1932 ) to capture

Read the complete text