Theoretical guarantees for neural estimators in parametric statistics

Almut Rödder, Manuel Hentschel, Sebastian Engelke · 2025

Read Theoretical guarantees for neural estimators in parametric statistics by Almut Rödder, Manuel Hentschel, Sebastian Engelke (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-18508.txt.

Introduction

Almut Rödder, Manuel Hentschel, and Sebastian Engelke give the first theoretical guarantees for neural estimators in parametric statistics, published in 2025. Neural estimators map a data sample straight to the parameters of a statistical model. Once trained, they cost almost nothing to apply to new data, and many studies show them working well in practice, yet theory had not backed those results.

The authors split the estimator's risk into parts that can be studied one by one. For each part they state easy-to-check assumptions that drive it to zero as data grows. They verify the assumptions for popular uses of neural estimators. The result is a general recipe: the same route can certify broader network types and new estimation problems, turning a practical trick into a method with proven behavior.

Catalog record

CREATORAlmut Rödder, Manuel Hentschel, Sebastian Engelke
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE120 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-18508

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Theoretical guarantees for neural estimators in parametric statistics Almut Rödder, Manuel Hentschel, Sebastian Engelke Abstract Neural estimators are simulation-based estimators for the parameters of a family of statistical models, which build a direct mapping from the sample to the parameter vector. They benefit from the versatility of available network architectures and efficient training methods developed in the field of deep learning. Neural estimators are amortized in the sense that, once trained, they can be applied to any new data set with almost no computational cost. While many papers have shown very good performance of these methods in simulation studies and real-world applications, so far no statistical guarantees are available to support these observations theoretically. In this work, we study the risk of neural estimators by decomposing it into several terms that can be analyzed separately. We formulate easy-to-check assumptions ensuring that each term converges to zero, and we verify them for popular applications of neural estimators. Our results provide a general recipe to derive theoretical guarantees also for broader classes of architectures and estimation problems. 1 Introduction Estimation of the parameters of a family of distributions \mathopen{}\mathclose{{\left\{\mathcal{P}_{\theta}}}\right\}_{\theta\in\Theta} , taking values in \mathcal{Z}\subseteq\mathbb{R}^{d} and parametrized by \theta\in\Theta\subseteq\mathbb{R}^{p} , is one of the most classical problems in statistics. The two main approaches are frequentist maximum-likelihood estimators and Bayes estimators relying on posterior distributions. In both settings we can define the set of all point estimators as \mathcal{F}=\mathopen{}\mathclose{{\left\{f:\mathcal{Z}^{m}\rightarrow\mathbb{R}^{p}}}\right\} that map m i.i.d. samples Z=(Z^{1},\dots,Z^{m}) from some \mathcal{P}_{\theta} to an estimate \hat{\theta} . The asymptotic properties of such estimators is well-understood ( van der Vaart 1998 ) . In many modern applications, the likelihood function is not available, or its evaluation is computationally prohibitive. This includes examples in cosmology ( Alsing et al. 2019 ) , spatial Gaussian random fields ( Rue and Held 2005 ) or max-stable distributions ( de Haan 1984 ) . In some cases, composite likelihood approaches are a computationally faster alternative, but may result in significant efficiency loss ( Varin et al. 2011 ) . Simulation from such models is typically possible and computationally relatively cheap. For this reason, different simulation-based (or likelihood-free) inference methods have been developed, including indirect inference ( Gourieroux et al. 1993 ) and the well-known approach of approximate Bayesian computation (ABC) ( Beaumont et al. 2002 ) . Designing informative summary statistics can however be difficult and bad choices may lead to suboptimal performance. Similar to ABC, neural estimators are simulation-based methods in the Bayesian setting. For a given loss function \ell , the Bayes estimator f^{*}_{m} minimizes the weighted risk \displaystyle R(f) \displaystyle=\int_{\Theta}\mathbb{E}_{\theta}\left[\ell(f(Z),\theta)\right]\pi(\theta)d\theta, (1) over all estimators f\in\mathcal{F} , where the weight function \pi(\theta) is the density of the prior distribution on \Theta ( van der Vaart 1998 , e.g.,) . The corresponding Bayes risk R(f_{m}^{*}) is thus the smallest possible population error for sample size m . We focus here on the quadratic loss \ell(\hat{\theta},\theta)=\mathopen{}\mathclose{{\left\|\hat{\theta}-\theta}}\right\|_{2}^{2} where the Bayes estimator is given by the conditional expectation f_{m}^{*}(z)=\mathbb{E}(\theta\mid Z=z) , z\in\mathcal{Z}^{m} . With this perspective of an estimator as a function from data to parameters, neural networks are a natural candidate to learn this function in light of the universal approximation property ( Cybenko 1989 ) . A neural estimator frames parameter estimation as supervised learning problem with the following steps. We first generate the training data consisting of N samples \theta_{1},\dots,\theta_{N} from the prior distribution \pi(\theta) and consider one sample Z_{i}=(Z_{i}^{1},\dots,Z_{i}^{m})\in\mathcal{Z}^{m} from \mathcal{P}_{\theta_{i}}^{m} for each of them, i=1,\dots,N . The pair (Z_{i},\theta_{i}) can be seen as an independent draw of a predictor-response pair of the training distribution in a prediction problem. The neural estimator is then the minimizer of the empirical version R_{N}(f) of the population risk in ( 1 ) based on this simulated data set, i.e., the empirical risk minimizer \displaystyle\phi_{m}^{N}=\argmin_{\phi\in\Phi}\frac{1}{N}\sum_{i=1}^{N}\mathopen{}\mathclose{{\left\|f_{\phi}(Z_{i})-\theta_{i}}}\right\|_{2}^{2}, (2) where we optimize over all neural networks f_{\phi}:\mathcal{Z}^{m}\to\mathbb{R}^{p} parametrized by the set \Phi of all considered network parameters, architectures and activation functions. Figure 1 illustrates the architecture of a neural estimator with a fully connected neural network; we will use the parameter \phi and its induced neural network f_{\phi} exchangeably. In practice, it is typically not possible and not desired to obtain the global minimizer \phi_{m}^{N} exactly. Instead, regularization methods such as penalties, dropout and early stopping are employed during training to avoid overfitting ( Goodfellow et al. 2016 ) . The resulting estimator \tilde{\phi}_{m}^{N} usually has a larger training risk ( 2 ) but a smaller population risk ( 1 ). The fundamental difference of neural estimators to classical simulation-based approaches such as ABC is that they are amortized, that is, the only significant computational cost is at training time. Once the neural estimator is trained, it can be applied to any new data set with m samples from \mathcal{P}_{\theta} for an arbitrary \theta\in\Theta essentially for free, since the forward pass in a neural network is computationally very cheap. Moreover, unlike ABC, neural Bayes estimators do not rely on fixed summary statistics but directly build a map from the data set to the parameter. Intuitively, the summary statistics are learned automatically as latent representations of the network. The concept of neural estimators exists since a long time and they have been applied to various problems such as parameter estimation in times series ( Chon and Cohen 1997 ) and birth-and-death processes ( Bokma 2006 ) using dense networks. More recently, different architectures such as convolutional layers for efficient inference in temporal ( Rudi et al. 2022 ) or spatial observations ( Gerber and Nychka 2021 ; Lenzi et al. 2021 ; Walchessen et al. 2024 ) or deep sets for multiple replicates ( Sainsbury-Dale et al. 2023 ) have been proposed. While these methods seem to work well in practice and allow for very fast inference, so far no theoretical guarantees for the asymptotic or pre-asymptotic behavior of neural estimators exist; see, for instance, the discussion in Zammit-Mangion et al. 2025 . Our contribution is threefold. We first decompose the population risk R(\tilde{\phi}_{m}^{N}) of the fitted neural estimator with m replicates and based on N simulated training data as \displaystyle R(\tilde{\phi}_{m}^{N}) \displaystyle=\underbrace{R(f_{m}^{*})}_{\text{Bayes Risk}}+\underbrace{R(\phi_{m}^{*})-R(f_{m}^{*})}_{\text{Approximation Error}}+\underbrace{R(\phi_{m}^{N})-R(\phi_{m}^{*})}_{\text{Generalization Error}}+\underbrace{R(\tilde{\phi}_{m}^{N})-R(\phi_{m}^{N})}_{\Delta{\text{Algorithm}}}; (3) see Figure 2 for a geometric illustration of this decomposition. The risk of a neural estimator therefore differs from the optimal Bayes risk through three error terms: the approximation error quantifies how well the class of neural networks \Phi can approximate the Bayes estimator; the generalization error measures the amount of overfitting of the empirical risk minimizer \phi_{m}^{N} compared to the optimal neural network estimator; and a term that depends on the optimization algorithm and regularization scheme used during training. This last term is typically negative due to the positive effect of regularization on the generalization of the network. We do not consider it here and refer to the literature on numerical analysis and statistical learning theory for details (e.g., Xing et al. 2018 , and references therein) . We study all other terms of decomposition ( 3 ) separately and formulate easy-to-check assumptions under which they converge to zero as m,N\to\infty . This involves results from different fields such as the approximation theory of neural networks ( Cybenko 1989 ) , robustness ( Xu and Mannor 2012 ) , and statistical

Read the complete text