Bernstein-Von Mises for semiparametric mixtures

Stefan Franssen, Jeanne Nguyen, Aad van der Vaart · 2024

Read Bernstein-Von Mises for semiparametric mixtures by Stefan Franssen, Jeanne Nguyen, Aad van der Vaart (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-00219.txt.

Introduction

Stefan Franssen, Jeanne Nguyen, and Aad van der Vaart prove a Bayesian limit theorem for models with hidden variables in two thousand twenty-four. Such models pair a known kernel with an unknown mixing spread. A Bayesian places prior guesses on both the target value and that spread.

The team first shows the estimates settle on the truth in broad terms. They then prove a general Bernstein-von Mises result, which says the Bayesian spread ends up shaped like the best frequentist one. Tools are given to check the terms in concrete cases. Two hard cases are worked through: a frailty model for survival data and an errors-in-variables model.

Catalog record

CREATORStefan Franssen, Jeanne Nguyen, Aad van der Vaart
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE139 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-00219

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Bernstein-Von Mises for semiparametric mixtures Stefan Franssen, Jeanne Nguyen, Aad van der Vaart Abstract 1 Introduction Statistical modelling often involves striking a delicate balance between flexibility and simplicity. While parametric models offer a straightforward representation of data, they may fail to capture the intricate complexities present in real-world phenomena. On the other hand, nonparametric models provide remarkable flexibility, but their complexity can also be a drawback. Semiparametric modelling is a compromise that tries to combine the strengths of parametric and nonparametric models. (Strict) semiparametric models incorporate both a flexible nonparametric component and a simpler parametric component. Typically, the parameter of interest is part of the parametric component, and the nuisance parameters are part of the nonparametric components. The advantage of semiparametric models over nonparametric models is that it typically is easier to give an interpretation to the parametric part of the model. Moreover, they typically are able to achieve at n^{-1/2} rate, which is a parametric rate of learning, which is faster than most nonparametric models. They do this while being more flexible and robust that parametric models, by allowing the nuisance part of the parameter to be modelled fully nonparametrically. This gives the robustness against misspecification of nonparametric models while learning as fast as parametric models. There are several distinct ways of creating estimators for a quantity of interest. Ideally, we use the most accurate estimator possible, and have a method of quantifying its uncertainty. Examples of estimators which are known to be asymptotically the most accurate are the one-step estimator and the maximum likelihood estimator. However, these methods do not provide uncertainty quantification in the method, one needs to use other methods like the bootstrap. Bayesian methods do provide automatic uncertainty quantification in the form of credible sets. However, these credible sets may fail to provide valid frequentist coverage, and the Bayesian posterior might not converge at an optimal rate. One way of proving that the Bayesian posterior is optimal and has optimal and valid frequentist coverage is the Bernstein-von Mises theorem (BvM). In this paper, we will prove such a Bernstein-von Mises theorem for semiparametric mixtures. This means that practitioners can use this result to show that the Bayesian method has not only Bayesian but also frequentist interpretability. 1.1 Mixture models Suppose we observe a random sample from a mixture density of the form x\mapsto p_{{\theta},F}(x):=\int p_{\theta}(x\mathchar 25194\relax z)\,dF(z). (1) The kernel x\mapsto p_{\theta}(x\mathchar 25194\relax z) is the density of a random variable X given a latent random variable Z , which has marginal distribution F . The kernel is indexed by a parameter {\theta}\in\mathbb{R}^{d} that we wish to estimate. Equivalently, we can consider this as a latent variable model. We have some latent variable Z\sim F that we do not observe, and the distribution of X\mathchar 25194\relax Z\sim p_{\theta}(x\mathchar 25194\relax Z) . Because we do not observe Z , we might lose information for estimating \theta . Because latent variables have widespread usage in statistics, semiparametric mixture models have a widespread application throughout statistics. In this paper we will be considering two examples: exponential frailty and the errors-in-variables models. Exponential frailty models are a particular case of frailty models. We observe pairs of exponential distributed random variables (X_{i},Y_{i}) where X_{i}|Z_{i} and Y_{i}|Z_{i} are independent exponential random variables with rates Z_{i} and \theta Z_{i} respectively. Here Z_{i} is a latent random variable which follows an unknown distribution F . They are mixtures given by p_{\theta,F}(x,y)=\int z^{2}\theta e^{-z(x+\theta y)}\dd F(z) The parameter of interest is \theta , so this naturally forms a semiparametric mixture model. We will consider this model in Section 5 . The frailty models are used in biomedical research. Semiparametric Bayesian frailty models have been considered in [ WM97 , PD06 , TDIGM19 ] . A frequentist analysis of the semiparametric exponential frailty model with nonparametric maximum likelihood estimator has been done in [ van96 ] . See for examples of general frailty models [ BP20 ] . The errors-in-variables model is the other example we will consider. In this model we want to run a regression of Y on Z , however, we only observe Y and a noisy version of Z . This means that Z is a latent random variable which we do not observe, only a noisy version of this. The mixture is given by p_{\alpha,\beta,\sigma,\tau,F}(x,y)=\int\phi_{\sigma}(x-z)\phi_{\tau}(y-\alpha-\beta z)\dd F(z), where \phi is the density of a standard normal Gaussian distribution. We are interested in estimating \alpha,\beta , while we consider F as a nuisance parameter. Thus, Errors-in-Variables is another semiparametric mixture model. This model is more of a fundamental nature, being able to do linear regression in the situation where the covariates are observed with noise. For more details on the model see Section 6 . Bayesian errors-in-variables have been considered before, see for example [ DS95 , MG96 , MR97 ] . Now we want to construct estimators which are efficient. One possible estimator is the full maximum likelihood estimator, which given a sample X_{1},\ldots,X_{n} from the distribution of X , maximizes the likelihood ({\theta},F)\mapsto{\prod_{i=1}^{n}}\int p_{\theta}(X_{i}\mathchar 25194\relax z)\,dF(z). These estimators were shown to be consistent in some generality in [ KW56 ] . For a few special models the {\theta} -component of the maximum likelihood estimator is known to be asymptotically normal and efficient from [ van88 , Mv96 , van96 ] . A Bayesian alternative is to equip F with a species sampling prior (See e.g. [ Gv17 , Chapter 14] ), and, independently \theta with a smooth prior. Two common choices for a mixture prior would be the Dirichlet process prior or a finite discrete prior with a random number of components. These two choices are the priors we will study in our examples. The Dirichlet process is a stochastic process characterised by two parameters, the prior precision M and the centre measure G . Then a random variable F is said to be Dirichlet process distributed if for any partition A_{1},\dots A_{k} (F(A_{1}),\dots,F(A_{k}))\sim\textrm{Dir}\left(k;MG(A_{1}),\dots,MG(A_{k})\right). As an example of a finitely discrete prior with a random number of components, pick \lambda,\sigma>0 and G a probability distribution. Then we draw K\sim\textrm{Geom}(\lambda) , and given K , draw Z_{1},\dots,Z_{K}\overset{\textrm{i.i.d.}}{\sim}G , W\sim\textrm{Dir}(K;\sigma,\dots,\sigma) . Then F=\sum_{k=1}^{K}W_{k}\delta_{Z_{k}} We will give more details on these priors in section 4 . In that section, we will also recall the facts that are needed for the purpose of this paper. More generally, a species sampling process can be defined using the exchangeable partition probability function (EPPF) and a centre measure G . There are many more examples of species sampling process priors, for example the Pitman-Yor processes, Gibbs processes or Poisson-Kingman processes. For more details, see [ Gv17 , Chapter 14] . 1.2 Semiparametric efficiency The goal of the statistician is to provide a reliable estimate for the parameter of inference \theta , and specify its uncertainty. We want to learn \theta as accurately as possible. The first notion we need to guarantee is consistency, that is, the ability to find the true parameter at all. After having established consistency, we want to understand if our method is asymptotically optimal. Efficiency will be how we will measure asymptotic optimality. Very roughly, this means that the posterior is as accurate as possible: there is no estimator which can be

Read the complete text