Sub-Gaussian Concentration and Entropic Normality of the Maximum Likelihood Estimator
Leighton P. Barnes 1 , Alex Dytso 2 · 2026
Read Sub-Gaussian Concentration and Entropic Normality of the Maximum Likelihood Estimator by Leighton P. Barnes 1 , Alex Dytso 2 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-07107.txt.
Introduction
Leighton P. Barnes and Alex Dytso strengthen a classic statistical result about maximum likelihood estimation. With large samples, such estimates are known to settle into a bell-shaped distribution. The authors prove stronger forms of that behavior under added assumptions on the score function behind the estimate.
They establish tight sub-Gaussian tail bounds and convergence of all moments for the normalized error. For a smoothed version of the estimator, they prove convergence in relative entropy to the limiting Gaussian law. Under further conditions, the smoothing can be dropped entirely. Their supporting tools, including entropy-control arguments, may interest other researchers.
Catalog record
| CREATOR | Leighton P. Barnes 1 , Alex Dytso 2 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.07107
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Sub-Gaussian Concentration and Entropic Normality of the Maximum Likelihood Estimator Leighton P. Barnes 1 , Alex Dytso 2 Abstract It is well known that, under standard regularity conditions, the maximum likelihood estimator (MLE) satisfies a central limit theorem and converges in distribution to a Gaussian random variable as the sample size grows. This paper strengthens this classical result by developing several stronger forms of asymptotic normality for the normalized MLE. With additional assumptions on the score, we first establish sub-Gaussian tail bounds and convergence of all moments for the normalized estimation error. We then prove an entropic central limit theorem for a smoothed version of the estimator, showing convergence in relative entropy to the limiting Gaussian law. When the Fisher information of the normalized estimate is bounded, or its density has bounded first derivative, we further show that the smoothing can be removed, yielding entropic normality of the MLE itself. The proofs develop auxiliary tools that may be of independent interest, including exponential consistency bounds, high-moment estimates, and entropy-control arguments for the estimator. I Introduction Suppose we have samples X_{1},\ldots,X_{n} drawn i.i.d. from a density f(x|\theta) with a parameter \theta\in\Theta\subset\mathbb{R} . The density f(x|\theta) is with respect to a base measure \mu on the sample space \mathcal{X} . This paper investigates the asymptotic behavior of the distribution of the maximum likelihood estimator (MLE). In particular, we study the distribution of the properly normalized estimation error Z_{n}=\sqrt{n}\,(\widehat{\theta}_{n}-\theta_{0}) as the sample size n\to\infty . Under standard regularity conditions, this quantity is known to satisfy a classical central limit theorem (CLT) and converge in distribution to a Gaussian random variable. The main goal of this paper is to strengthen this classical result in several directions. These strengthened results come at the expense of additional assumptions on the score, which can be considered natural in many statistical models. First, we establish sub-Gaussian tail behavior for the normalized estimation error Z_{n} , and use this control to upgrade convergence in distribution to convergence of moments. Second, we prove an entropic form of asymptotic normality for a smoothed version of the estimator, \widetilde{Z}_{n}=\sqrt{1-\varepsilon}Z_{n}+\sqrt{\varepsilon}Z, where Z\sim\mathcal{N}(0,1) is independent of Z_{n} . Finally, under an additional Fisher information (or bounded first derivative) regularity condition on the densities of Z_{n} , we remove the smoothing and obtain entropic normality of Z_{n} itself. Convergence in relative entropy is significantly stronger than convergence in distribution and belongs to a class of strong statistical divergences. In particular, convergence in relative entropy implies convergence in total variation distance via Pinsker’s inequality and convergence in Wasserstein distance via Talagrand’s inequality. I-A Prior Work The asymptotic normality of the MLE is a classical result in statistics and forms one of the cornerstones of parametric inference. Under suitable regularity conditions, it is well known that the properly normalized estimation error \sqrt{n}(\widehat{\theta}_{n}-\theta_{0}) converges in distribution to a Gaussian random variable whose variance is determined by the Fisher information [ 14 , 24 , 16 ] . This CLT result can be extended to a large class of estimators known as M-estimator and L-estimators; the interested reader is referred to classical text such as [ 22 , 17 ] . The entropic CLT for the sample mean was first studied by [ 18 ] and later considerably generalized by [ 6 ] . There has also been recent activity around finding the rates of convergence and the interested reader is referred to [ 2 , 15 , 19 , 8 , 13 , 10 ] and references therein. Besides the sample mean, the entropic CLT is also known to hold for all central order statistics [ 11 ] . Recent work in learning theory has studied the Kullback–Leibler divergence of MLEs, especially monotonicity properties [ 23 , 21 ] . However, those works consider KL divergence over the sample space between the distributions induced by the estimated and true parameters. This differs from our goal, which is to establish a stronger mode of convergence for the distribution of the MLE itself. I-B Outline Section II introduces the necessary preliminaries, formulates the problem, and presents several illustrative examples. Section III states our main result. Section IV is dedicated to proofs. In Section V we conclude and discuss some future directions for this work. The remainder of this section establishes the notation used throughout the paper. Random variables are denoted by upper case letters (e.g., X ), while specific deterministic values they can take are denoted by lower case letters (e.g., x ). Throughout the paper, logarithms are taken in the natural base. For two random variables W\sim f_{W} and V\sim f_{V} , the relative entropy (KL divergence) is defined as D(W\|V)=\int_{\mathbb{R}}f_{W}(x)\log\frac{f_{W}(x)}{f_{V}(x)}\,dx. We denote by \mathcal{N}(0,1) the standard normal distribution and use Z to denote a standard normal random variable. Convergence in distribution is denoted by \xrightarrow{d} .We use the standard asymptotic notation: for two functions f and g , we write f(n)=O(g(n)) if there exists a constant C>0 and n_{0} such that |f(n)|\leq Cg(n) for all n\geq n_{0} . II Problem Definition and Some Preliminaries In this section we set up the required technical preliminaries. We will make a number of definitions and assumptions here that are standard in the existing literature/results regarding the asymptotic normality of the MLE. The more nonstandard assumptions that we make regarding the tail behavior of the score and the smoothness of the statistical model we enumerate explicitly in Section II-A . We assume the density f(\cdot|\theta) has common support at each \theta (or equivalently we just assume f(x|\theta)>0 , since we can ignore parts of the sample space with zero probability density). We consider a parameter space \Theta that is an open subset of \mathbb{R} and that has a compact closure K with |\theta|<B for all \theta\in K . Let \displaystyle L_{n}(\theta)=\frac{1}{n}\sum_{i=1}^{n}\log f(X_{i}|\theta) (1) be the normalized log-likelihood. Let \widehat{\theta}_{n}(X_{1},\ldots,X_{n}) be a maximum likelihood estimate of \theta from the data X_{1},\ldots,X_{n} in the sense that \widehat{\theta}_{n}\in\arg\max_{\theta\in K}L_{n}(\theta). (2) We assume that \log f(x|\theta) is twice-differentiable as a function of \theta on the set \Theta , and that it is continuous on the closure K (for almost all x\in\mathcal{X} ). By this continuity, L_{n}(\theta) achieves its maximum over K . Note that if \widehat{\theta}_{n} is in \Theta itself instead of just its closure, then it satisfies \displaystyle L^{\prime}_{n}(\widehat{\theta}_{n})=0\;. (3) The key quantity that we seek to study is the normalized error given by Z_{n}=\sqrt{I_{X}(\theta_{0})n}(\widehat{\theta}_{n}-\theta_{0})\;, (4) where I_{X}(\theta_{0}) is the Fisher information defined as \displaystyle I_{X}(\theta_{0})=-\mathbb{E}_{X|\theta_{0}}\left[s_{\theta_{0}}^{\prime}(X)\right]. (5) and (\theta,x)\mapsto s_{\theta}(x) is the score function given by s_{\theta}(x)=\frac{\partial}{\partial\theta}\log f(x|\theta). (6) The classical CLT for the maximum likelihood estimate can be stated as follows: Z_{n}\xrightarrow{d}Z (7) as n\to\infty where Z\sim\mathcal{N}(0,1) . We assume the Fisher information at this true parameter is nonzero. Finally, for simplicity we will assume that the model f(x|\theta) is identifiable in the sense that f(\cdot|\theta)=f(\cdot|\theta^{\prime}) only when \theta=\theta^{\prime} . The above assumptions are rather standard for the MLE setup. II-A Additional Regularity Conditions We now collect the additional assumptions used to upgrade the classical distributional CLT to convergence in relative entropy. Assumption 1 gives the uniform concentration needed for exponential consistency. Assumptions 1 and 2 control the second-order Taylor expansion of the likelihood and yield the high-moment bounds. Assumption 3 is used only