Predictive Inference via Kernel Density Estimates
Torey Hilbert · 2026
Read Predictive Inference via Kernel Density Estimates by Torey Hilbert (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-14008.txt.
Introduction
Hilbert studies kernel density estimation as a rule for Bayesian predictive inference, where each predictive distribution for the next observation is specified from past data. He examines two kernel-based rules, the classic estimator and a recursive version built for online problems, and shows both converge weakly almost surely. That convergence opens new Bayesian readings of kernel density estimation, though the classic rule lands on a compactly supported measure while the recursive one does not.
Catalog record
| CREATOR | Torey Hilbert |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22731 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.14008
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Predictive Inference via Kernel Density Estimates Torey Hilbert Abstract Kernel density estimation is a widely used nonparametric approach to estimate an unknown distribution. Recent work in Bayesian predictive inference has considered stochastic processes formed by specifying the predictive distribution for the next data point given all observed data such that the resulting predictive distributions converge weakly almost surely. We study two kernel based prediction rules: the classic kernel density estimator, and a recursive version previously introduced for online problems. We show that both processes converge weakly almost surely, which opens the door for new Bayesian interpretations of kernel density estimation. Surprisingly, the process based on the classic kernel density estimates converges to a compactly supported measure, while the recursive version converges to a non-compactly supported measure. 1 Introduction Recent work on Bayesian predictive inference has considered modeling data by specifying a collection of prediction rules for X_{n+1}|X_{1:n} , which when supplemented with a distribution over X_{1} gives a joint distribution over an infinite sequence of data X_{1:\infty} . Denoting the (random) predictive measures by P_{n}(B)=\mathbb{P}(X_{n+1}\in B|X_{1:n}) for Borel sets B\in\mathcal{B}(\mathbb{R}^{d}) , if the sequence of predictive measures P_{n} almost surely converges weakly to some (random) probability measure P , then the conditional limiting predictive measure P|X_{1:n} is a posterior distribution for a Bayesian analysis ( 7 ; 6 ; 8 ) . The recent Bayesian predictive inference literature has largely focused on predictive distributions satisfying the conditionally identically distributed (CID) property ( 3 ; 6 ; 8 ) , or on predictive distributions that are “almost” a CID sequence as in 1 . Part of the appeal of the Bayesian predictive inference framework, however, is the idea that the analyst could use any predictive distributions that they feel are “natural”. When handed a non-parametric prediction problem, the first approach for many statisticians is to use the empirical distribution, leading to Bayesian bootstrap methods ( 5 ; 12 ; 10 ) . These methods are generalized in the works on measure-valued Pólya sequences ( 2 ; 13 ; 4 ) . A second natural approach for many statisticians is to smooth the empirical distribution, for example with a kernel density estimator ( 14 ) . Let K be a probability measure on \mathbb{R}^{d} , and let h_{n}>0 be a sequence of bandwidths. For each n and Borel set B\in\mathcal{B}(\mathbb{R}^{d}) , we consider the predictive rule \displaystyle\mathbb{P}(X_{n+1}\in B|X_{1:n})=\frac{1}{n}\sum_{i=1}^{n}K\bigg(\frac{B-X_{i}}{h_{n}}\bigg). (1) Here we use the notation (B-X_{i})/{h_{n}}=\{(b-X_{i})/h_{n}:b\in B\}\subseteq\mathbb{R}^{d} . To fully specify the model we will arbitrarily set X_{1}=0 . Denote the sequence of random predictive measures by P_{n}(B)=\mathbb{P}(X_{n+1}\in B|X_{1:n}) . For example, if K=N(0,1) on \mathbb{R} , this is equivalent to using the probability density function \displaystyle f(x|X_{1:n})=\frac{1}{n}\sum_{i=1}^{n}\frac{1}{h_{n}}\frac{1}{\sqrt{2\pi}}\exp\bigg(-\frac{1}{2}\bigg(\frac{x-X_{i}}{h_{n}}\bigg)^{2}\bigg), the classic kernel density estimator (KDE). In this work, we show that for modestly heavy-tailed kernels K and bandwidths h_{n} that decay at order n^{-\delta} for some \delta>0 , the sequence of predictive distributions defined by the KDE process above converges weakly almost surely to a limiting random measure P . Furthermore, with additional moment assumptions, the limiting random measure P is supported on a random compact set. While the almost sure weak convergence of P_{n} is expected, the fact that the limiting measure P is almost surely compactly supported was surprising to us. While our primary interest is model 1 , we also consider a recursive kernel estimator introduced by 16 . For standard implementations, the KDE process and this alternative recursive kernel process behave qualitatively differently, since the limiting measure of the recursive kernel process almost surely has non-compact support. For the recursive kernel estimator, 1 find that for very rapidly decaying bandwidths, such as h_{n}=e^{-n} , the sequence of predictive measures converges weakly almost surely for some kernels K . However, their approach does not work for h_{n} satisfying nh_{n}\to\infty , which is a well-known condition for frequentist point-wise consistency of the KDE ( 14 ) . We show, under much weaker conditions, that the recursive kernel estimator also converges weakly almost surely. 2 Kernel Density Process Another way to write the same sequence of variables in model 1 is as follows: \displaystyle Y_{1:\infty} \displaystyle\overset{\mathrm{iid}}{\sim}K, \displaystyle M_{n} \displaystyle\overset{ind}{\sim}\text{Unif}(\{1,\ldots,n\}), \displaystyle X_{n+1} \displaystyle=X_{M_{n}}+h_{n}Y_{n}. In contrast to X_{1:\infty} , we also consider a different process \tilde{X}_{1:\infty} defined by recursive kernel estimates, as in 16 ; 1 . We will use a tilde to differentiate objects connected to X_{1:\infty} vs \tilde{X}_{1:\infty} . We define the recursive kernel process by \displaystyle\tilde{P}_{n}(B)=\mathbb{P}(\tilde{X}_{n+1}\in B|\tilde{X}_{1:n})=\bigg(1-\frac{1}{n}\bigg)\tilde{P}_{n-1}(B)+\frac{1}{n}K\bigg(\frac{B-\tilde{X}_{n}}{h_{n}}\bigg). (2) Again we arbitrarily select \tilde{X}_{1}=0 to start the sequence. The key difference between X_{1:\infty} and \tilde{X}_{1:\infty} is that the former model uses the same bandwidth h_{n} around all points and recomputes the KDE at every step, while the latter model never updates the bandwidth for an already observed point in the sequence. Subjectively, we prefer the classic KDE approach defining the sequence X_{1:\infty} , but, as the two models have qualitatively different asymptotic behavior, we analyze both. Another way to write the recursive process is \displaystyle Y_{1:\infty} \displaystyle\overset{\mathrm{iid}}{\sim}K, \displaystyle M_{n} \displaystyle\overset{ind}{\sim}\text{Unif}(\{1,\ldots,n\}), \displaystyle\tilde{X}_{n+1} \displaystyle=\tilde{X}_{M_{n}}+h_{M_{n}}Y_{n}. The following two results show that both the classic KDE predictive rule and the recursive predictive rule are asymptotically exchangeable, and thus can be used in the Bayesian predictive inference paradigm laid out by 8 . Theorem 1 (Weak convergence for KDE process) . Suppose 0<h_{n}\leq Cn^{-\delta} for some C,\delta>0 , and suppose \mathbb{E}(\|Y_{1}\|)<\infty . Then \mathbb{P}(P_{n}\overset{weak}{\to}P)=1 for some random probability measure P on \mathbb{R}^{d} . Additionally, if \mathbb{E}(\|Y_{1}\|^{1/c})<\infty for some 0<c<\delta , then P is almost surely supported on a (random) compact set V , and \mathbb{P}(\sup_{n\in\mathbb{N}}\|X_{n}\|<\infty)=1 . Theorem 2 (Weak convergence for recursive process) . Suppose 0<h_{n}\leq Cn^{-\delta} for some C,\delta>0 , and suppose \mathbb{E}(\|Y_{1}\|)<\infty . Then \mathbb{P}(\tilde{P_{n}}\overset{weak}{\to}\tilde{P})=1 for some random probability measure \tilde{P} on \mathbb{R}^{d} . A common choice for densities on \mathbb{R} is h_{n}=O(n^{-1/5}) , which in a frequentist setting minimizes asymptotic mean integrated square error under regularity conditions e.g., 11 ; 14 . Our theorems establish that, for any choice of kernel K with finite first moment, both the KDE process P_{n} and the recursive process \tilde{P}_{n} will converge weakly almost surely. Furthermore, if K has finite sixth moments then P almost surely has compact support. For densities on \mathbb{R}^{d} , typically we use h_{n}=O(n^{-1/(d+4)}) , so that if \|Y_{1}\| has d+4 moments then P has compact support. Lastly, when K has exponential tails, P has compact support for any \delta>0 . Typically one would use a kernel K with \mathbb{E}[Y_{1}]=0 , but to demonstrate a critical difference between the KDE process and the recursive kernel process, we will consider an artificial example with K([0,\infty))=1 that is substantially more straightforward mathematically. Let K=|N(0,1)| , the absolute value of a N(0,1) random variable, and set h_{n}=n^{-\delta} . Immediately, we see that \tilde{P}_{n} will always include a \frac{1}{n}|N(0,h_{1}^{2})| term coming from X_{1}=0 , so by the Borel-Cantelli lemma with \sum_{n\in\mathbb{N}}1/n=\infty , we have \mathbb{P}(\sup_{n\in\mathbb{N}}\|\tilde{X}_{n}\|=\infty)=1 , in contrast to \mathbb{P}(\sup_{n\in\mathbb{N}}\|X_{n}\|<\infty)=1 . Fix M and suppose \tilde{X}_{n}>M for some n . Let \tilde{p}_{m}=\frac{1}{m}|\{j\geq n:\tilde{X}_{j}\text{ is a descendant of }\tilde{X}_{n}\}| . Using a Polya urn scheme, we get \tilde{p}_{m}\overset{a.s.}{\to}\tilde{p}\sim\text{Beta}(1,n-1) . Then \displaystyle\tilde{P}_{m}(\{x:|x|>M\})\geq\tilde{p}_{m}\not\to 0. But for any M>0 , almost surely \tilde{X}_{n}>M for some n . Thus the limiting measure \tilde{P} is not compactly supported. The same result holds for K=N(0,1) ,