Adaptive Kernel Density Estimation with Pre-training

Ruitong Zhang, Ke Deng · 2026

Read Adaptive Kernel Density Estimation with Pre-training by Ruitong Zhang, Ke Deng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13092.txt.

Introduction

Zhang and Deng bring pretraining to kernel density estimation in high dimensions, where classic kernel smoothing struggles to pick kernels that fit each spot. Their pretrained neural network recommends a suitable kernel for each sample point, making efficient adaptive estimation possible. Numerical tests show large accuracy gains when the target distribution resembles the pretraining family. When the target differs substantially, gains dilute but return with an extra fine-tuning step.

Catalog record

CREATORRuitong Zhang, Ke Deng
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41196
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.13092

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Adaptive Kernel Density Estimation with Pre-training Ruitong Zhang, Ke Deng Abstract Density estimation in high-dimensional settings is an important and challenging statistical problem. Traditional methods based on kernel smoothing are inefficient in high dimensions due to the difficulties in specifying appropriate location-adaptive kernels. In this work, we introduce pre-training, a key idea behind many cutting-edge AI technologies, to the context of non-parametric density estimation. By establishing a pre-trained neural network that can recommend an appropriate location-adaptive kernel for each sample point, efficient density estimation with adaptive kernels is achieved in high dimensions. A wide range of numerical experiments show that this strategy is highly effective for improving density-estimation accuracy, when the target distribution is close to the distribution family for pre-training. When the target distribution is substantially different from the pre-training distribution family, the benefit from the proposed pre-training strategy may be diluted, but can be reactivated by an additional fine-tuning procedure. 1 Introduction Nonparametric density estimation is a fundamental problem in statistics and machine learning. Given n i.i.d. samples X_{1:n}=(X_{1},\cdots,X_{n}) from a continuous target distribution F_{0} on \mathbb{R}^{d} , the goal is to estimate f_{0} , the density function of F_{0} , in a non-parametric fashion. In the statistical literature, a rich collection of methods have been proposed for this important problem, including kernel density estimation (KDE) ( 20 ; 18 ) and smoothing splines ( 29 ; 10 ) . In parallel, modern machine learning has developed expressive neural density estimators, including autoregressive models ( 8 ; 17 ) , normalizing flows ( 6 ) , and score-based generative models ( 26 ) . Among these methods, KDE remains attractive in many applications, because it enjoys an explicit analytic form with easy evaluation and well-developed statistical theory ( 25 ; 30 ) . Like many other methods in this field, however, KDE typically suffers significant performance degradation due to the curse of dimensionality. When d , the dimensionality of the target distribution, is large, the geometry of f_{0} is difficult to capture, and the kernel function controls not only the amount of smoothing, but also the orientation and anisotropy of the estimator ( 30 ; 4 ) . In these cases, using a common global kernel to smooth all regions of the sample space is often inadequate, especially when the target density has heterogeneous local scale or anisotropic geometry: sharp high-density regions may require narrow and directionally sensitive kernels, while sparse or diffuse regions may require flatter kernel functions. Adaptive KDE methods ( 27 ) mitigate this problem by allowing the bandwidth to vary across locations, with two common formulations being balloon estimators and sample-point estimators ( 13 ) . Existing adaptive KDE methods often construct these adaptive bandwidths from pilot density estimates ( 1 ) , nearest-neighbor distances ( 3 ) , or other local geometric information. Although statistically well motivated, these approaches typically adapt restricted kernel structures rather than learning a general rule to recommend appropriate location-adaptive kernels. In this work, we fill this gap by introducing pre-training , a key idea behind many cutting-edge AI technologies, to the context of non-parametric density estimation. By establishing a pre-trained neural network \phi_{\theta} with \theta as parameters that can recommend an appropriate location-adaptive kernel for each observed sample X_{i} based on its local geometry in X_{1:n} , we successfully resolve the problem of location-adaptive kernel specification that is often extremely difficult in the traditional learning framework with a limited number of samples. With the support of \phi_{\theta} , which encodes information learned during pre-training beyond the limited information contained in the observed sample X_{1:n} , efficient adaptive kernel density estimation can be achieved in relatively high-dimensional settings. To establish the pre-trained neural network \phi_{\theta} for kernel recommendation, we prepare a distribution family {\cal F} that covers a wide range of distributions of potential interest, and generate a large collection of synthetic data {\cal D} based on {\cal F} for pre-training of \phi_{\theta} . Numerical experiments show that this neural-network-guided KDE (NNKDE) significantly outperforms existing KDE methods when the target distribution F_{0} is close to {\cal F} . When F_{0} is far away from {\cal F} , the advantage of NNKDE may be diluted, but it can be reactivated by an additional fine-tuning procedure, where the global scale of the recommended location-adaptive kernels can be further adjusted under the guidance of self-exclusive leave-one-out likelihood of X_{1:n} . Such a “pre-training + fine-tuning” strategy further enhances the adaptability of the proposed NNKDE, providing a powerful tool for non-parametric density estimation. Our contributions are threefold. First, we formally introduce the idea of pre-training to non-parametric statistics and demonstrate its ability to improve the efficiency of statistical inference using synthetic data. Second, we show that the benefits of pre-training for non-parametric statistics can be further preserved and enhanced by an additional fine-tuning procedure, suggesting that the widely used “pre-training + fine-tuning” strategy may also be beneficial for statistical models. Third, we provide a systematic evaluation protocol to empirically compare different KDE methods in various settings. 2 Related Works Bandwidth selection and adaptive KDE. Kernel density estimation has a long history in nonparametric statistics, with classical treatments covering both theoretical properties and practical bandwidth selection ( 25 ; 30 ) . In multivariate KDE, the bandwidth matrix is the central smoothing parameter, controlling scale, orientation, and anisotropy of the estimator ( 30 ; 4 ) . Classical bandwidth-selection rules include rule-of-thumb methods, plug-in selectors, and cross-validation-based criteria ( 25 ; 24 ; 21 ; 2 ) . Extensions to full bandwidth matrices have also been studied, including data-driven smooth cross-validation methods ( 7 ) . However, a single global bandwidth matrix can be inadequate for heterogeneous densities with spatially varying smoothness, and bandwidth optimization can be numerically unstable in multivariate settings ( 11 ) . Adaptive KDE methods address this limitation by allowing the bandwidth to vary locally. Classical examples include balloon estimators ( 15 ) and sample-point estimators ( 3 ; 22 ) . The latter are particularly relevant here because, under standard constructions, they retain a properly normalized density form ( 27 ) . Existing adaptive methods often rely on pilot estimates or scalar local rescaling rules, such as Abramson’s square-root law ( 1 ; 27 ) . In contrast, our work learns observation-specific positive definite bandwidth matrices within an explicit sample-point KDE. Neural density estimation. A separate line of work uses neural networks to construct expressive density estimators. Autoregressive density estimators such as Masked Autoencoder for Distribution Estimation (MADE) model the joint density through conditional factorizations ( 8 ) . Normalizing flows, including Real-valued Non-Volume Preserving transformations (Real NVP) ( 6 ) and masked autoregressive flow (MAF) ( 17 ) , learn invertible transformations from simple base distributions to complex target distributions. Score-based generative models learn the score function and generate samples through stochastic dynamics ( 26 ) . Other neural approaches have been developed for high-dimensional density estimation using deep generative models ( 14 ) . These approaches rely on neural approximation, but they primarily target direct density, score, or generative-map estimation. NNKDE differs in that the neural component is embedded inside an explicit KDE estimator rather than replacing the estimator itself. Amortized and pre-trained density learning. The pre-training

Read the complete text