Adaptive Density Estimation Using Projection Kernels and Penalized Comparison to Overfitting

Van Ha Hoang , Tien Dat Nguyen , Thi Mong Ngoc Nguyen · 2025

Read Adaptive Density Estimation Using Projection Kernels and Penalized Comparison to Overfitting by Van Ha Hoang , Tien Dat Nguyen , Thi Mong Ngoc Nguyen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-07800.txt.

Introduction

Van Ha Hoang and 2 co-authors design an adaptive method in 2025 for estimating a probability density with wavelet projection estimators. Density estimation rebuilds the shape of randomness from samples. Wavelet projection estimators build the estimate at a chosen resolution level. That choice sets the bias-variance trade-off. Too coarse blurs real features, and too fine fits noise.

Their selection rule is called Penalized Comparison to Overfitting. Each candidate is compared in the L2 norm with a single overfitting reference. An explicit penalty corrects the comparison. For the chosen estimator, the authors prove oracle inequalities. One holds with high probability, and one holds for expected risk. Over Besov classes of densities, the risk reaches the classical minimax rate. It adapts to the unknown smoothness. Numerical tests on several density shapes confirm the data-driven level sits close to the oracle choice.

Catalog record

CREATORVan Ha Hoang , Tien Dat Nguyen , Thi Mong Ngoc Nguyen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-07800

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Adaptive Density Estimation Using Projection Kernels and Penalized Comparison to Overfitting Van Ha Hoang , Tien Dat Nguyen , Thi Mong Ngoc Nguyen Abstract In this work, we study wavelet projection estimators for density estimation, focusing on their construction from 𝒮 \mathcal{S} -regular, compactly supported wavelet bases. A key aspect of such estimators is the choice of the resolution level, which controls the balance between bias and variance. To address this, we employ the Penalized Comparison to Overfitting (PCO) method, providing a fully data-driven selection of the multiresolution level. This approach ensures both statistical accuracy and computational feasibility, while retaining the adaptability of wavelet methods. Our results establish an oracle inequality for the proposed estimator and show that it attains the optimal rate of convergence in density estimation. Keywords: Wavelet projection estimator; Density estimation; Penalized Comparison to Overfitting (PCO); Oracle inequality; Adaptive methods; 1 Introduction The selection of estimators has long been recognized as a central challenge in nonparametric estimation, where the quality of an estimator crucially depends on the choice of certain tuning parameters. Such parameters typically take the form of the bandwidth in kernel estimation, the resolution level in projection methods, or the thresholding level in wavelet estimation. Determining these parameters in a data-driven manner is essential for achieving adaptivity, that is, ensuring that the estimator automatically adjusts to the unknown regularity of the underlying function. Several major approaches to adaptive estimation have been proposed in the literature. One classical route involves the minimization of penalized empirical risk, such as penalized likelihood or penalized least squares. Another widely studied approach is based on pairwise comparison of estimators, with Lepski’s method [ 15 ] being a landmark contribution that provides an adaptive solution for bandwidth selection in kernel estimation. In parallel, Donoho and collaborators [ 7 , 8 ] initiated a major line of research on wavelet thresholding, which has since become a cornerstone for adaptive estimation in high-dimensional and irregular contexts. Furthermore, the minimum complexity principle introduced by Barron and Cover [ 2 ] —later developed into the general framework of model selection by Birgé and Massart [ 1 , 3 ] —has provided profound insights into the intimate connection between adaptivity and penalization. More recently, attention has shifted towards designing fully data-driven procedures for tuning parameter selection, with particular emphasis on bandwidth and resolution choice. In kernel estimation, Lerasle [ 16 ] proposed an optimal penalty formulation for bandwidth selection via penalized least squares. The Goldenshluger–Lepski (GL) method [ 11 ] further advanced this field by combining pairwise comparison with an implicit bias-variance calibration. Lacour and Massart [ 13 ] clarified the role of the so-called minimal penalty within this framework, while Comte and Rebafka [ 6 ] pointed out some computational limitations of the GL procedure. In response, Lacour, Massart, and Rivoirard [ 14 ] introduced the Penalized Comparison to Overfitting (PCO) method, which compares candidate estimators to an overfitting reference and adjusts for the induced bias through an explicit penalty. This approach has several appealing features: it is computationally tractable, avoids the instability of cross-validation, and does not rely on plug-in estimates of unknown quantities. Theoretical guarantees such as oracle inequalities and minimax rates have been established for PCO, and subsequent works [ 18 , 5 ] have expanded its scope and demonstrated its empirical efficiency. Despite these advances, the application of PCO to wavelet methods remains relatively unexplored. Wavelets offer a powerful representation system for functions and densities, combining localization in both time and frequency domains. They are particularly well suited for capturing heterogeneous features such as smooth trends, local irregularities, or boundary effects. Classical histogram or kernel estimators are often less flexible in such scenarios, whereas wavelet-based methods provide a natural multiresolution framework that adapts to varying smoothness. The thresholding paradigm for wavelet coefficients has already proved successful in signal processing and density estimation. However, the specific problem of selecting the wavelet resolution level in a principled, fully data-driven way via PCO has not been addressed in the literature. This leaves a significant gap in adaptive estimation theory: while bandwidth selection by PCO is now well understood, its extension to multiresolution settings is largely open. In this work, we aim to fill this gap by developing an adaptive wavelet density estimator in which the resolution level is selected according to a criterion inspired by the PCO methodology. We establish non-asymptotic oracle inequalities and derive convergence rates over Besov classes, thereby demonstrating that the procedure achieves near-optimal risk bounds while retaining computational simplicity. Numerical experiments further illustrate the estimator’s robustness across a range of models with different structural properties. The paper is organized as follows. In Section 2.1 , we introduce the collection of wavelet projection estimators and describe their basic properties. Section 3 presents the PCO-inspired selection rule for choosing the resolution level. Theoretical guarantees, including an oracle inequality and convergence rates, are established in Section 4 . Section 5 reports numerical experiments that highlight the practical performance of the method. Notations. In the sequel, if there is no specification, \left\|\cdot\right\| and \langle\cdot,\cdot\rangle denotes the norm and the scalar product in \mathbb{L}^{2}(\mathbb{R}) , respectively. 2 Construction of the collection of wavelet estimators 2.1 Approximation kernel and collection of density estimators Let X_{1},X_{2},\ldots,X_{n} be an i.i.d. sample of size n drawn from a density function f . We aim to construct a wavelet projection estimator of f . Let \varphi\in\mathbb{L}^{2}(\mathbb{R}) be a scaling function such that \{\varphi(\cdot-k):k\in\mathbb{Z}\} forms an orthonormal basis of \mathbb{L}^{2}(\mathbb{R}) . For any N\in\mathbb{N} and k\in\mathbb{Z} , define the dilated and translated functions \varphi_{N,k}(x)=2^{N/2}\varphi(2^{N}x-k),\quad x\in\mathbb{R}. The linear spaces V_{N}:=\operatorname{span}\{\varphi_{N,k}:k\in\mathbb{Z}\},\quad N\in\mathbb{N}, constitute a multiresolution analysis, i.e. V_{N}\subset V_{N+1} for all N\in\mathbb{N} . The associated projection kernel on V_{N} is given by K_{N}(x,y):=\sum_{k\in\mathbb{Z}}\varphi_{N,k}(x)\,\varphi_{N,k}(y),\quad x,y\in\mathbb{R}. For any f\in\mathbb{L}^{2}(\mathbb{R}) , the orthogonal projection of f onto V_{N} is f_{N}(x):=K_{N}(f)(x)=\int_{\mathbb{R}}K_{N}(x,y)f(y)\,dy=\sum_{k\in\mathbb{Z}}\alpha_{N,k}\,\varphi_{N,k}(x), (1) where the wavelet coefficients are \alpha_{N,k}=\int_{\mathbb{R}}f(y)\,\varphi_{N,k}(y)\,dy. Given the sample \{X_{1},\dots,X_{n}\} , an estimator \widehat{f}_{N} of f_{N} is proposed as \widehat{f}_{N}(x)\equiv\widehat{f}_{N,n}(x):=\sum_{k\in\mathbb{Z}}\widehat{\alpha}_{N,k}\,\varphi_{N,k}(x), (2) with empirical coefficients \widehat{\alpha}_{N,k}:=\frac{1}{n}\sum_{u=1}^{n}\varphi_{N,k}(X_{u}). If the scaling function \varphi is compactly supported (e.g., Daubechies wavelets), only finitely many terms in the above sum are nonzero for each fixed x , which makes the estimator computationally feasible. Moreover, for every N and k , we have \mathbb{E}[\widehat{\alpha}_{N,k}]=\alpha_{N,k} , so that \widehat{f}_{N} is an unbiased estimator of the projection f_{N} . Since the spaces V_{N} are nested, they can be written as V_{N}=V_{0}\,\oplus\,\left(\bigoplus_{\ell=0}^{N-1}W_{\ell}\right), where W_{\ell}:=V_{N+1}\ominus V_{N} . Let \psi be the mother wavelet such that for every \ell\in\mathbb{N}\cup\{0\} , \big\{\psi_{\ell k}(.)=2^{\ell/2}\psi(2^{\ell}(.)-k):k\in\mathbb{Z}\big\} is an orthonormal basis that spans W_{\ell} . Then \big\{\varphi(.-k),\psi_{\ell k}(.)=2^{\ell/2}\psi(2^{\ell}(.)-k):k\in\mathbb{Z},\ell\in\mathbb{N}\cup\{0\}\big\} is an orthonormal wavelet basis of L^{2}(\mathbb{R}) , then the projection kernel of f onto V_{N} can be also represented as K_{N}(f)(x)=\sum_{k\in\mathbb{Z}}\alpha_{k}\varphi_{k}(x)+\sum_{\ell=0}^{N-1}\sum_{k\in\mathbb{Z}}\beta_{\ell k}\psi_{\ell k}(x), (3) where \varphi_{k}=\varphi_{0k} and \alpha_{k}=\alpha_{0k} . Consequently, we obtain an equivalent form for the estimator \hat{f}_{N} introduced in Equation ( 2 ): \widehat{f}_{N}(x)=\sum_{k\in\mathbb{Z}}\hat{\alpha}_{k}\varphi_{k}(x)+\sum_{\ell=0}^{N-1}\sum_{k\in\mathbb{Z}}\hat{\beta}_{\ell k}\psi_{\ell k}(x), (4) where \hat{\alpha}_{k}=\hat{\alpha}_{0,k} and \hat{\beta}_{\ell k}=n^{-1}\sum_{u=1}^{n}\psi_{\ell k}(X_{u}) . More discussion on these two expressions can be found in Giné and Nickl [ 9 , 10 ] . The second expression of \hat{f}_{N} is useful for the proof of Proposition 6.2 . 2.2 Technical conditions In this work, we consider a scaling fucntion (father wavelet) \varphi on the real line satisfying the following conditions (A1) The

Read the complete text