Is model selection possible for the ℓ p \ell_{p} -loss? PCO estimation for regression models
Claire Lacour, Pascal Massart, Vincent Rivoirard · 2025
Read Is model selection possible for the ℓ p \ell_{p} -loss? PCO estimation for regression models by Claire Lacour, Pascal Massart, Vincent Rivoirard (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-11217.txt.
Introduction
Lacour, Massart, and Rivoirard ask whether statisticians can still choose between models when error is measured in non-Euclidean ways, in this 2025 paper. Model choice usually leans on squared error. Many modern problems grade fits with other power losses, where the old guarantees may fail.
They study a procedure that penalizes comparison against overfitting, under noise that is at most Gaussian-like in its tails. Sharp oracle inequalities come from new concentration bounds for heavier-tailed variables. With well-chosen model collections and penalties, the method reaches the best possible rates over standard smoothness classes. The results are then carried over to curve fitting. The paper widens the safe ground for data-driven model choice.
Catalog record
| CREATOR | Claire Lacour, Pascal Massart, Vincent Rivoirard |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 133 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-11217
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Is model selection possible for the ℓ p \ell_{p} -loss? PCO estimation for regression models Claire Lacour, Pascal Massart, Vincent Rivoirard Abstract This paper addresses the problem of model selection in the sequence model Y = θ + ε ξ Y=\theta+\varepsilon\xi , when ξ \xi is sub-Gaussian, for non-euclidian loss-functions. In this model, the Penalized Comparison to Overfitting procedure is studied for the weighted ℓ p \ell_{p} -loss, p ≥ 1 . p\geq 1. Several oracle inequalities are derived from concentration inequalities for sub-Weibull variables. Using judicious collections of models and penalty terms, minimax rates of convergence are stated for Besov bodies ℬ r , ∞ s \mathcal{B}_{r,\infty}^{s} . These results are applied to the functional model of nonparametric regression. MSC Classification: - Primary: 62G05, 62C20 - Secondary: 62G08, 60E15 Keywords: Model selection, Oracle inequalities, Minimax rates, \ell_{p} -loss, Sub-Gaussian sequence model, Nonparametric regression 1 Introduction The problem of selecting a model from among several candidates is essential in statistics and machine learning, as well as in many application fields. In the most general sense, the aim of model selection is to construct data-driven criteria for selecting a model m from a given collection \mathcal{M} . In other words, if one observes some random variable \xi^{(n)} (which can be typically a random vector of size n ) with unknown distribution depending on some quantity f (the target) belonging to a set \mathcal{S} , a flexible approach to estimate f is to consider some collection of preliminary estimators \big(\widehat{f}_{m}\big)_{m\in\mathcal{M}} and then try to design some genuine data-driven procedure \widehat{m}\in\mathcal{M} to produce a new estimator \widehat{f}_{\widehat{m}} . Considering some loss function \ell , we measure the quality of each estimator \widehat{f}_{m} , through the quantity \ell\big(f,\widehat{f}_{m}\big) and mathematical results on estimator selection are formulated in terms of upper bounds on \ell\big(f,\widehat{f}_{\widehat{m}}\big) that allow to measure how far this quantity is from what is usually called the oracle risk \inf_{m\in\mathcal{M}}\mathbb{E}\,\ell\big[\big(f,\widehat{f}_{m}\big)\big] . These comparison inequalities are called oracle inequalities. The quadratic loss \ell\big(f,\widehat{f}_{m}\big)=\big\|f-\widehat{f}_{m}\big\|^{2} is a standard choice, but of course other losses are considered in the literature. Actually, in the classical model selection framework, developed and popularized by Birgé and Massart [ BM01 ] , [ BM07 ] , [ Mas07 ] , the list of estimators and the loss function are intimately related in the sense that they derive from the same contrast function (also called empirical risk in the machine learning literature). More precisely, a contrast function L_{n} is a function on the set \mathcal{S} depending on the observation \xi^{(n)} in such a way that g\in{\mathcal{S}}\longmapsto\mathbb{E}\left[L_{n}\left(g\right)\right] achieves a minimum at point f . Given some collection of subsets (S_{m})_{m\in\mathcal{M}} of \mathcal{S} , called models in the sequel, for every m\in\mathcal{M} , some estimator \widehat{f}_{m} of f is obtained by minimizing L_{n} over S_{m} ( \widehat{f}_{m} is called minimum contrast estimator or empirical risk minimizer ). In the case where \xi^{\left(n\right)}=\left(\xi_{1},\ldots,\xi_{n}\right) , an empirical criterion L_{n} can be defined as an empirical mean L_{n}\left(g\right)=P_{n}\left[L\left(g,.\right)\right]:=\frac{1}{n}\sum_{i=1}^{n}L\left(g,\xi_{i}\right)\text{,} which justifies the terminology of empirical risk. Empirical risk minimization includes maximum likelihood and least squares estimation. The penalized empirical risk selection procedure consists in considering some proper penalty function \operatorname*{pen} : \mathcal{M}\rightarrow\mathbb{R}_{+} and taking \widehat{m} minimizing L_{n}\left(\widehat{f}_{m}\right)+\mathrm{pen}(m) (1) over \mathcal{M} . We can then define the selected model S_{\widehat{m}} and the corresponding selected estimator \widehat{f}_{\widehat{m}} . Penalized criteria have been proposed in the early seventies by Akaike or Schwarz (see [ Aka73 ] and [ Sch78 ] ) for penalized maximum log-likelihood in the density estimation framework and Mallows for penalized least squares regression (see [ Mal73 ] ). In both cases the penalty functions are proportional to the number of parameters D_{m} of the corresponding model S_{m} . In such settings, the performance of model selection estimators can be studied for the “natural” (non negative) loss function \ell attached to L_{n} through the simple definition \ell\left(f,g\right)=\mathbb{E}[L_{n}(g)]-\mathbb{E}[L_{n}(f)],\quad g\in\mathcal{S}. (2) This approach has been successfully applied in many settings: density estimation or Poisson intensity estimation where \ell is the Kullback-Leibler divergence or the \mathbb{L}_{2} -loss, nonparametric regression for the \mathbb{L}_{2} -loss, binary classification for the 0-1 loss or the Gaussian white noise model still for the \mathbb{L}_{2} -loss. See [ Mas07 ] and references therein. Although it is not derived from an empirical contrast, we mention the model selection work associated with the Hellinger distance [ Bir86 ] , [ Bar11 ] . This explains why, although successfully applied in many settings, model selection procedures have only been defined and analyzed for very specific loss functions. In particular, non-Euclidian loss functions, as \mathbb{L}_{p} -losses, have rarely been considered for model selection procedures. It is not the case for other classical nonparametric estimation procedures. Although the use of \mathbb{L}_{p} -loss in nonparametric statistics goes back at least to [ BH79 , IH80 , Sto82 ] , an important advance was made by [ Nem85 ] who established optimal rates of convergence in the problem of multivariate nonparametric regression for \mathbb{L}_{p} -loss and functions belonging to \mathbb{L}_{q} -Sobolev classes with possibly p\neq q . Some years later, fundamental works have been made by Oleg Lepski, who proposed the so-called famous Lepski-method for selecting a bandwidth of a kernel estimator. In [ Lep91 ] , he studied adaptive estimation under \mathbb{L}_{p} -loss, 1\leq p\leq\infty over the collection of Hölder classes. Then [ LMS97 ] introduced a local bandwidth selection scheme to give kernel estimates which achieve optimal rates of convergence over Besov classes in the Gaussian white noise model, the anisotropic multivariate case being examined by [ KLP01 ] . Then [ GL08 ] developed a powerful methodology for selecting a bandwidth of a kernel estimate, which works in a multitude of contexts and allows to establish oracle inequalities and to derive optimal rates of convergence: see [ GL11 ] , [ GL13 ] , [ GL14 ] . Nevertheless, the implementation of this method needs two steps of minimization, each requiring a thorough calibration. Note that the estimation under the \mathbb{L}_{1} -loss for bandwidth-selected kernel estimators has been investigated in [ DL96 ] , and the \mathbb{L}_{p} -aggregation of estimators has been studied by [ Gol09 ] . In a very different spirit, another very popular method for adaptive estimation of functions is wavelet thresholding. In a series of papers, Donoho, Johnstone, Kerkyacharian and Picard have shown the power of wavelet thresholding for \mathbb{L}_{p} -estimation over the scale of Besov classes: see [ DJKP95 ] , [ DJKP96 ] , [ DJKP97 ] , [ DJ98 ] . Refinements have been proposed in [ Jud97 ] , [ KPT96 ] , [ HKP99 ] , [ JS05 ] , to name but a few. It must be noted that these thresholding methods generally suffer from logarithmic losses in the rates of convergence. We refer the reader to the book by [ HKPT98 ] which details the construction of wavelet bases, their use in statistical estimation, \mathbb{L}_{p} -minimax results as well as computational aspects. Johnstone’s recent book [ Joh19 ] addresses nonparametric function estimation by carefully studying the infinite Gaussian sequence model, with many results and thoughts about wavelet thresholding and adaptive minimaxity over ellipsoids. We finally mention the forthcoming manuscript [ ACE25 ] which considers Bayesian nonparametric concentration rates of