Estimating a regression function under possible heteroscedastic and heavy-tailed errors. Application to shape-restricted regression.

Yannick BARAUD, Guillaume MAILLARD · 2025

Read Estimating a regression function under possible heteroscedastic and heavy-tailed errors. Application to shape-restricted regression. by Yannick BARAUD, Guillaume MAILLARD (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00852.txt.

Introduction

Yannick Baraud and Guillaume Maillard tackle in this 2025 statistics paper a weak point of the classic least squares method. Least squares fits a regression line well when errors are tame, but real errors can vary in spread across points and carry heavy tails. The authors build a stand-in estimator for that rough setting and prove a risk bound for it that reads like an oracle promise.

Their estimator adapts to the best members of a candidate class, as least squares does for tame data. Yet it stays steady when error spread varies, and can even converge at a fast rate where least squares falls apart. Tests on shape-tied classes, such as monotone and convex fits, round out the case.

Catalog record

CREATORYannick BARAUD, Guillaume MAILLARD
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE127 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-00852

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Estimating a regression function under possible heteroscedastic and heavy-tailed errors. Application to shape-restricted regression. Yannick BARAUD, Guillaume MAILLARD Abstract . We consider a regression framework where the design points are deterministic and the errors possibly non-i.i.d. and heavy-tailed (with a moment of order p p in [ 1 , 2 ] [1,2] ). Given a class of candidate regression functions, we propose a surrogate for the classical least squares estimator (LSE). For this new estimator, we establish a nonasymptotic risk bound with respect to the absolute loss which takes the form of an oracle type inequality. This inequality shows that our estimator possesses natural adaptation properties with respect to some elements of the class. When this class consists of monotone functions or convex functions on an interval, these adaptation properties are similar to those established in the literature for the LSE. However, unlike the LSE, we prove that our estimator remains stable with respect to a possible heteroscedasticity of the errors and may even converge at a parametric rate (up to a logarithmic factor) when the LSE is not even consistent. We illustrate the performance of this new estimator over classes of regression functions that satisfy a shape constraint: piecewise monotone, piecewise convex/concave, among other examples. The paper also contains some approximation results by splines with degrees in { 0 , 1 } \{0,1\} and VC bounds for the dimensions of classes of level sets. These results may be of independent interest. 1991 Mathematics Subject Classification 62G05, 62G08, 62G35. 1. Introduction In this paper, we consider the problem of estimating a regression function f^{\star} on some set {\mathcal{X}} under a fixed design setting with independent and centred real-valued errors \xi_{1,n},\ldots,\xi_{n,n} . More precisely, we choose n deterministic points x_{1,n},\ldots,x_{n,n} in {\mathcal{X}} (possibly not all different) and associate each of them with a response Y_{i,n} with values in {\mathbb{R}} of the following form: (1) Y_{i,n}=f^{\star}(x_{i,n})+\xi_{i,n}\quad\text{so that}\quad f^{\star}(x_{i,n})={\mathbb{E}}\left[{Y_{i,n}}\right]\quad\text{for all $i\in\{1,\ldots,n\}$.} We only assume that the \xi_{i,n} are independent (not necessarily i.i.d.) and that all of them have a finite moment of order p\in[1,2] with {\mathbb{E}}[\xi_{i,n}]=0 . On the basis of these observations, our aim is to estimate the so-called regression function f^{\star} from {\mathcal{X}} to {\mathbb{R}} . Our approach is based on a model {\mathcal{F}} for f^{\star} , which means that we choose some family {\mathcal{F}} of candidate regression functions and pretend that f^{\star} belongs to {\mathcal{F}} , although this may not be true. To get a good estimator of f^{\star} , it actually suffices that {\mathcal{F}} provides a good enough approximation of it. For instance, {\mathcal{F}} may be a finite dimensional linear space or the set of all nonincreasing functions on [0,1] . Under the fixed design framework, a classical loss \ell_{s} (with s\geqslant 1 ) between two functions f,g on {\mathcal{X}} is given by (2) \ell_{s}(f,g)=\left[{\frac{1}{n}\sum_{i=1}^{n}\left|{f(x_{i,n})-g(x_{i,n)}}\right|^{s}}\right]^{1/s}, the case of s=2 being the most common. The corresponding risk of an estimator \widehat{f} of the true regression function f^{\star} is then {\mathbb{E}}\left[{\ell_{s}\left({f^{\star},\widehat{f}}\right)}\right]={\mathbb{E}}\left[\left({\frac{1}{n}\sum_{i=1}^{n}\left|{f^{\star}(x_{i,n)}-\widehat{f}(x_{i,n})}\right|^{s}}\right)^{1/s}\right]. As this quantity may involve {\mathbb{E}}\left[{|\xi_{i,n}|^{s}}\right] for 1\leqslant i\leqslant n , it is commonly assumed in the literature that these moments are finite when one uses the \ell_{s} -loss. The value of p being unknown, but not smaller than one, the moment condition is always satisfied with s=1 and we shall therefore measure the loss by \ell_{1} . Throughout the paper, this loss is simply denoted by \ell so that for two functions f,g on {\mathcal{X}} , \ell(f,g)=\frac{1}{n}\sum_{i=1}^{n}\left|{f(x_{i,n})-g(x_{i,n})}\right|. Our purpose is to bound {\mathbb{E}}\left[{\ell\left({f^{\star},\widehat{f}}\right)}\right] for a suitable estimator \widehat{f} whose construction will be described in Section 2.2 . As in the papers Baraud and Birgé ( Baraud and Birgé, 2016 ) and Baraud, Halconruy and Maillard ( Baraud et al., 2022 ) , both in density estimation, this construction is based on a model, namely {\mathcal{F}} , and a suitable family of tests. Given {\mathcal{F}} , one can actually define a nondecreasing sequence \{{\mathcal{O}}(D),1\leqslant D\leqslant n\} of subsets of {\mathcal{F}} with respect to set inclusion, for which the risk of our estimator is uniformly bounded by some quantity R_{D} on each subset {\mathcal{O}}(D) with D\in\{1,\ldots,n\} . This bound is increasing with D so that the smaller D the better the estimation for f^{\star}\in{\mathcal{O}}(D) . For a general f^{\star} and a given value of D , the risk of our estimator is the sum of R_{D} and an approximation term 3\ell(f^{\star},{\mathcal{O}}(D)) , where, for {\mathcal{G}}\subset{\mathcal{F}} , \ell(f,{\mathcal{G}}) denotes the quantity \inf_{g\in{\mathcal{G}}}\ell(f,g) , with the convention that it takes the value +\infty when {\mathcal{G}} is empty. As to R_{D} , it is proportional to \sigma_{p}({\mathbf{x}}) with {\mathbf{x}}=(x_{1,n},\ldots,x_{n,n}) provided that (3) \sigma_{p}^{p}({\mathbf{x}})=\frac{1}{n}\sum_{i=1}^{n}{\mathbb{E}}\left[{\left|{\xi_{i,n}}\right|^{p}}\right]=\frac{1}{n}\sum_{i=1}^{n}{\mathbb{E}}\left[{\left|{Y_{i,n}-{\mathbb{E}}\left[{Y_{i,n}}\right]}\right|^{p}}\right]<+\infty, while D plays a role which is similar to that of the dimension of the parameter space in Gaussian linear regression. Indeed, for p=2 we obtain that {\mathbb{E}}\left[{\ell\left({f^{\star},\widehat{f}}\right)}\right]\leqslant 3\ell(f^{\star},{\mathcal{O}}(D))+\kappa\sigma_{2}({\mathbf{x}})\sqrt{\frac{D}{n}}, for some universal positive constant \kappa . More generally, for every D\in\{1,\ldots,n\} , p\in[1,2] and every regression function f^{\star} on {\mathcal{X}} , (4) {\mathbb{E}}\left[{\ell\left({f^{\star},\widehat{f}}\right)}\right]\leqslant 3\ell\left({f^{\star},{\mathcal{O}}(D)}\right)+\kappa\left[{\sigma_{p}({\mathbf{x}})\left({\frac{D}{n}}\right)^{1-\frac{1}{p}}}\right], this bound being also true if \sigma_{p}({\mathbf{x}})=+\infty . Since it is valid for every D and p , the estimator automatically adapts to the optimal values of D and p , although neither the values \sigma_{p}({\mathbf{x}}) nor those of p for which these quantities are finite are assumed to be known. The case p=1 is the minimal requirement that ensures the existence of the regression function in ( 1 ). The bound also implies that the estimator is robust in the sense that if f^{\star} does not belong to {\mathcal{O}}(D) for a given value of D , the extra incurred loss is of the form 3\ell(f^{\star},{\mathcal{O}}(D)) . Moreover \sigma_{p}({\mathbf{x}}) only depends on the mean of the p -th absolute moments of the \xi_{i,n} , not their maximum. It may therefore be small even if one of these moments is quite large. For instance, an outlier with a large value of {\mathbb{E}}\left[{\left|{\xi_{i,n}}\right|^{p}}\right] while all the others are small will not perturbate the result too much, which can be viewed as another robustness property of the method. The problem of estimating a regression function under a fixed design has received a great deal of attention in statistics with a particular emphasis on the least squares method. Actually, we are not aware of many alternatives to the Least Squares Estimator (LSE for short), particularly in a heteroscedastic setting. A notable exception is Efromovich and Pinsker ( Efromovich and Pinsker, 1996 ) . Under the assumption that the errors are Gaussian, the authors designed an estimator which is both adaptive and sharp minimax over classes of Sobolev spaces. It is beyond the scope of this paper to provide a complete review of the use of least squares in statistics. As we shall mainly apply our procedure for the purpose of estimating a regression function with a given shape, we restrict our references to papers dealing with applications to such models. The case of the class {\mathcal{F}} of monotone functions on an interval {\mathcal{X}}\subset{\mathbb{R}} has been the most studied in the literature. Zhang ( Zhang, 2002 ) studied the performance of the LSE on {\mathcal{F}} when the errors \xi_{i,n} are independent (but not necessarily i.i.d.) and their variances are not larger than \sigma^{2} . He proved that the \ell_{s} -risk of the LSE, s\geqslant 1 , for estimating a nondecreasing regression function

Read the complete text