Heavy-tailed and Horseshoe priors for regression and sparse Besov rates

Sergios Agapiou, Ismaël Castillo, Paul Egels · 2025

Read Heavy-tailed and Horseshoe priors for regression and sparse Besov rates by Sergios Agapiou, Ismaël Castillo, Paul Egels (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15543.txt.

Introduction

Nonparametric statistics must handle functions of wildly varying smoothness, ideally with one method that adapts on its own. The three authors study a class of heavy-tailed priors on basis coefficients, called oversmoothed heavy-tailed priors, and ask whether their posteriors contract at the right rates. They also examine horseshoe priors in this nonparametric frame for the first time and show they fit the same framework.

For Sobolev-smooth signals under squared error they derive contraction rates, plus a lower bound showing the prior's scaling form is necessary for full adaptation. For the wider Besov setting with general losses they get minimax rates that adapt to smoothness, including the sparse zone. An implementation and a simulation study round out the paper, which offers flexible priors with proven adaptive performance.

Catalog record

CREATORSergios Agapiou, Ismaël Castillo, Paul Egels
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE103 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-15543

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Heavy-tailed and Horseshoe priors for regression and sparse Besov rates Sergios Agapiou, Ismaël Castillo, Paul Egels Abstract The large variety of functions encountered in nonparametric statistics, calls for methods that are flexible enough to achieve optimal or near-optimal performance over a wide variety of functional classes, such as Besov balls, as well as over a large array of loss functions. In this work, we show that a class of heavy-tailed prior distributions on basis function coefficients introduced in [ 1 ] and called Oversmoothed heavy-Tailed (OT) priors, leads to Bayesian posterior distributions that satisfy these requirements; the case of horseshoe distributions is also investigated, for the first time in the context of nonparametrics, and we show that they fit into this framework. Posterior contraction rates are derived in two settings. The case of Sobolev–smooth signals and L 2 L_{2} –risk is considered first, along with a lower bound result showing that the imposed form of the scalings on prior coefficients by the OT prior is necessary to get full adaptation to smoothness. Second, the broader case of Besov-smooth signals with L p ′ L_{p^{\prime}} –risks, p ′ ≥ 1 p^{\prime}\geq 1 , is considered, and minimax posterior contraction rates, adaptive to the underlying smoothness, and including rates in the so-called sparse zone, are derived. We provide an implementation of the proposed method and illustrate our results through a simulation study. keywords Bayesian nonparametrics, Besov spaces, frequentist analysis of posterior distributions, heavy-tailed prior distributions, horseshoe prior, sparse Besov rates. ?? Department of Mathematics and Statistics, University of Cyprus, Nicosia, Cyprus. ?? ?? Sorbonne Université, LPSM; 4, place Jussieu, 75005 Paris, France. ?? ?? Sorbonne Université, LPSM; 4, place Jussieu, 75005 Paris, France. ?? 1 Introduction The ability of estimators to be flexible, in the sense that their properties remain excellent in a variety of contexts, is a particularly sought-after characteristic. In nonparametric statistics, where the quantity of interest is typically an unknown function (which can represent a signal, an image etc. in practical applications), wavelet thresholding methods are a fundamental class that arguably satisfies the previous flexibility desideratum. Simple non-linear thresholding rules such as hard or soft thresholding are very broadly used in many areas of statistics and signal processing; from the mathematical perspective, they achieve asymptotic near-minimaxity over a very broad variety of functional classes and loss functions. While they are simple to implement and overall display very good empirical behaviour, wavelet thresholding methods still face a number of challenges numerically, in particular for low or moderate sample sizes, as one needs to find good values for thresholding constants, which may require some tuning. Bayesian nonparametric methods, on the other hand, have experienced rapid growth since the early 2000’s, when the development of sampling algorithms to simulate from posterior distributions or approximations thereof (MCMC, ABC, Variational Bayes to name just a few) has been paralleled by a progressive mathematical understanding of the properties needed for the prior distribution to achieve optimal or near-optimal convergence properties for the corresponding posterior distributions. We refer to [ 30 , 12 ] for recent overviews on the field. Among the most popular classes of priors on functions in statistics and machine learning are Gaussian processes [ 44 ] (henceforth GPs). For a number of function classes, in particular ones for which the signal’s smoothness is ‘homogeneous’ over the considered input space, one can show that GPs achieve optimal posterior contraction rates, provided their parameters are well chosen [ 50 , 10 ] , and adaptation to smoothness can be achieved by, for example, drawing a scaling parameter at random [ 51 ] . However, for more ‘inhomogeneous’ signals, Gaussian processes can be shown to lead to suboptimal rates [ 5 ] for certain losses at least (e.g. quadratic) and, more generally, properties of posterior distributions for functions with ‘heterogeneous’ regularity across the input space are currently less understood; we review below a few recent results in this direction. Regarding optimality of rates, historically the first systematic study of non-matched situations where the parameter of the loss function does not match the norm index defining the functional class, is due to Nemirovskii [ 42 ] and to Nemirovskii, Polyak and Tsybakov [ 40 , 41 ] , where in particular it was noticed that linear estimators can be suboptimal. Donoho and Johnstone [ 23 ] studied this type of questions in sequence space, while Donoho, Johnstone, Kerkyacharian and Picard [ 26 ] investigated adaptive minimax rates in density estimation for Besov classes. For such classes, it turns out that the information-theoretic boundaries can be described by three ‘zones’, depending on the loss function and the characteristics of the Besov space – we refer to [ 34 ] for an in-depth discussion –: a regular zone, where best linear estimators achieve the minimax rate and the rate is the classical n^{-\beta/(2\beta+1)} rate in terms of regularity \beta and number of observations n , an intermediate zone, where the ‘global’ minimax rate is still the same but linear estimators are suboptimal, and a sparse zone, where the rate deteriorates and can be expressed as a combination of the regularity and loss function parameters. This illustrates the delicate interplay between measures of loss and regularity, which in particular leads to significant, polynomial–in– n differences in convergence rates. Aside of wavelet methods, an adaptive local bandwidth selector based on Lepski’s method was developed for kernel–based estimators in [ 38 ] and latter for anisotropic functions in [ 37 ] . While working on this paper, we learned of the very recent work [ 36 ] , where the authors derive oracle inequalities and convergence rates in non-matched situations for penalisation methods. Coming back to Bayesian methods, as noted above, Gaussian Processes can be suboptimal when estimating functions with inhomogeneous smoothness (a fact formally proved in [ 5 ] in the white noise model, with mean square loss). Recently, it has been noted that taking heavier tails than Gaussian in defining priors can help in obtaining more diverse behaviour. In [ 3 ] , p -exponential priors on coefficients of a basis expansion are considered; while the posterior convergence rates in mean square loss improve in a range of aspects compared to the Gaussian case, they are still not adaptive to smoothness. A main advantage is that a wider range of Besov spaces is accessible with optimal contraction rates with these priors, for well chosen parameters. For example, unlike GP priors, priors with exponential tails can be tuned to attain the minimax rate in mean square loss, over both classes of homogeneously smooth functions as well as classes permitting spatial inhomogeneities, see [ 3 ] and the recent preprint [ 21 ] . To make these priors adaptive, it is still necessary, however, and similar to GPs as discussed above, to draw at least one extra parameter at random; such adaptive counterparts have been obtained in [ 4 ] for the white noise model, while [ 33 ] considered density estimation. In [ 1 ] , the following (surprising at first) result is obtained: putting heavy-tailed priors (in the sense that they have polynomially decreasing tails) on coefficients can lead to automatic adaptation to smoothness. By taking well-chosen (but, crucially,

Read the complete text