Singularities in Bayesian Inference: Crucial or Overstated?
Maria De Iorio, Andreas Heinecke, Beatrice Franzolini, Rafael Cabral · 2025
Read Singularities in Bayesian Inference: Crucial or Overstated? by Maria De Iorio, Andreas Heinecke, Beatrice Franzolini, Rafael Cabral (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-06618.txt.
Introduction
Maria De Iorio, Andreas Heinecke, Beatrice Franzolini, and Rafael Cabral examine shrinkage priors in a 2025 arXiv paper. These priors push small effects toward zero while keeping large ones intact. Horseshoe-type priors are popular, yet users lack clear guidance on which prior fits a given task.
The team builds a framework that covers a large class of such priors, with and without a spike at zero. They recast the priors through ideas from reliability theory and wealth distributions. A folded form, which they call the Gambel distribution, can be written as a ratio of two known distributions. That form reveals behavior near zero and in the tails, supports new consistency results, and leads to sampling schemes. The paper ends with guidance on setting the prior's parameters.
Catalog record
| CREATOR | Maria De Iorio, Andreas Heinecke, Beatrice Franzolini, Rafael Cabral |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 81 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-06618
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Singularities in Bayesian Inference: Crucial or Overstated? Maria De Iorio, Andreas Heinecke, Beatrice Franzolini, Rafael Cabral Abstract Over the past two decades, shrinkage priors have become increasingly popular, and many proposals can be found in the literature. These priors aim to shrink small effects to zero while maintaining true large effects. Horseshoe-type priors have been particularly successful in various applications, mainly due to their computational advantages. However, there is no clear guidance on choosing the most appropriate prior for a specific setting. In this work, we propose a framework that encompasses a large class of shrinkage distributions, including priors with and without a singularity at zero. By reframing such priors in the context of reliability theory and wealth distributions, we provide insights into the prior parameters and shrinkage properties. The paper’s key contributions are based on studying the folded version of such distributions, which we refer to as the Gambel distribution. The Gambel can be rewritten as the ratio between a Generalised Gamma and a Generalised Beta of the second kind. This representation allows us to gain insights into the behaviours near the origin and along the tails, compute measures to compare their distributional properties, derive consistency results, devise MCMC schemes for posterior inference and ultimately provide guidance on the choice of the hyperparameters. Keywords : Bayesian Variable Selection, Folded Distribution, Penalised Regression, Shrinkage Priors, Tails 1 Introduction In a variety of fields, ranging from signal processing to statistics and machine learning, extensive attention has been devoted to the problem of estimating a sparse vector of regression coefficients. As a main example, consider the classical linear regression problem of estimating an unknown signal \bm{\theta}=(\theta_{1},\ldots,\theta_{p})\in\mathbb{R}^{p} from an n -dimensional observation vector \mathbf{y}\in\mathbb{R}^{n} , assumed to be generated according to \mathbf{y}=\mathbf{X}\bm{\theta}+\bm{\epsilon} where \mathbf{X}\in\mathbb{R}^{n\times p} is known and \bm{\epsilon}\sim\mathcal{N}_{n}(\mathbf{0},\sigma^{2}\mathbf{I}_{n}) follows a multivariate normal distribution with independent and homoscedastic components. In many high-dimensional problems (large p ), it can be assumed that the signal generating the observation is either sparse or compressible. Precisely, the true signal \bm{\theta} is called s -sparse if at most s\leq p of its entries are non-zero, and compressible when the ordered absolute values of its entries rapidly decrease in amplitudes and, thus, can be well approximated by a sparse vector ( Dziwoki and Kucharczyk, 2020 , see, e.g.,) . The problem of estimating sparse or compressible signals is routinely approached via regularised least squares estimators. Assuming without loss of generality \sigma^{2}=1 , a regularised least squares estimator for \bm{\theta} is obtained as \widehat{\bm{\theta}}=\argmin_{\bm{\theta}\in\mathbb{R}^{p}}\left\{\tfrac{1}{2}\|\mathbf{y}-\mathbf{X}\bm{\theta}\|^{2}+\pen(\bm{\theta})\right\} (1) under a sparsity-promoting and separable (i.e., additive) penalty function \pen(\bm{\theta})\colon\mathbb{R}^{p}\to\mathbb{R} . Famous examples of penalty functions are Lasso ( Tibshirani, 1996 ) , i.e., \pen(\bm{\theta})=\lambda\sum_{j=1}^{p}|\theta_{j}| ; and Elastic-Net ( Zou and Hastie, 2005 ) , i.e., \pen(\bm{\theta})=\lambda_{1}\sum_{j=1}^{p}|\theta_{j}|+\lambda_{2}\sum_{j=1}^{p}\theta_{j}^{2} . Regularised estimators come with many useful advantages over non-regularised estimators (like ordinary least squares), such as the possibility of performing variable selection, reframing of the inferential problem into a well-defined optimisation problem also for p>n scenarios, and exploiting of the bias-variance trade-off to improve out-of-sample prediction performance of regression models. For detailed accounts on regularised estimators for variable selection, see, for instance, Lv and Fan (2009) and Wen et al. (2018) . The connection between penalised/regularised regression and Bayesian shrinkage priors is well known. To favour a sparse solution in the Bayesian framework, a prior \pi(\bm{\theta}) placed on the regression coefficients aims at shrinking small entries in the signal towards zero while maintaining “true” large effects. The posterior distribution corresponding to the regression model and a prior \pi(\bm{\theta}) for the coefficients is \pi(\bm{\theta}\mid\mathbf{y})\propto\pi(\mathbf{y}\mid\bm{\theta})\pi(\bm{\theta}) , where \pi(\mathbf{y}\mid\bm{\theta}) is the density of a n -variate Normal distribution with expectation \mathbf{X}\bm{\theta} and covariance matrix \sigma^{2}\mathbf{I}_{n} , resulting from the likelihood. Assuming \sigma^{2}=1 , the Bayesian maximum-a-posteriori (MAP) estimator is then \displaystyle\argmax_{\bm{\theta}\in\mathbb{R}^{p}}\pi(\bm{\theta}\mid\mathbf{y})=\argmin_{\bm{\theta}\in\mathbb{R}^{p}}\left\{\tfrac{1}{2}\|\mathbf{y}-\mathbf{X}\bm{\theta}\|^{2}-\log\pi(\bm{\theta})\right\} which is equivalent to ( 1 ) for \pen(\bm{\theta})=-\log\pi(\bm{\theta}) . Moreover, \pi(\bm{\theta}) is often defined via (conditional) independent priors for the components of the signal, i.e., \theta_{j}\mid\eta\overset{iid}{\sim}\pi(\theta_{j};\eta) , for j=1,\ldots,p , for some auxiliary parameter \eta . Conditional independence (given \eta ) in the Bayesian framework mimics the separability of the corresponding penalty in ( 1 ). For instance, the popular Lasso is equivalent to the MAP estimator obtained as \pi(\bm{\theta})=\prod_{j=1}^{p}(2\lambda)^{-1}\exp\{-\lambda|\theta_{j}|\} from independent and identical Laplace priors for each coefficient. A wealth of shrinkage priors have been proposed in the Bayesian literature ( Hsiang, 1975 ; Carvalho et al., 2009 ; Polson and Scott, 2009 ; Brown and Griffin, 2010 ; Carvalho et al., 2010b ; Armagan et al., 2011 ; Armagan et al., 2013 ; Polson et al., 2014 ; Bhattacharya et al., 2015 ; Zhang et al., 2016 ; Bhadra et al., 2017 ; Bai and Ghosh, 2018b ; Ročková and George, 2018 , see, e.g.,) , often with little theoretical justification for the comparative performance of these priors and limited guidance on hyperparameter selection. The choice of a shrinkage prior in a particular sparse regression application is usually driven by qualitative and computational reasoning. Arguably, this is due to a lack of criteria or measures to compare the shrinkage behaviour of such priors. Such problems are further amplified by the fact that shrinkage priors are usually obtained as scale mixture distributions, leading to a marginal distribution of the regression coefficients corresponding to a higher transcendental function. Considerations to distinguish among different amounts of shrinkage are often posed on the behaviour of the prior around zero ( Carvalho et al., 2009 ; Bhadra et al., 2019 , see, for instance,) and, in particular, we may distinguish between priors with a point mass at zero ( Mitchell and Beauchamp, 1988 ; George, 2000 ; George and Foster, 2000 ; Yuan and Lin, 2005 , i.e., two-groups models or Spike-and-Slab priors associating positive probability to the event \theta_{j}=0 ; see, e.g.,) and continuous priors which associate null probability to the event \theta_{j}=0 ( Armagan et al., 2013 ; Carvalho et al., 2010a ; Griffin et al., 2010 ; Hans, 2011 ; Park and Casella, 2008 ; Zhang et al., 2012 , see, for instance) . Among continuous priors, we can further distinguish between priors with unbounded density at zero (to which we refer as shrinkage priors with singularity ) or continuous bounded density (to which we refer as shrinkage priors without singurality ). Nonetheless, a comprehensive and general study of the distributional properties of shrinkage prior distributions is currently lacking, though it can provide useful insights into their shrinkage behaviour. For instance, we show how continuous priors with and without a singularity at the origin often perform similarly in terms of shrinkage if other characteristics, e.g., tail shape and Gini index, are similar. In this regard, also Song and Liang (2023) - considering general shrinkage priors - obtained asymptotic rates of convergence that depend not only on the mass of the prior around the origin but also on the fatness or flatness of the tails. In particular, considering polynomial-tailed priors, they conclude that under certain conditions the posterior contraction rate and the variable selection performance of continuous shrinkage priors are close to those of priors with a point mass. To build a general framework encompassing a variety of shrinkage priors and to study their distributional properties, we draw a connection between the problem of sparse recovery using shrinkage priors