Rényi’s α \alpha -divergence variational Bayes for spike-and-slab high-dimensional linear regression

Chadi Bsila, Yiqi Tang, Kaiwen Wang, Laurie Heyer · 2025

Read Rényi’s α \alpha -divergence variational Bayes for spike-and-slab high-dimensional linear regression by Chadi Bsila, Yiqi Tang, Kaiwen Wang, Laurie Heyer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-00627.txt.

Introduction

Sparse regression picks a few true predictors from a crowd of candidates, and priors must encode that sparsity. Chadi Bsila and co-authors develop a variational Bayes method for spike-and-slab regression in 2025, replacing the usual divergence objective with Rényi's alpha-divergence. Their setting is high-dimensional linear regression.

The method uses spike-and-slab Laplace priors within a mean-field approximation. The alpha parameter interpolates between zero-forcing and mass-covering behavior. Flexible methods matter because exact Bayesian computation is impossible at the dimensions genomics and economics now demand. Spike-and-slab priors put real probability on a coefficient being exactly zero. That honesty about exclusion is what variable selection needs. For folks who face wide data with few true clues, the gain is choice. One knob tunes how bold or how shy the fit will be as it picks its clues.

Catalog record

CREATORChadi Bsila, Yiqi Tang, Kaiwen Wang, Laurie Heyer
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-00627

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Rényi’s α \alpha -divergence variational Bayes for spike-and-slab high-dimensional linear regression Chadi Bsila, Yiqi Tang, Kaiwen Wang, Laurie Heyer Abstract Sparse high-dimensional linear regression is a central problem in statistics, where the goal is often variable selection and/or coefficient estimation. We propose a mean-field variational Bayes approximation for sparse regression with spike-and-slab Laplace priors that replaces the standard Kullback–Leibler (KL) divergence objective with the Rényi’s α \alpha divergence: a one-parameter generalization of KL divergence indexed by α ∈ ( 0 , ∞ ) ∖ { 1 } \alpha\in(0,\infty)\setminus\{1\} that allows flexibility between zero-forcing and mass-covering behavior. We derive coordinate ascent variational inference (CAVI) updates via a second-order delta method and develop a stochastic variational inference algorithm based on a Monte Carlo surrogate Rényi lower bound. In simulations, our two methods perform comparably to state-of-the-art Bayesian variable selection procedures across a range of sparsity configurations and α \alpha values for both variable selection and estimation, and our numerical results illustrate how different choices of α \alpha can be advantageous in different sparsity configurations. 2020 Mathematics Subject Classification: Primary 62F15; Secondary 65K10. Keywords: Variational Bayes; spike-and-slab prior; Rényi divergence; coordinate ascent variational inference; variable selection. 1 Introduction We consider the familiar linear regression model y=X\theta+z, (1) where y\in\mathbb{R}^{n} is the n\times 1 observed response vector, X is a given, deterministic n\times p design matrix, \theta\in\mathbb{R}^{p} is the vector of regression coefficients, and z\sim\mathsf{N}_{n}(0,I_{n}) denotes independent Gaussian noise. We are particularly interested in the high-dimensional setting, where p\gg n . Specifically, we focus on variable selection and parameter estimation. This common but challenging problem cannot be solved without additional structure; we assume that the coefficient vector \theta is sparse, where most of the entries in \theta are zero. This assumption is common in the literature, and many, both frequentists and Bayesians, have investigated the sparse high-dimensional linear regression problem. In the frequentist realm, the most widely-used method is lasso ( 21 ) and variants of lasso such as the adaptive lasso ( 22 ) . On the Bayesian front, the sparsity structure is typically incorporated into the prior distribution to solve high-dimensional problems. The two popular types of priors are spike-and-slab mixture priors ( 9 ; 6 ; 7 ) and continuous shrinkage priors like the horseshoe ( 17 ; 5 ) . One main benefit of using a Bayesian framework in such problems is a readily available posterior probability distribution for uncertainty quantification; however, a drawback of Bayesian methods, especially in the high-dimensional realm or, even more so, the ultra-high-dimensional settings, is the time and resources the computations require. Bayesian solutions to problems are often more computationally-intensive than their frequentist counterparts to the same problems. This is mainly because most Bayesian methods require Markov Chain Monte Carlo (MCMC) to get to an approximate posterior distribution. This becomes challenging in high-dimensional problems, especially as the number of variables p increases to be in the thousands. A typical Bayesian method that uses MCMC may take up to minutes to compute when p is in the hundreds, and may be too slow to use for p in the thousands. For computational efficiency, many now use alternatives to MCMC instead; variational inference (VI) ( 1 ) is one such popular alternative. Rather than doing integration like MCMC, VI converts the problem into optimization, making it a more efficient way to find a solution. In VI, we first posit a family of models and then find a member of the family that minimizes the Kullback-Leibler (KL) divergence between that member and the original posterior of interest (that is difficult to compute). The method is also referred to as variational approximation, since we are seeking out the family member that would best approximate the posterior distribution. In traditional VI, the best approximation is measured by the smallest KL-divergence between the family member and the posterior distribution; of course, there are alternative measures of distance, and those can be used instead for variational Bayes. 15 outlines a number of different divergences including Rényi’s \alpha -divergences and f -divergences, and how they may be applied to message passing, a version of variational approximation. In our work, we focus on Rényi’s \alpha -divergence ( 20 ) , a flexible measure of distance that exhibits different behavior as its hyperparameter \alpha changes. It can be considered as an extension of the traditional VI with KL-divergence: as \alpha\rightarrow 1 , Rényi’s \alpha -divergence is equivalent to KL-divergence. VI using Rényi’s \alpha -divergence instead of the traditional KL-divergence has been studied in a few different contexts, but has not been widely studied like the KL-divergence VI has been. 13 used Rényi’s \alpha -divergence by proposing a variational Rényi bound, comparable to the Evidence Lower-bound (ELBO), a bound used in the KL-divergence VI, and applied their method to Bayesian neural networks and variational auto-encoders in numerical experiments. Others have also mostly focused on complex models such as deep generative models and Bayesian neural networks ( 3 ; 14 , e.g.,) . Different \alpha values exhibit different behaviors, and a better understanding of those behaviors is needed for different problems ( 13 ) . Our aim is to provide a solution to this question for a common and important setting, the sparse high-dimensional linear regression. We adopt the Rényi’s \alpha -divergence as the variational objective and investigate how varying \alpha influences variable selection and parameter estimation. Specifically, we use a Laplace spike-and-slab model and use a mean-field family for the variational family. Compared to using KL-divergence, having the \alpha hyperparameter provides more flexibility, but at the same time, the additional flexibility also makes our objective harder to optimize and more difficult to bound. To overcome this obstacle, we used techniques including the multivariate Delta method and Jensen’s inequality. We were able to successfully derive a coordinate-ascent algorithm (CAVI), commonly used for VI. As a comparison, we also derive a second algorithm based on the variational Rényi (VR) bound proposed in 13 , and empirically investigate how varying \alpha affects parameter estimation and variable selection for our two methods. The remainder of the paper is organized as follows. Section 2 reviews VI, with emphasis on the Rényi’s \alpha -divergence formulation, and describes the hierarchical prior for the Laplace spike-and-slab model, highlighting its sparsity-inducing properties. Section 3 derives CAVI updates. Section 4 develops stochastic VI updates based on a Monte Carlo variational Rényi bound: an analogue of the evidence lower bound (ELBO) used in standard VI. Section 5 presents numerical experiments showing the empirical performance of our two methods as \alpha varies, and compares our two methods to existing Bayesian variable selection procedures. Finally, Section 6 discusses implications and directions for future work. 2 Background 2.1 Notation and preliminaries We first define some notations that will be used throughout the paper. For a probability measure P absolutely continuous with respect to a measure Q , we write \frac{dP}{dQ} for the Radon–Nikodym derivative. For a vector v\in\mathbb{R}^{d} , we use \|v\|_{2} to denote its Euclidean ( \ell_{2} ) norm. We use the shorthand D_{\alpha} for D_{\alpha}(Q\,\|\,P) to denote Rényi’s \alpha -divergence. 2.2 Variational inference for high-dimensional linear regression As is typical, we employ a hierarchical structure for the prior. Write \theta=(\theta_{S},S) , where S\subseteq[p] denotes the set of indices

Read the complete text