The Lasso Distribution: Properties, Sampling Methods, and Applications in Bayesian Lasso Regression

by Mohammad Javad Davoudabadi, Jonathon Tidswell, Samuel Muller, Garth Tarr, John T. Ormerod · 2025

Read The Lasso Distribution: Properties, Sampling Methods, and Applications in Bayesian Lasso Regression by by Mohammad Javad Davoudabadi, Jonathon Tidswell, Samuel Muller, Garth Tarr, John T. Ormerod (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07394.txt.

Introduction

Mohammad Davoudabadi and four co-authors introduce a new probability distribution in a 2025 paper. It arises from the Lasso, a popular regression method that shrinks coefficients, used in Bayesian form. They derive the distribution's key properties in closed form, including its moments and generating function, and place it inside a known broad family. An efficient, stable sampling algorithm makes it practical to draw from.

The payoff is direct: inside an existing Bayesian sampler, each coefficient's conditional distribution is exactly this new one. Swapping it in makes the sampler faster and better grounded. They ship an R package so others can use the method.

Catalog record

CREATORby Mohammad Javad Davoudabadi, Jonathon Tidswell, Samuel Muller, Garth Tarr, John T. Ormerod
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-07394

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Lasso Distribution: Properties, Sampling Methods, and Applications in Bayesian Lasso Regression by Mohammad Javad Davoudabadi, Jonathon Tidswell, Samuel Muller, Garth Tarr, John T. Ormerod Abstract In this paper, we introduce a new probability distribution, the Lasso distribution. We derive several fundamental properties of the distribution, including closed-form expressions for its moments and moment-generating function. Additionally, we present an efficient and numerically stable algorithm for generating random samples from the distribution, facilitating its use in both theoretical and applied settings. We establish that the Lasso distribution belongs to the exponential family. A direct application of the Lasso distribution arises in the context of an existing Gibbs sampler, where the full conditional distribution of each regression coefficient follows this distribution. This leads to a more computationally efficient and theoretically grounded sampling scheme. To facilitate the adoption of our methodology, we provide an R package implementing the proposed methods. Our findings offer new insights into the probabilistic structure underlying the Lasso penalty and provide practical improvements in Bayesian inference for high-dimensional regression problems. 1 Introduction The Lasso (Least Absolute Shrinkage and Selection Operator) regression method, introduced by Tibshirani 1996 , has become a cornerstone in high-dimensional statistical modeling due to its ability to perform variable selection and regularization simultaneously. The Bayesian formulation of the Lasso has been extensively studied, often relying on a Laplace prior for the regression coefficients, which can be expressed as a scale mixture of a Gaussian distribution ( Park and Casella 2008 ) . However, despite its widespread use, existing formulations often face computational challenges, particularly in efficiently sampling from full conditional distributions in a Gibbs sampling framework ( Hans 2009 ) . In this paper, we propose a new probability distribution, referred to as the Lasso distribution, which arises naturally in the Bayesian formulation of Lasso regression. We fully develop this distribution and derive several of its key properties, including moments, a moment-generating function, and an efficient, numerically stable method for sampling from it. Key to these derivations is the accurate and numerically stable evaluation of the Mill’s ratio that arises in several of these functions ( Mills 1926 ) . Further, we establish that the Lasso distribution belongs to the class of exponential family distributions, making it a theoretically attractive choice for modeling and inference. It is important to note that our formulation of the Lasso distribution differs from the one implemented in the LaplacesDemon package ( Statisticat and LLC. 2021 ) , which instead corresponds to the scale mixture of normals prior used in Park and Casella 2008 . As an application of the Lasso distribution, we note that it arises naturally as part of a Gibbs sampling scheme, where each coefficient is sampled individually ( Hans 2009 , see) . By leveraging the Lasso distribution as the full conditional distribution for the regression coefficients, we achieve a more computationally efficient sampling scheme. To facilitate reproducibility and practical implementation, we provide an R package that implements our proposed methods. The remainder of the paper is structured as follows. Section 2 outlines our Bayesian hierarchical model, which motivates the Lasso distribution. In Section 3 , we introduce the Lasso distribution. In Section 3.1 , we describe how to sample efficiently in a numerically stable manner from the Lasso distribution. Section 4 describes how to use the package BayesianLasso . An application of the Lasso distribution via Gibbs sampling is given in Section 5 . We present a performance comparison on benchmark datasets in Section 6 and conclude with a brief discussion in Section 7 . 2 The Bayesian Lasso We consider the standard linear regression model {\bf y}\sim\mathcal{N}({\bf X}{\boldsymbol{\beta}},\sigma^{2}{\bf I}_{n}) for the observed dataset {\mathcal{D}}=\{{\bf y},{\bf X}\} , where {\bf y} is an n -dimensional vector of centered responses, {\bf X} is an n\times p matrix of standardized predictors, {\boldsymbol{\beta}} is a p -dimensional vector of regression coefficients, and \sigma^{2} denotes the residual variance. In practice, it is often desirable to perform variable selection alongside parameter estimation, particularly when many covariates may be irrelevant. To this end, Tibshirani 1996 proposes the Lasso, which introduces an \ell_{1} penalty to encourage sparsity in the estimated coefficients. The regularization strength is governed by a non-negative tuning parameter \lambda . \beta_{j}|\sigma^{2},\tau_{j}\sim N(0,\sigma^{2}\tau_{j}),\quad\mbox{and}\quad\tau_{j}\stackrel{{\scriptstyle\text{iid}}}{{\sim}}\text{Gamma}(1,\lambda^{2}/2),\quad 1\leq j\leq p. (1) This hierarchical representation allows for full posterior inference while promoting sparsity through the prior structure. Here, we adopt independent conjugate priors for \sigma^{2} and \lambda^{2} : \sigma^{2}\sim\text{IG}(A,B) and \lambda^{2}\sim\mbox{Gamma}(u,v) where A>0 , B>0 , u>0 , and v>0 are fixed prior hyperparameters. We propose a modification to the hierarchical prior model in ( 1 ), which enables a kernel representation of the full conditional distribution of each \beta_{j} , as given in ( 3 ), through the following alternative specification: \beta_{j}|\sigma^{2},a_{j}\sim N\left(0,\frac{\sigma^{2}}{a_{j}\lambda^{2}}\right),\quad\text{and}\quad a_{j}\stackrel{{\scriptstyle\text{iid}}}{{\sim}}\text{IG}(1,1/2),\quad 1\leq j\leq p. (2) Hans 2009 takes a different approach. Instead of using the auxiliary variable representations in ( 1 ) and ( 2 ), the standard Gibbs sampler of Hans 2009 samples from the full conditional distributions of the parameters individually, employing a weighted combination of two truncated normal distributions. It can be shown that the full conditional distribution for each \beta_{j} is proportional to \displaystyle p(\beta_{j}|{\mathcal{D}},{\boldsymbol{\beta}}_{-j},\sigma^{2},\lambda^{2})\propto\exp\left(-\tfrac{1}{2}a\beta_{j}^{2}+b\beta_{j}-c|\beta_{j}|\right) (3) where a , b , and c are constants. This kernel corresponds to the weighted combination of two truncated normal distributions utilized by Hans 2009 . Further details can be found in the Supplementary Material. Notably, this kernel does not correspond to a well-known probability distribution that can be directly sampled within a standard Gibbs sampling framework. To address this challenge, we develop a new distribution, which we term the Lasso distribution , as it arises naturally in the context of the Lasso regression model. In the next section, we formally define the Lasso distribution and derive several of its key properties, including its probability density function (PDF), cumulative distribution function (CDF), moment generating function (MGF), and moments. 3 The Lasso distribution We derive several properties of the Lasso distribution, including the probability density function (PDF), cumulative density function (CDF), moment generating function (MGF), and moments. For detailed derivations, please refer to Section 11 of the Supplementary Materials. A random variable X that maps from a probability space to the real line \mathbb{R} has a Lasso distribution with parameters a , b , and c , denoted X\sim\text{Lasso}(a,b,c) , if its PDF is \displaystyle p(X,a,b,c)=Z^{-1}\exp\left(-\tfrac{1}{2}aX^{2}+bX-c|X|\right), where a\geq 0 , b\in\mathbb{R} , c\geq 0 (with a and c not both 0), and normalizing constant Z given by Z(a,b,c)=\int_{-\infty}^{\infty}\exp\left(-\tfrac{1}{2}aX^{2}+bX-c|X|\right)dx=\sigma\left[\frac{\Phi(\mu_{1}/\sigma)}{\phi(\mu_{1}/\sigma)}+\frac{\Phi(-\mu_{2}/\sigma)}{\phi(-\mu_{2}/\sigma)}\right], with \mu_{1}=(b-c)/a , \mu_{2}=(b+c)/a , and \sigma^{2}=1/a . Here, \Phi(\cdot) and \phi(\cdot) denote the CDF and PDF of the standard normal distribution, respectively. Evaluating Z can be numerically challenging due to potential overflow, underflow, or divide-by-zero problems. To address this, our R package BayesianLasso implements a numerically stable and efficient method for computing Z , automatically managing these numerical difficulties ( Ormerod et al. 2025 ) . The Lasso distribution can be written as a mixture of two truncated normal distributions. As we shall see, for different extreme values of a , b and c , the Lasso distribution can behave like a normal distribution, a Laplace distribution, an asymmetric Laplace distribution, a positively truncated normal distribution, or a negatively truncated normal distribution. By characterizing

Read the complete text