Stability Selection via Variable Decorrelation

Mahdi Nouraie, Connor Smith, Samuel Muller · 2025

Read Stability Selection via Variable Decorrelation by Mahdi Nouraie, Connor Smith, Samuel Muller (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20864.txt.

Introduction

Mahdi Nouraie, Connor Smith, and Samuel Muller steady a popular variable-picking tool by untangling its inputs first, in a 2025 paper. The Lasso selects a few key variables from many, yet correlated inputs make its picks jumpy. Past fixes changed the tool's scoring; this one cleans the data before the tool runs.

Decorrelating the variables first made selections steadier, whatever the direction of the links among them. The authors show a key consistency condition for the Lasso then holds under two mild assumptions. The gain shows in low-dimensional data too, not just in the many-variable case. Tests across several selection methods confirm the lift. An R package, DVS, ships the method for working analysts.

Catalog record

CREATORMahdi Nouraie, Connor Smith, Samuel Muller
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-20864

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Stability Selection via Variable Decorrelation Mahdi Nouraie, Connor Smith, Samuel Muller Abstract The Lasso is a prominent algorithm for variable selection. However, its instability in the presence of correlated variables in the high-dimensional setting is well-documented. Although previous research has attempted to address this issue by modifying the Lasso loss function, this paper introduces an approach that simplifies the data processed by Lasso. We propose that decorrelating variables before applying the Lasso improves the stability of variable selection regardless of the direction of correlation among predictors. Furthermore, we highlight that the irrepresentable condition, which ensures consistency for the Lasso, is satisfied after variable decorrelation under two assumptions. In addition, by noting that the instability of the Lasso is not limited to high-dimensional settings, we demonstrate the effectiveness of the proposed approach for low-dimensional data. Finally, we present empirical results that indicate the efficacy of the proposed method across different variable selection techniques, highlighting its potential for broader application. The DVS R package is developed to facilitate the implementation of the methodology proposed in this paper. Keywords: Bioinformatics, Consistency, Feature Selection, Gram-Schmidt Process, Lasso, QR Factorisation, Stability, Variable Selection 1 Introduction Linear regression is formally defined as \boldsymbol{Y}=X\boldsymbol{\beta}+\boldsymbol{\varepsilon} , where \boldsymbol{Y} denotes the n- vector of the response variable, X\in\operatorname{mat}(n,p) denotes the fixed design matrix, \boldsymbol{\beta} denotes the p- vector of regression coefficients, and \boldsymbol{\varepsilon} denotes the n- vector of random errors. The intercept term is omitted here because, without loss of generality, we assume that the response vector is centred, that is, \sum_{i=1}^{n}\boldsymbol{Y}_{i}=0 . The least absolute shrinkage and selection operator ( Tibshirani, 1996 , Lasso;) is a prominent variable selection method that achieves sparsity by adding an \ell_{1}- norm to the least-squares loss function. The regression coefficients of the Lasso are estimated through \boldsymbol{\hat{\beta}}(\lambda)=\argminA_{\boldsymbol{\beta}\in\mathbb{R}^{p}}\left(\|\boldsymbol{Y}-X\boldsymbol{\beta}\|_{2}^{2}+\lambda\|\boldsymbol{\beta}\|_{1}\right), (1) where \lambda\in\mathbb{R}^{+} denotes the Lasso regularisation parameter. Variable selection, following the terminology of Meinshausen and Bühlmann (2010) , refers to categorising variables into two distinct groups: the signal group S:=\{k\neq 0\mid\beta_{k}\neq 0\} and the noise group N:=\{k\neq 0\mid\beta_{k}=0\} , where S\cap N=\emptyset . The primary objective of variable selection is to accurately identify the signal group S . When using the Lasso, the set of non-zero coefficients \hat{S}(\lambda):=\{k\neq 0\mid\hat{\beta}_{k}(\lambda)\neq 0\} , can be derived from solving Equation ( 1 ) through convex optimisation. The stability of a variable selection method is related to its ability to consistently identify the same variables across different training sets sampled from the same underlying distribution ( Kalousis et al., 2007 ) . This property serves as an indicator of the reproducibility and generalisability of the selection results and has received significant attention in recent years (see, e.g., Meinshausen and Bühlmann (2010) , Nogueira et al. (2018) , and Soloff et al. (2024) ). In the high-dimensional setting, where p\gg n , the presence of correlated variables within X can directly affect the stability of the variable selection carried out by Lasso ( Wang et al., 2018 ) . To be more specific, when two variables are highly correlated and their effect sizes are unconstrained, they exhibit a nearly identical effect on the response variable ( Wang et al., 2018 ) . In such cases, Lasso tends to select at random only one of these variables ( Xu et al., 2012 ; Wang et al., 2018 ) . In general, if an irrelevant variable is highly correlated with the relevant variables, Lasso may not be able to distinguish it from the relevant variables with any amount of data and any amount of regularisation ( Leng et al., 2006 ; Zhao and Yu, 2006 ) . In this paper, we propose to decorrelate predictor variables before performing variable selection by Lasso. The proposed variable decorrelation process transforms the predictors into an orthonormal basis aiming to reduce noise from inter-variable dependencies and improve interpretability. We show that this transformation improves the stability of variable selection for the Lasso. Although the orthonormal design matrix has long been a subject of interest (see, e.g., Tibshirani (1996) and Yuan and Lin (2006) ), to the best of our knowledge, this is the first paper to propose addressing the instability of the Lasso through an orthonormalisation process. We further show that the irrepresentable condition ( Zhao and Yu, 2006 ; Meinshausen and Bühlmann, 2006 ) , which guarantees consistent variable selection by the Lasso, holds after variable decorrelation under two assumptions. In addition, we highlight that the instability of the Lasso is not confined to high-dimensional settings, and variable decorrelation can serve as an effective strategy for achieving stable variable selection in low-dimensional data as well. Several previous studies have attempted to improve the stability of the Lasso by modifying its loss function to better handle correlated variables (see, e.g., Chen et al. (2013) , Takada et al. (2018) , and Wang et al. (2018) ). A complementary approach, stability selection, enhances the stability of variable selection by applying a selection method like the Lasso to multiple random half-samples of the original dataset ( Meinshausen and Bühlmann, 2010 ; Shah and Samworth, 2013 ) . Stability selection identifies relevant variables by evaluating their selection frequencies across various randomly drawn sub-samples of observations. In this paper, we use the latter approach to identify stable variables following the decorrelation of the variables. The Gram–Schmidt process ( Björck, 1994 ) is a fundamental technique for generating an orthonormal set of vectors that spans the same subspace as the original set. The Gram–Schmidt process is traditionally used to compute the thin QR factorisation ( Golub and Van Loan, 2013 ) , and in recent years, has been applied to variable ranking, screening, and selection problems (see, e.g., Stoppiglia et al. (2003) , Liu et al. (2018) , Wang et al. (2020) , and Skogholt et al. (2023) ). The Gram–Schmidt process is presented in Algorithm 2 in the appendix. The resulting matrix Q in Algorithm 2 has the same dimensions as the input matrix X , with its columns mutually orthogonal and normalised to have an \ell_{2} -norm of one. In this paper, we show that using Q , obtained by applying Algorithm 2 to the design matrix X , as a surrogate for X in variable selection results in greater stability; however, we first comment on its limitations and then focus on refinements Algorithm 2 sequentially orthogonalises columns of X from index 1 to p , giving earlier variables greater influence over the resulting basis when correlations are non-zero. This creates a competitive dynamic, where later correlated variables must offer additional information to be represented. Here, we present a toy example to illustrate the index sensitivity of Algorithm 2 . The dataset contains n=100 samples and p=5 variables, with X_{2}=0.8X_{1} introducing correlation. The response \boldsymbol{Y} is generated from \boldsymbol{\beta}=(1,0,0,0,0)^{\top} with standard normal noise. We apply the Lasso to the orthonormal matrix Q in the two cases: once obtained from the original design matrix, and once from the reordered design matrix after swapping X_{1} and X_{2} . In both cases, only the first coefficient is non-zero, highlighting how the order of the variables affects the selection results. The corresponding implementation code is available on GitHub 1 1 1 https://github.com/MahdiNouraie/Variable-Decorrelation/blob/main/Code/Toy_example.R . Section 4 presents further simulations showing that applying Lasso

Read the complete text