Fast Debiasing of the LASSO Estimator
Shuvayan Banerjee, James Saunderson, Radhendushka Srivastava, Ajit Rajwade · 2025
Read Fast Debiasing of the LASSO Estimator by Shuvayan Banerjee, James Saunderson, Radhendushka Srivastava, Ajit Rajwade (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-19825.txt.
Introduction
Shuvayan Banerjee, James Saunderson, Radhendushka Srivastava, and Ajit Rajwade speed up a fix for a biased estimator in statistics, in two thousand twenty-five. The Lasso method gives strong guarantees for sparse problems, but its estimates lean away from the truth. A known fix needs a costly matrix found by slow, repeated optimization.
The team reframe the problem to compute the needed product matrix directly. That change keeps the theory behind the debiased estimates and their confidence intervals. Under like conditions, plus one extra condition on the data matrix, they give a simple closed-form answer. Their version also has a unique best solution, which the older form lacked.
Catalog record
| CREATOR | Shuvayan Banerjee, James Saunderson, Radhendushka Srivastava, Ajit Rajwade |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-19825
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Fast Debiasing of the LASSO Estimator Shuvayan Banerjee, James Saunderson, Radhendushka Srivastava, Ajit Rajwade Abstract In high-dimensional sparse regression, the Lasso estimator offers excellent theoretical guarantees but is well-known to produce biased estimates. To address this, [ 5 ] introduced a method to “debias” the Lasso estimates for a random sub-Gaussian sensing matrix 𝑨 \boldsymbol{A} . Their approach relies on computing an “approximate inverse” 𝑴 \boldsymbol{M} of the matrix 𝑨 ⊤ 𝑨 / n \boldsymbol{A}^{\top}\boldsymbol{A}/n by solving a convex optimization problem. This matrix 𝑴 \boldsymbol{M} plays a critical role in mitigating bias and allowing for construction of confidence intervals using the debiased Lasso estimates. However the computation of 𝑴 \boldsymbol{M} is expensive in practice as it requires iterative optimization. In the presented work, we re-parameterize the optimization problem to compute a “debiasing matrix” 𝑾 := 𝑨 𝑴 ⊤ \boldsymbol{W}:=\boldsymbol{AM}^{\top} directly, rather than the approximate inverse 𝑴 \boldsymbol{M} . This reformulation retains the theoretical guarantees of the debiased Lasso estimates, as they depend on the product 𝑨 𝑴 ⊤ \boldsymbol{AM}^{\top} rather than on 𝑴 \boldsymbol{M} alone. Notably, we provide a simple, computationally efficient, closed-form solution for 𝑾 \boldsymbol{W} under similar conditions for the sensing matrix 𝑨 \boldsymbol{A} used in the original debiasing formulation, with an additional condition that the elements of every row of 𝑨 \boldsymbol{A} have uncorrelated entries. Also, the optimization problem based on 𝑾 \boldsymbol{W} guarantees a unique optimal solution, unlike the original formulation based on 𝑴 \boldsymbol{M} . We verify our main result with numerical simulations. 1 Introduction In high-dimensional sparse regression, where the number of predictors significantly exceeds the number of observations, the Lasso (Least Absolute Shrinkage and Selection Operator) is a widely used method for variable selection and estimation. By incorporating an \ell_{1} regularization term, Lasso promotes sparsity in the estimated coefficients, enabling effective performance for sparse signal vectors even if the number of predictors far exceeds the number of samples. The Lasso estimator has well-established theoretical guarantees for signal and support recovery [ 4 ] . Despite its strengths, a well-recognized limitation of Lasso is its tendency to produce biased estimates. This bias arises from the shrinkage imposed by the \ell_{1} penalty. Consequently, the bias compromises estimation accuracy and impedes statistical inference tasks such as construction of confidence intervals or hypothesis tests. These challenges are especially pronounced in high-dimensional regimes, where traditional inference tools fail due to high dimensionality. To address these limitations, several methods have been developed to “debias” the Lasso estimator, allowing for valid statistical inference even in high-dimensional settings. Notably, [ 10 ] introduced a decorrelated score-based approach, leveraging the Karush–Kuhn–Tucker (KKT) conditions of the Lasso optimization problem to construct bias-corrected estimators. Their framework relies on precise estimation of the precision matrix (inverse covariance matrix), which can be computationally challenging and sensitive to regularization choices. Similarly, [ 7 ] proposed a methodology rooted in node-wise regression, where each variable is regressed on the remaining variables to estimate the precision matrix. While effective, this method is computationally intensive. This may limit its applicability, particularly in scenarios where the design matrix lacks favorable properties like sparsity of the rows of the precision matrix. [ 5 ] introduced a simple yet powerful approach that constructs debiased Lasso estimates using an “approximate inverse” of the sample covariance matrix. Their method avoids direct precision matrix estimation and instead employs an optimization framework to compute a debiasing matrix \boldsymbol{M} that corrects for bias while ensuring asymptotic normality of the debiased estimates. A key advantage of this method is its applicability for random sub-Gaussian sensing matrices, enabling valid inference across a broad range of high-dimensional applications. In this work, we build upon the technique of [ 5 ] , addressing one of its primary computational bottlenecks: the optimization step required to compute the approximate inverse \boldsymbol{M} . By reformulating the problem to work directly with the “weight matrix” \boldsymbol{W}:=\boldsymbol{AM^{\top}} , we entirely eliminate the need to solve this optimization problem in many practical cases. Our proposed reformulation leverages the insight that the theoretical guarantees of the debiased Lasso estimator depend on the product \boldsymbol{AM}^{\top} rather than the individual debiasing matrix \boldsymbol{M} . By shifting the focus to the “weight matrix” \boldsymbol{W}:=\boldsymbol{AM}^{\top} , we simplify the optimization problem while retaining all theoretical properties of the original framework. Under certain deterministic assumptions, we provide a simple, exact, closed form optimal solution for the optimization problem to obtain \boldsymbol{W} . We show that this assumption is satisfied with high probability for the ensembles of sensing matrices considered in [ 5 ] , under the additional condition that the elements of the rows of \boldsymbol{A} are uncorrelated. In practice, sensing matrices with uncorrelated entries are commonly used in many applications [ 3 , 6 ] and are also widely used in many theoretical results in sparse regression [ 4 ] . This closed form solution eliminates the computationally intensive optimization step required to compute \boldsymbol{M} , significantly improving runtime efficiency. It is applicable in many natural situations, including sensing matrices with i.i.d. isotropic sub-Gaussian rows (such as i.i.d. Gaussian, or i.i.d. Rademacher entries). Notation: Throughout this paper, we denote matrices by bold-faced uppercase symbols, e.g., \boldsymbol{A} . If \boldsymbol{A} is an n\times p matrix then \boldsymbol{a}_{i.}\in\mathbb{R}^{p} denotes the i^{\textrm{th}} row of \boldsymbol{A} , thought of as a column vector. Similarly if \boldsymbol{A} is an n\times p matrix then \boldsymbol{a}_{.j}\in\mathbb{R}^{n} denotes the j^{\textrm{th}} column of \boldsymbol{A} , again thought of as a column vector. Vectors are denoted by bold-faced lower case symbols, e.g., \boldsymbol{w} . The i th entry of a vector \boldsymbol{w} is denoted w_{i}\in\mathbb{R} . The identity matrix of size p\times p for any positive integer p is denoted by \boldsymbol{I_{p}} , and its i^{\textrm{th}} column vector is denoted by \boldsymbol{e}_{i} . For a positive integer p , we use the shorthand [p]=\{1,2,\ldots,p\} . For a vector \boldsymbol{w}\in\mathbb{R}^{m} , we denote the \ell_{q} -norm by \|\boldsymbol{w}\|_{q}:=\left(\sum_{i=1}^{m}|w_{i}|^{q}\right)^{1/q} if 1\leq q<\infty and the \ell_{\infty} -norm by \|\boldsymbol{w}\|_{\infty}:=\max_{i\in[m]}|w_{i}| . 2 An Overview of the Debiased LASSO We consider the high-dimensional linear model \boldsymbol{y}=\boldsymbol{A\beta^{*}}+\boldsymbol{\eta}, (1) where \boldsymbol{\beta^{*}}\in\mathbb{R}^{p} is a s -sparse signal (i.e., s:=\|\boldsymbol{\beta^{*}}\|_{0} where s\ll p ), \boldsymbol{A} is a n\times p design/sensing matrix (where n\ll p ), and \boldsymbol{y}\in\mathbb{R}^{n} is the measurement vector. Also, \boldsymbol{\eta}\in\mathbb{R}^{n} is an additive noise vector that consists of independent and identically distributed elements drawn from \mathcal{N}(0,\sigma^{2}) , where \sigma^{2} is the noise variance. The Lasso estimate \boldsymbol{\hat{\beta}_{\lambda}} of the sparse signal \boldsymbol{\beta^{*}} is defined as the solution to the following optimization problem: \boldsymbol{\hat{\beta}_{\lambda}}:=\arg\min_{\boldsymbol{\beta}}\frac{1}{2n}\|\boldsymbol{y}-\boldsymbol{A\beta}\|_{2}^{2}+\lambda\|\boldsymbol{\beta}\|_{1}, (2) where \lambda>0 is a regularization parameter chosen appropriately. The Lasso estimator is known to be a consistent estimator of the sparse signal \boldsymbol{\beta^{*}} under the condition that the sensing matrix \boldsymbol{A} satisfies the Restricted Eigenvalue Condition (REC) [ 4 , Chapter 11] . 2.1 Debiasing the LASSO Estimator The Lasso estimator is well-known to produce biased estimates, i.e., E(\boldsymbol{\hat{\beta}_{\lambda}})\neq\boldsymbol{\beta^{*}} where the expectation is computed over noise instances. This bias arises from the \ell_{1} regularization term, which induces shrinkage in the estimate \boldsymbol{\hat{\beta}_{\lambda}} . Moreover, there is no known method to compute a confidence interval of \boldsymbol{\beta^{*}} directly from \boldsymbol{\hat{\beta}_{\lambda}} . To reduce this bias and also construct confidence intervals of \boldsymbol{\beta^{*}} , [ 5 ] introduced a debiased Lasso estimator \boldsymbol{\hat{\beta}_{d}} ,