A New Perspective on High Dimensional Confidence Intervals

Logan Harris, Patrick Breheny · 2025

Read A New Perspective on High Dimensional Confidence Intervals by Logan Harris, Patrick Breheny (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-03504.txt.

Introduction

Logan Harris and Patrick Breheny wrote this paper in two thousand twenty-five. It takes a fresh view of confidence intervals when many features are in play, where the classic demand of steady coverage at each parameter value breaks down.

The snag is bias in methods like the lasso, and fixes that remove the bias can hide the link to the first fit. The authors aim instead for right coverage on average. Their method, built on the Relaxed Lasso, gives near-right average coverage and keeps the first fit in sight. They set it next to debiased rivals and revisit the bootstrap, which was known to fail for the lasso. It fails under the new coverage sense too, they find.

Catalog record

CREATORLogan Harris, Patrick Breheny
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-03504

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A New Perspective on High Dimensional Confidence Intervals Logan Harris, Patrick Breheny Abstract Classically, confidence intervals are required to have consistent coverage across all values of the parameter. However, this will inevitably break down if the underlying estimation procedure is biased. For this reason, many efforts have focused on debiased versions of the lasso for interval construction. In the process of debiasing, however, the connection to the original estimates are often obscured. In this work, we offer a different perspective focused on average coverage in contrast to individual coverage. This perspective results in confidence intervals that better reflect the original assumptions, as opposed to debiased intervals, which often do not even contain the original lasso estimates. To this end we propose a method based on the Relaxed Lasso that gives approximately correct average coverage and compare this to debiased methods which attempt to produce correct individual coverage. With this new definition of coverage we also briefly revisit the bootstrap, which Chatterjee and Lahiri (2010) showed was inconsistent for lasso, but find that it fails even under this alternative coverage definition. 1 Introduction The objective function for lasso-penalized linear regression ( Tibshirani 1996 ) is Q(\boldsymbol{\beta}|\mathbf{X},\mathbf{y},\lambda)=\frac{1}{2n}\lVert\mathbf{y}-\mathbf{X}\boldsymbol{\beta}\rVert_{2}^{2}+\lambda\lVert\boldsymbol{\beta}\rVert_{1}, where \mathbf{y} is a length n vector of independent outcomes, \mathbf{X} is an n\times p matrix of features, \boldsymbol{\beta} is a length p vector of regression coefficients, and \lambda is a regularization parameter controlling the amount of penalization. Note that the objective function involves the addition of the L_{1} penalty, \lambda\lVert\boldsymbol{\beta}\rVert_{1}=\lambda\sum_{j=1}^{p}|\beta_{j}| , to the squared error loss. This typically results in sparse estimates for some of the regression coefficients (i.e., \widehat{\beta}_{j}=0 ) depending on the choice of the regularization parameter \lambda . Its ability to carry out both variable selection and estimation is particularly attractive, especially in scenarios where both predictive accuracy and interpretability are important. The lasso performs particularly well in cases where the number of features is large and the underlying model is sparse ( Hastie et al. 2009 ) , but has become popular in a wide variety of settings. Nevertheless, inference for the lasso has proven challenging. By introducing both sparsity and shrinkage, the L_{1} penalty greatly complicates the sampling distribution of the estimators. This complexity has given rise to a wide variety of inferential approaches. The majority of these approaches have focused on controlling the false discovery rate (FDR) of the selected features. Examples include the Covariance test ( Lockhart et al. 2014 ) , the Knockoff Filter ( Barber and Candès 2015 ; Candès et al. 2018 ) , the marginal FDR ( Breheny 2019 ) , and the Gaussian mirror ( Xing et al. 2023 ) . There have also been various proposals for constructing confidence intervals (CIs), although the shrinkage/bias introduced by the L_{1} penalty poses a number of challenges here. Several methods ( Zhang and Zhang 2014 ; Javanmard and Montanari 2014 ) focus on “debiasing” the original point estimates from a lasso fit to facilitate more traditional forms of inference. An alternative approach, which accounts for the uncertainty in model selection by conditioning on the selected model, is known as Selective Inference ( Lee et al. 2016 ) , although it is worth noting that this approach only produces intervals for variables that were selected. In this manuscript, we offer a different perspective that allows for biased intervals and focuses on correct average coverage instead of correct individual coverage. This perspective results in confidence intervals that better reflect the original assumptions that motivated the use of the lasso for estimation — as opposed to debiased intervals, which often do not even contain the original lasso estimates. The goal of this paper is not to argue that either definition is inherently superior, but rather to explore the differences between them; we hope the reader finds the debate illuminating. Section 2 examines the underlying concept of average coverage in more detail and shows that the bootstrap does not even produce average coverage. Section 3 introduces a method based on the Relaxed Lasso which does have approximately correct average coverage. Then Section 4 examines the performance of the proposed method across a number of simulations and includes a comparison to Selective Inference and the de-sparsified lasso. Lastly, in Section 5, we show the application of the proposed method to two data sets, one for acute respiratory illness and the other for gene expression data in mammalian eyes. For the sake of simplicity, we focus on lasso-penalized linear regression, but most of the discussion is relevant to all penalized regression models. 2 Average coverage When using penalized regression, we are introducing bias into the estimators by design. This has direct implications for confidence intervals constructed around these biased estimates. All methods we are aware of propose debiasing as a way to counteract (either directly or indirectly) the bias introduced in attempts to obtain traditional frequentist coverage properties. In Section 2.1 , we instead propose an alternate perspective that focuses on targeting average coverage inspired by the connection between the penalties in penalized regression and Bayesian priors. Given the connection between the bootstrap and Bayesian posteriors, one might suppose that bootstrap confidence intervals also meet this alternate definition for coverage. However, we show in Section 2.2 that bootstrap intervals fall increasingly short of average coverage as the dimension grows. 2.1 Individual vs average coverage Classical frequentist inference is concerned with achieving proper coverage for each parameter individually. In penalized regression, bias is explicitly being introduced in the estimation procedure which poses a problem when targeting nominal interval coverage for individual parameters. Here we propose shifting focus to average coverage across all p confidence intervals. In penalized regression, these two definitions of coverage can be quite different. Letting \mathcal{A}(\mathbf{y}) denote a process that produces an interval based on data \mathbf{y} , the coverage probability for the process is defined as \textrm{Cover}(\theta)=\mathbb{P}\{\theta\in\mathcal{A}(\mathbf{y})\} . Classical frequentist inference requires valid intervals to satisfy \textrm{Cover}(\theta)=1-\alpha for all values of \theta (or potentially \geq 1-\alpha ). This is, however, incompatible with Bayesian inference. Bayesian credible intervals cannot, in general, have the same coverage for each \theta . What they satisfy instead is maintaining the expected coverage with respect to the prior distribution of \theta : \int\textrm{Cover}(\theta)p(\theta)\,d\theta=1-\alpha (this is not the definition of credibility, but it is a consequence, as we show later in this section). Unless the prior is uniform, the coverage of a Bayesian credible interval will be greater than 1-\alpha for some \theta and less than 1-\alpha for other values of \theta . For example, consider the credible intervals for \theta in a \textrm{N}(\theta,\sigma^{2}) model with prior \theta\sim\textrm{N}(0,\tau^{2}) with \sigma=\tau=1 . The left side of Figure 1 illustrates the coverage probability for the 80% Bayesian credible interval over a range of \theta values. Where the prior density for \theta is highest, the coverage is above 80%, whereas regions where the prior density is low have coverage below 80%. The expected coverage, however, is exactly 80% when integrated with respect to the prior. This is fundamentally true of any Bayesian model with an non-uniform prior: \textrm{Cover}(\theta)=1-\alpha for all values of \theta will never be satisfied. The right side, which illustrates that a similar phenomenon happens for the method we propose, will be discussed in Section

Read the complete text