Linear Shrinkage Convexification of Penalized Linear Regression With Missing Data
Seongoh Park, Seongjin Lee, Nguyen Thi Hai Yen, Nguyen Phuoc Long, Johan Lim · 2024
Read Linear Shrinkage Convexification of Penalized Linear Regression With Missing Data by Seongoh Park, Seongjin Lee, Nguyen Thi Hai Yen, Nguyen Phuoc Long, Johan Lim (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-19963.txt.
Introduction
Seongoh Park and co-authors present a 2024 arXiv paper on penalized regression when data have missing values. Missing entries bias results and can make the covariance matrix lose a key property, which turns the fitting problem non-convex and hard. The team proposes a linear shrinkage fix that restores the property with a closed-form formula.
With the fixed matrix, the penalized problem becomes convex again, so sparse solutions can be found reliably. The authors prove selection consistency and give a convergence rate for the estimator. A cancer drug dataset, with drug responses and protein levels full of gaps, serves as the real-world test bed.
Catalog record
| CREATOR | Seongoh Park, Seongjin Lee, Nguyen Thi Hai Yen, Nguyen Phuoc Long, Johan Lim |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 142 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-19963
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Linear Shrinkage Convexification of Penalized Linear Regression With Missing Data Seongoh Park, Seongjin Lee, Nguyen Thi Hai Yen, Nguyen Phuoc Long, Johan Lim Abstract One of the common challenges faced by researchers in recent data analysis is missing values. In the context of penalized linear regression, which has been extensively explored over several decades, missing values introduce bias and yield a non-positive definite covariance matrix of the covariates, rendering the least square loss function non-convex. In this paper, we propose a novel procedure called the linear shrinkage positive definite (LPD) modification to address this issue. The LPD modification aims to modify the covariance matrix of the covariates in order to ensure consistency and positive definiteness. Employing the new covariance estimator, we are able to transform the penalized regression problem into a convex one, thereby facilitating the identification of sparse solutions. Notably, the LPD modification is computationally efficient and can be expressed analytically. In the presence of missing values, we establish the selection consistency and prove the convergence rate of the ℓ 1 \ell_{1} -penalized regression estimator with LPD, showing an ℓ 2 \ell_{2} -error convergence rate of square-root of log p \log p over n n by a factor of ( s 0 ) 3 / 2 (s_{0})^{3/2} ( s 0 s_{0} : the number of non-zero coefficients). To further evaluate the effectiveness of our approach, we analyze real data from the Genomics of Drug Sensitivity in Cancer (GDSC) dataset. This dataset provides incomplete measurements of drug sensitivities of cell lines and their protein expressions. We conduct a series of penalized linear regression models with each sensitivity value serving as a response variable and protein expressions as explanatory variables. Keyword: General missing dependency, lasso, positive definiteness. 1 Introduction Regularized or penalized linear regression has been largely explored for decades, motivated from a variety of modern applied fields ( Lee et al.,, 2003 ; Ghosh and Chinnaiyan,, 2005 ; Daye et al.,, 2012 ; Han and Tsay,, 2020 ) where the sample size is much smaller than the number of variables to be analyzed. Among different regularizations in linear regression such as ridge ( Hoerl and Kennard,, 1970 ) , lasso ( Tibshirani,, 1996 ; Zou,, 2006 ) , Dantzig selector ( Candes and Tao,, 2007 ) , elastic net ( Zou and Hastie,, 2005 ) , SCAD ( Fan and Li,, 2001 ) , the lasso regression has gained its popularity because its statistical properties ( Zhao and Yu,, 2006 ; Lee et al.,, 2015 ; Zou,, 2006 ; van de Geer and Bühlmann,, 2009 ; Fu and Knight,, 2000 ) and computational aspects ( Efron et al.,, 2004 ; Osborne et al.,, 2000 ; Friedman et al.,, 2007 ) are well established. Though the technology for data collection has exceptionally advanced in recent years, one common issue that researchers face in data analyses is missing values. Our motivating example is drug response data ( https://www.cancerrxgene.org/ , Release v8.4, July 2022) and the pan-cancer proteomic profile of 8,498 proteins from 949 human cancer cell lines (28 tissue types, more than 40 cancer types) ( Gonçalves et al.,, 2022 ) . This study was to measure the sensitivities (IC50/AUC) of cells to different drugs and aimed to find the association between drug responses and protein levels. Missing data are widely seen in mass spectrometry (MS)-based proteomics ( Webb-Robertson et al.,, 2015 ) or metabolomics ( Wei et al.,, 2018 ) . Causes for missing values could be biological or technical (e.g., stochastic fluctuations during data acquisition) and of random or not at-random ( Karpievitch et al.,, 2012 ) . Unless treated appropriately, incomplete data often lead to biased results and hamper study reproducibility ( Dabke et al.,, 2021 ) . For instance, for the lasso regression Sørensen et al., (2015) showed that a naive approach using the incomplete data without correction does not satisfy estimation consistency (see Proposition 1 therein). Many researchers have come up with different solutions to address this issue under linear regression models. First, the expectation-maximization (EM) algorithm is developed by Städler and Bühlmann, (2010) where they aimed to find the sparse inverse covariance matrix and used it in the sparse linear regression. However, the EM algorithm is model-specific and known to converge slowly. Alternatively, variable selection can be combined with multiple imputation that is commonly used in practice. For example, one can perform majority votes based on selection results from multiply imputed datasets ( Heymans et al.,, 2007 ; Wood et al.,, 2008 ; Lachenbruch,, 2011 ; Long and Johnson,, 2015 ) . To avoid the ad-hoc rules for combining different sets of selected variables, Wan et al., (2015) and Li et al., (2023) considered stacking imputed datasets and selected the same variables across all datasets, which is termed as a stacked method in Du et al., (2022) . In Chen and Caramanis, (2013) , they proposed the group-wise selection approach to consistently choose variables across imputed datasets, which is named a grouped method in Du et al., (2022) . These methods exhibited satisfactory performance in simulated and real data analyses; however, theoretical evidences are elusive. To fill this gap, researchers have paid attention on de-biasing approaches. These are based on the observation that a loss function, for example, mean squared error, is biased if data are not completely observed. Thus, related work adjusted it by adding or multiplying de-biasing constants to the covariance part or Gram matrix (e.g. see ( 5 )) and solved the corrected optimization problem with different penalization methods; for example, Liang and Li, (2009) used the SCAD penalty, and Loh and Wainwright, (2012) adopted the lasso penalty. Following Loh and Wainwright, (2012) where estimation consistency is proved, Sørensen et al., (2015) additionally showed sign consistency under the irrepresentable condition adapted to their contexts. This line of work, however, has a computational issue that the modified loss function is no longer convex. It was sidestepped in Rosenbaum and Tsybakov, (2010) and Wang et al., (2019) by using Dantzig selector that is always defined as a linear programming regardless of the modification. A more fundamental remedy for the non-convexity is to modify the corrected covariance factor \widehat{\boldsymbol{\Sigma}} to be positive definite (PD). To this end, Datta and Zou, (2017) found the closest PD matrix to \widehat{\boldsymbol{\Sigma}} using the element-wise maximum norm: \tilde{\boldsymbol{\Sigma}}^{CoCo}=\argmin_{\boldsymbol{\Sigma}\succ 0}\|\widehat{\boldsymbol{\Sigma}}-\boldsymbol{\Sigma}\|_{\max}. (1) Using it, they solved the \ell_{1} -penalized regression problem, which is named CoColasso, and proved estimation and selection consistency under regular conditions including the irrepresentable condition. This area of research has been recently studied further. Though handling the measurement error not missing data, Zheng et al., (2018) and Zhang et al., (2022) proposed to use different penalty functions, a combination of \ell_{1} - and concave penalty, and \ell_{0} -penalty, respectively, to ensure better theoretical properties of estimators (i.e. faster oracle inequality). Escribe et al., (2021) considered partially corrupted data where some of explanatory variables are corrupted under some measurement error model and the others are not. Thus, they only solved ( 1 ) for a smaller dimension at which the measurement errors are found. On the other hand, in solving ( 1 ), Takada et al., (2019) suggested to downweight components at which samples are highly missing. To do