Robust transfer regression with corrupted labels
Sheng Pan · 2025
Read Robust transfer regression with corrupted labels by Sheng Pan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-16594.txt.
Introduction
Sheng Pan presents a method for borrowing strength from a related data set when the labels in the target data are partly corrupted. In transfer learning, a model trained on one domain is adapted to another. Here a large share of the target labels are wrong, and no one knows which ones. That setting breaks standard methods that trust every label.
Pan proves that the error of the new method splits into three parts: error from the source data, error from the shift between domains, and error from the corrupted labels. The theory shows the method beats using the target data alone. Tests include rebuilding damaged signals and a study linking a gene marker to gene activity in brain tumor patients.
Catalog record
| CREATOR | Sheng Pan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 90 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-16594
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Robust transfer regression with corrupted labels Sheng Pan Abstract In this paper, we introduce a robust transfer regression method designed to handle corrupted labels in target data, under the scenarios that the corruption affects a substantial portion of the labels and the locations of these corruptions are unknown. Theoretical analysis substantiates our approach, illustrating that the estimation error consists of three components: the first relates to the source data; the second encompasses the domain shift ; and the third captures the estimation error attributed to the corrupted vector. Our theoretical framework ensures that the proposed method surpasses estimations based solely on target data. We validate our method through numerical experiments aimed at reconstructing corrupted compressed signals. Additionally, we apply our method to analyze the association between O6-methylguanine-DNA methyltransferase (MGMT) methylation and gene expression in Glioblastoma (GBM) patients. Keywords : robust transfer regression; adversarial corruption; lasso; high-dimensional; signal recovery 1 Introduction In the field of data collection and analysis, data corruption refers to information that has been changed or damaged, leading to inaccuracies and reducing reliability. Especially in networked data compressive sensing (CS), it’s not uncommon for a small number of sensors to report incorrect measurements or, in some cases, provide data points that are completely irrelevant( Haupt et al. (2008) ). Similarly, real-world studies often face challenges from measurement errors such as misclassification and irregular assessment frequencies, which can harm the accuracy and credibility of research findings. While some inconsistencies are easy to spot and fix or remove during data cleaning, others fall within normal variability ranges, making them hard to detect ( Ackerman et al. (2024) ). These complexities require careful attention to maintain the integrity of conclusions based on the data. In this work, we focus on problems caused by corrupted labels. The corruption can be adversarial, covering scenarios like Huber’s \epsilon -contamination model, which might affect a significant portion of the observations whose locations are unknown. In such cases, the data no longer follows an independently and identically distributed (i.i.d.) pattern, and the noise may not be symmetrically distributed. Traditional methods like Lasso Tibshirani (1996) and L1-Norm Quantile Regression Li and Zhu (2008) do not perform well under these conditions. To tackle these challenges, several advanced methods have been developed for high-dimensional data. Extended Lasso techniques Nguyen and Tran (2012) ; Descloux et al. (2022) aim to recover the true signal while also identifying error locations. The Median-of-Means approach Lecu’e and Lerasle (2017) ; Lecué and Lerasle (2019) ; Geoffrey et al. (2020) enhances robustness by dividing the dataset into smaller groups, calculating the mean for each group, and then taking the median of these means. This reduces the impact of outliers and heavy-tailed distributions. The robust gradient estimation method Liu et al. (2019) ; Holland and Ikeda (2019) proposes estimating more reliable gradients during each iteration. These methods typically operate under the assumption that only target data is accessible for analysis. However, when source data is also available, transfer learning provides a potent alternative. By leveraging structural similarities across different but related domains or tasks, transfer learning has found successful application in numerous real-world scenarios. In this paper, we focus on robust high-dimensional transfer regression. Various adaptation methods have been developed for transductive transfer learning, which can be applied to scenarios involving corrupted labels. The marginal adaptation method proposed by Pan et al. (2010) assumes that the conditional distributions of the target and source data are identical. Under this assumption, the Maximum mean discrepancy (MMD) introduced by Gretton et al. Gretton et al. (2012) can be employed to measure the difference in predictor distributions between the two domains. Both joint distribution adaptation (JDA) by Long et al. Long et al. (2013) and balanced distribution adaptation (BDA) by Wang et al. (2017b) rely on the assumption that the MMD of class-conditional distributions can be approximated by replacing the true target labels with pseudo labels. However, these assumptions do not hold in our scenario, necessitating the development of alternative methods to achieve robust high-dimensional transfer regression. Bastani (2021) , Li et al. (2022) , Tian and Feng (2023) and Li et al. (2024) developed supervised transfer regression technique to improve the conventional estimation with L1 penalty. Cai et al. (2024) proosed a semi-supervised triply robust inductive transfer learning under the assumption of scarce label of target data and covariate shift. In this paper, we present a robust transfer Lasso algorithm specifically designed for signal reconstruction from potentially corrupted labels. Our contributions and findings can be summarized as follows: • We propose a source data selection algorithm aimed at identifying suitable source datasets in the presence of potentially corrupted target data labels. By comparing the reconstructed signals derived from integrating target data with each source dataset to those obtained solely from the source datasets, our method effectively identifies and excludes source datasets that exhibit significant domain shifts. • We present a transfer regression strategy designed for Lasso estimators that adjusts for both domain shifts and label corruption. Theoretical analysis reveals that the estimation error consists of three components: the Lasso estimation error on the aggregated selected source data, the impact of domain shift, and the estimation error due to label corruption. Furthermore, we establish the sign consistency property of our proposed algorithm. • To validate our approach, we conducted numerical experiments focusing on the reconstruction of corrupted compressed signals. Notably, our method demonstrates a breakdown point exceeding 50%. Additionally, we applied our method to explore the relationship between O6-methylguanine-DNA methyltransferase (MGMT) methylation and gene expressions in brain tissues of Glioblastoma (GBM) patients. Gene Ontology (GO) enrichment analysis of our results highlighted several pathways closely associated with GBM, underscoring the potential clinical relevance of our findings. The paper is outlined as follows: In Section 2.2, we first introduce the oracle version of robust transfer learning and present its theoretical results. In Section 2.3, we describe the source data selection process and detail the robust transfer Lasso algorithm. Sections 3 and 4 cover the simulations and the analysis of MGMT methylation and gene expression associations, respectively. All proofs are provided in the Appendix. 2 Methodology and main results 2.1 Notations For a matrix A=\{a_{ij}\}_{i\in[n],j\in[p]} , define the following norms: \|A\|_{\infty}=\max_{i,j}|a_{ij}|,\quad\|A\|_{L_{1}}=\max_{i}\sum_{j=1}^{p}|a_{ij}|. For sets of indices T and E , A_{T} denotes the submatrix obtained by extracting those columns indexed by T and A_{r(E)} denotes the submatrix obtained by extracting those rows indexed by E . A_{ET} denotes the submatrix obtained by extracting those rows indexed by E and those columns indexed by T . \lambda_{\min}(A) represents the smallest eigenvalue of matrix A ; \text{diag}(A) denotes the vector composed of the diagonal elements of matrix A . For a vector a=\{a_{i}\}_{i\in[n]} , its norms are defined as follows: \|a\|_{1}=\sum_{i\in[n]}|a_{i}|,\quad\|a\|_{2}=\sqrt{\sum_{i\in[n]}a_{i}^{2}},\quad\|a\|_{0}=|\{j:a_{j}\neq 0\}|. a_{E} denotes a vector where a_{E_{j}}=a_{j} if j\in E , and a_{E_{j}}=0 if j\notin E . a_{(E)} indicates extracting the elements indexed by E . HT_{\lambda}(a)=a\mathbf{1}\{|a_{j}|\geq\lambda\} denotes a threshold at \lambda . \mathrm{S}(a)=\{j:a_{j}\neq 0\} denotes the support set of vector a . For a random sequence x_{n} , x_{n}\xrightarrow{P}0 means that x_{n} converges in probability to 0 as n\to\infty . 2.2 Oracle robust transfer Lasso Throughout this article, we interpret the corruptions as follows: Y_{i}=\mbox{\boldmath$X$}_{i}^{\top}\mbox{\boldmath$\beta$}^{\ast}+e_{i}^{\ast}+\varepsilon_{i},\quad i=1,\ldots,n_{0}, (1) where \varepsilon_{i} denotes natural noise, and