Mean-Shift PCA by Knockoff Mean
Mengda Li, Zeng Li, Jianfeng Yao · 2026
Read Mean-Shift PCA by Knockoff Mean by Mengda Li, Zeng Li, Jianfeng Yao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-25460.txt.
Introduction
Mengda Li, Zeng Li, and Jianfeng Yao remove mean-shift noise from PCA by adding knockoff noise on purpose. Standard PCA breaks when shifted samples skew leading components, and existing robust methods cannot handle this mixture contamination in high dimensions. Random-matrix theory shows the mean-shift spikes separate spectrally from stable eigenvalues. The original eigenspace stays invariant in the limit whatever the mixture weight. The team exploits this stability in a two-stage algorithm that adds knockoff mean and identifies the shifted component using only standard PCA operations.
Catalog record
| CREATOR | Mengda Li, Zeng Li, Jianfeng Yao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 71945 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.25460
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Mean-Shift PCA by Knockoff Mean Mengda Li, Zeng Li, Jianfeng Yao Abstract Removing noise is difficult, but adding noise is easy. In this work, we show how to eliminate mean-shift noisy components from PCA by deliberately introducing knockoff mean-shift perturbation. Standard PCA is highly sensitive to shifts in the sample mean: a small fraction of samples from a shifted distribution can cause large deviations in the leading principal components. In high-dimensional regimes, existing Robust PCA approaches cannot handle the mean-shift contamination structure inherent in the mixture model. Using tools from Random Matrix Theory, we prove that the mean-shift spikes are spectrally separable from the stable eigenvalues of the original covariance. Furthermore, the original eigenspace remains asymptotically invariant to the contamination, independent of the mixture weight. Exploiting this spectral stability, we propose a simple, two-stage PCA algorithm by adding knockoff mean that identifies and removes the mean-shift component using only standard PCA operations. 1 Introduction High-dimensional data analysis faces inherent challenges due to the curse of dimensionality, necessitating robust dimensionality reduction techniques. Principal Component Analysis (PCA) has served as a cornerstone of multivariate statistics for decades, enabling efficient data representation and dimension reduction. Despite its widespread adoption, PCA exhibits a critical, well-documented sensitivity to violations of its core assumption—that data is centered around a single mean. This fragility is severely exposed in the prevalent setting of mean-shift contamination , where observations are drawn from a mixture of a primary distribution and a secondary, homoscedastic component with a shifted mean. In such a regime, even a small fraction of contaminating samples can systematically bias the empirical mean and, consequently, distort the recovered principal components away from the true covariance eigenvectors. Figure 1 : Failure of Classical PCA on Gaussian data with one mean-shift cluster. Data points (blue) are sampled from a 2D Gaussian mixture with two components: an inlier component centered at the origin (blue) and an outlier component with a mean shift (orange). The red line indicates the first principal component estimated by standard PCA, which is biased towards the outlier cluster and almost orthogonal to the first principal component (blue dashed line) of the uncontaminated data. d/n=0.1 , outlier proportion \pi_{1}=15\% . The vulnerability of standard PCA to systematic mean shifts presents a significant obstacle in modern machine learning pipelines, where data is frequently aggregated from multiple sources or contains uncurated batches. Despite numerous Robust PCA (RPCA) variants proposed in the literature ( 9 ; 26 ; 28 ; 27 ; 22 ; 7 ; 8 ; 15 ; 24 ) , standard PCA remains the dominant choice in practical applications due to its widespread implementation in statistical software packages. However, existing RPCA methods fail to recover the true principal components in the mean-shift contamination mixture models in high-dimensional regimes. The cosine similarity between the recovered low-rank components and the true eigenvectors converges to zero as the aspect ratio of dimension to sample size increases. Figure 2 : Failure of Robust PCA in high dimensions with only 5\% noisy samples Largest principal component cosine alignment of PCA methods on Gaussian data with one mean-shift cluster of outlier proportion \pi_{1}=5\% . The Robust PCA method fails to recover the true principal component as the dimension increases w.r.t. non-vanishing aspect ratio d/n=1 , while our Mean-Shift PCA consistently recovers the true component. Other settings are described in Figure 6 . In the scenario of first-moment contamination, RPCA methods are fundamentally mismatched. The mean-shift noise is not sparse , contradicting the core sparsity assumption of RPCA. Moreover, it exhibits a low-rank structure which falls under the same hypothesis as the true signal. Consequently, RPCA formulations do not handle mean-shift contamination well, as illustrated in Figure 2 , and this misalignment is exacerbated in high dimensions. In this work, we use the tools of Random Matrix Theory (RMT) to show that most principal components remain stable in a high-dimensional mean-shift mixture. Furthermore, we show that it is possible to only use vanilla PCA to identify and eliminate the contamination-induced components when contamination occurs in the first moment. RMT is essential in this regime because the aspect ratio d/n does not vanish: sample eigenvalues and eigenvectors exhibit deterministic high-dimensional bias and phase-transition behavior. This allows us to distinguish the constant-order spectral displacement caused by an added knockoff mean from the \mathcal{O}(n^{-1/2}) fluctuations of stable covariance-induced spikes. Problem Formulation Our objective is to recover the principal components of the original, uncontaminated data matrix \mathbf{X}_{n}\in\mathbb{R}^{d\times n} from the observed (k+1) -component mean-shift mixture \widetilde{\mathbf{X}}_{n}\in\mathbb{R}^{d\times n} . The original data consist of n i.i.d. samples arranged column-wise: \mathbf{X}_{n}=\bigl[\mathbf{x}_{(1)},\dots,\mathbf{x}_{(n)}\bigr]_{d\times n},\qquad\mathbf{x}_{(i)}\in\mathbb{R}^{d}. The contaminated matrix \widetilde{\mathbf{X}}_{n} is obtained by adding a structured mean-shift matrix \mathbf{A}_{n} : \widetilde{\mathbf{X}}_{n}=\mathbf{X}_{n}+\mathbf{A}_{n},\qquad\mathbf{A}_{n}=\sum_{i=1}^{k}\mathbf{m}_{(i)}\boldsymbol{\gamma}_{(i)}^{\top}. (1) The i -th subpopulation is characterized by a mean vector \mathbf{m}_{(i)}\in\mathbb{R}^{d} and a mixture weight \pi_{i}\in(0,1) . Its membership is encoded by a binary indicator vector \boldsymbol{\gamma}_{(i)}\in\{0,1\}^{n} , where (\boldsymbol{\gamma}_{(i)})_{j}=1 if and only if the j -th sample originates from the i -th subpopulation. The model therefore comprises a centered inlier component (with \mathbf{m}_{(0)}=\mathbf{0} ) and k outlier components with distinct mean shifts. Remark 1.1 . This formulation is commonly referred to as a signal-plus-noise model in the literature, where \mathbf{A}_{n} typically represents a low-rank signal matrix and \mathbf{X}_{n} corresponds to noise. However, in our context, the roles are reversed: the low-rank matrix \mathbf{A}_{n} constitutes the mean-shift contamination that we aim to eliminate, or at least eliminate its influence with respect to the covariance structure of the underlying data \mathbf{X}_{n} . Contribution We provide a precise characterization of PCA’s behavior under mean-shift mixture using RMT. Based on the additive low-rank random perturbation model ( 6 ) , we show that contamination from a shifted subpopulation induces an asymptotically orthogonal distortion to the sample covariance matrix, leaving the original covariance eigenspace invariant. Our main contributions are: • Perturbation-Based Correction: We introduce a 2-fold PCA algorithm that recovers the stable covariance structure by strategically introducing another artificial mean-shift perturbation to detect non-stable principal components induced by a mean-shift mixture. This procedure is effective when the mixture weight is sufficiently prominent. • Invariant Principal Components: We show that there exists an asymptotic invariance of the underlying covariance eigenspace of general data with compactly supported spectrum, i.e., most of the principal components of the original data are stable, regardless of the mean-shift mixture weight. Paper Organization The paper is organized as follows: Section 1.1 surveys relevant literature and positions our contribution within existing work. The proposed Mean-Shift PCA algorithm is stated in Section 2 . Our main theoretical results, including mathematical assumptions for each model and the RMT-based recovery guarantees, are presented in Section 3 . Section 4 provides numerical validation compared to robust PCA methods, robust covariance estimators, and simple preprocessing baselines. Finally, Section 5 discusses broader implications, limitations, and potential extensions of our work. 1.1 Related Work Principal Component Pursuit Robust Principal Component Analysis aims to decompose the data matrix into a low-rank signal component \mathbf{L} and a sparse outlier component \mathbf{S} . The original RPCA formulation proposed by 9 solves: \displaystyle\min_{\mathbf{L},\mathbf{S}} \displaystyle\|\mathbf{L}\|_{*}+\alpha\|\mathbf{S}\|_{1} (2) \displaystyle\text{subject to} \displaystyle\widetilde{\mathbf{X}}=\mathbf{L}+\mathbf{S}, where \|\cdot\|_{*} denotes the nuclear norm, \|\cdot\|_{1} is the vectorwise \ell_{1} -norm, and \alpha>0 is a regularization parameter typically set to \alpha=1/\sqrt{\max(d,n)} . Stable PCP ( 28 )