Principal Component Analysis When n < < p : Challenges and Solutions ———- PRE - PRINT VERSION ———-

2025

Read Principal Component Analysis When n < < p : Challenges and Solutions ———- PRE - PRINT VERSION ———- (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-17560.txt.

Introduction

Nuwan Weeraratne, Lyn Hunt, and Jason Kurz tackle a classic data method that falters when variables outnumber samples, in a 2025 paper. The method, principal component analysis, shrinks data while keeping its main patterns. It rests on a spread matrix estimated from the data. With few samples, the standard estimate is poor and the method suffers.

The team proposes a new spread estimate based on pairwise gaps between samples. Four tuned versions of that estimate are offered. Tests compared them with the standard estimate and its best-known rival. The new versions did better at recovering true spread and main patterns. They also cut two kinds of error in the results. Real data examples round out the study. The work targets fields where wide data sets are common.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-17560

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Principal Component Analysis When n < < p : Challenges and Solutions ———- PRE - PRINT VERSION ———- Abstract Abstract Principal Component Analysis is a key technique for reducing the complexity of high-dimensional data while preserving its fundamental data structure, ensuring models remain stable and interpretable. This is achieved by transforming the original variables into a new set of uncorrelated variables (principal components) based on the covariance structure of the original variables. However, since the traditional maximum likelihood covariance estimator does not accurately converge to the true covariance matrix, the standard principal component analysis performs poorly as a dimensionality reduction technique in high-dimensional scenarios n<p . In this study, inspired by a fundamental issue associated with mean estimation when n<p , we proposed a novel estimation called pairwise differences covariance estimation with four regularized versions of it to address the issues with the principal component analysis when n < p high dimensional data settings. In empirical comparisons with existing methods (maximum likelihood estimation and its best alternative method called Ledoit-Wolf estimation) and the proposed method(s), all the proposed regularized versions of pairwise differences covariance estimation perform well compared to those well-known estimators in estimating the covariance and principal components while minimizing the PCs’ overdispersion and cosine similarity error. Real data applications are presented. Keywords: High-dimensional Data, Dimensionality Reduction, Principal Component Analysis, Covariance Estimation 1 Introduction During the last two decades, technology has developed rapidly, minimizing the processing time of most statistical applications and algorithms. Because technology advances rapidly, researchers constantly introduce new technological concepts and methodologies into the research world. In that context, Machine Learning (ML) is a trending topic in the research and development (R&D) industry. As Arthur Samuel said in 1959, ML is a vital part of Artificial Intelligence (AI) that focuses on developing a statistical algorithm that can learn from data and generalize to new unsighted data without detailed human guidance or instruction [ 19 ] . Supervised, unsupervised, semi-supervised and reinforcement learning are the four main ML methods used in practice. A wide range of fields use ML algorithms. However, the choice of an ML algorithm depends on the characteristics of the data, expected usage, desired performance, domain of the statistical application, and other factors. However, ML algorithms are now often used to operate high-dimensional data in various applications in diverse fields, such as bioinformatics, forensic science, cyber security, agriculture, healthcare, finance, and computer science. However, fitting models in high-dimensional spaces requires substantial resources and time. The challenge Strengthens due to the ”Curse of Dimensionality,” [ 1 ] where increasing dimensions lead to sparsity in data, complicating model training, and potentially causing overfitting. Regularization or dimensional reduction (DR) techniques are essential to mitigate this challenge. DR helps compress data while retaining essential information, facilitating more efficient and effective model-building processes across domains. DR is a fundamental technique used in intermediate data analysis and Unsupervised ML algorithms, transforming the original data into a lower-dimensional space while preserving its essential characteristics. These algorithms find hidden patterns in the given data without any detailed human guidance or instruction. To gain better insights into high-dimensional data, it enhances the effectiveness of ML models by simplifying the structure of data, eliminating redundancy, reducing computation time, and improving visualization capability. In general, DR techniques can be divided into two main types: feature selection and feature extraction. The feature selection method comprises selecting the most critical subset of the original dimensions from the existing dataset [ 8 ] . It does not create any new dimensions based on the existing dataset. Meanwhile, the Feature extraction method creates new dimensions that are linear combinations of the original dimensions by transforming the original high-dimensional data into a lower-dimensional space [ 11 ] . Researchers use feature extraction methods more widely than feature selection methods. Several feature extraction methods exist, each with its unique approach and underlying principles. Some of the most widely used feature extraction techniques are principal component analysis (PCA), linear discriminant analysis, t-distributed stochastic neighbor embedding (t-SNE), and etc. PCA is the widely used method for DR among these feature extraction methods. PCA ensures reasonable data representation by transforming original high-dimensional data into a lower-dimensional space while keeping most of the total variance. When dealing with high-dimensional data, PCA facilitates more efficient ML algorithms by meaningfully shaping the data. Otherwise, the data’s redundancy minimizes the effectiveness of most high-dimensional statistical approaches. In recent years, there has been significant growth in utilizing high-dimensional data in situations where fewer observations than the number of dimensions. However, existing most commonly used DR techniques, including PCA, have not been effective in such kind of situations because most of the DR methods rely on the mean estimation, and with limited observations, the mean is more likely to be influenced by random noise and outliers, leading to inaccurate centering. This challenge has served a significant research interest to the researchers who focused mostly on issues of high-dimensional data analysis. Therefore, this research seeks to deeply study the effectiveness of PCA in these scenarios and understand how it can be optimized to handle the challenges posed by n < p high-dimensional data settings. By doing so, the goal is to enhance our ability to extract meaningful insights from high-dimensional data despite the limitations imposed by the insufficient sample size and the number of dimensions. 2 Problem Statement PCA transforms the original dataset into a differently structured dataset by using linear combinations of original dimensions while keeping the original dataset’s variance-covariance structure as much as possible. Therefore, PCA depends on the variance-covariance structure of the dataset and uses the Maximum Likelihood Estimation (MLE) of covariance estimation to obtain the sample variance-covariance structure of the original data set. Although traditional covariance MLE is recognized for its asymptotic unbiasedness, its efficacy becomes notably challenged in high-dimensional scenarios where n < p . This problem arises because MLE needs more observations ( n ) than the number of dimensions ( p ). When in n < p high-dimensional situations, the mean is more likely to be influenced by random noise and outliers, leading to inaccurate centering, and this increased variability translates into less reliable estimates of the actual central tendency of each dimension. That means, in n < p high-dimensional data settings, when n goes down compared to p , the Mean Squared Error (MSE) of the sample covariance matrix gets higher. This kind of higher MSE implies that the sample covariance matrix is not an accurate and reliable estimator for the true covariance [ 10 ] , [ 2 ] . These inaccurate sample estimates of variance-covariance structure result in misleading sample principal component estimates in high-dimensional data where n < p . we summarized those challenging issues concerning PCA as follows. 1. High Cosine Similarity Error (CSE) of sample PC due to overfitting, leading to sample PCs misaligning with population PCs. 2. Mean bias of eigenvalues increases as n decreases, causing inaccurate estimation of PCs. 3. Overestimated first n -1 eigenvalues and underestimated last p - n +1 eigenvalues, leading to misinterpretation. 4. Rank deficiency in sample covariance matrix limits the number of independent PCs to at most n -1, reducing explanatory power. 5. High condition number (ratio of largest to smallest

Read the complete text