High-Dimensional Partial Least Squares: Spectral Analysis and Fundamental Limitations
Victor Léger [email protected], Florent Chatelain [email protected] · 2025
Read High-Dimensional Partial Least Squares: Spectral Analysis and Fundamental Limitations by Victor Léger [email protected], Florent Chatelain [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-15684.txt.
Introduction
Victor Leger and Florent Chatelain analyze partial least squares in high dimensions in a 2025 paper. Partial least squares extracts latent components shared by paired data sets, and it has decades of practical success in data integration. Its behavior when both data sets are high dimensional, though, lacked precise theory.
The authors study two data matrices that share a low rank common structure plus individual parts. Using random matrix theory, they characterize how well the estimated latent directions align with the true ones. The analysis explains reconstruction performance of the SVD based variant and finds regimes with counter intuitive limits. That variant still beats separate principal component analysis at detecting the shared subspace.
Catalog record
| CREATOR | Victor Léger [email protected], Florent Chatelain [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 190 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-15684
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
High-Dimensional Partial Least Squares: Spectral Analysis and Fundamental Limitations Victor Léger [email protected], Florent Chatelain [email protected] Abstract Partial Least Squares (PLS) is a widely used method for data integration, designed to extract latent components shared across paired high-dimensional datasets. Despite decades of practical success, a precise theoretical understanding of its behavior in high-dimensional regimes remains limited. In this paper, we study a data integration model in which two high-dimensional data matrices share a low-rank common latent structure while also containing individual-specific components. We analyze the singular vectors of the associated cross-covariance matrix using tools from random matrix theory and derive asymptotic characterizations of the alignment between estimated and true latent directions. These results provide a quantitative explanation of the reconstruction performance of the PLS variant based on Singular Value Decomposition (PLS-SVD) and identify regimes where the method exhibits counter-intuitive or limiting behavior. Building on this analysis, we compare PLS-SVD with principal component analysis applied separately to each dataset and show its asymptotic superiority in detecting the common latent subspace. Overall, our results offer a comprehensive theoretical understanding of high-dimensional PLS-SVD, clarifying both its advantages and fundamental limitations. Victor Léger and Florent Chatelain † † heading: XX 2025 1- 12/25; Revised XX/XX XX/XX XX-XXXX † † shortheadings: High-Dimensional Partial Least Squares / Léger and Chatelain † † firstpage: 1 † † editor: XXX keywords Partial Least Squares; High-Dimensional Statistics; Spiked Random Matrix Models; Low-Rank Approximations. 1 Introduction Partial Least Squares (PLS) is a widely-used dimension reduction method originally developed by Hermann Wold ( 32 ) in the 1960s to address econometric problems. Since then, it has been applied across many scientific domains, including chemometrics and spectrometric modeling ( 12 ) , as well as bioinformatics and genomics ( 5 ) . PLS is particularly effective in high-dimensional settings with limited sample sizes, a typical scenario in genomics, where datasets may contain around 20,000 gene expression measurements for only a few hundred individuals. This strenght has led to its broad adoption in genomic research for diverse tasks, such as analyzing gene correlations in yeast cells ( 11 ) , imputing missing data ( 6 ) , classifying cancer subtypes ( 27 ) , and predicting therapeutic outcomes ( 16 ) . Although numerous PLS algorithm variants exist ( 31 ) , they all rely on the same underlying principle: constructing latent components that maximizes the covariance with the reponse variables. Let {\mathbf{X}}\in\mathbb{R}^{n\times p} denote the matrix of the predictors and {\mathbf{Y}}\in\mathbb{R}^{n\times q} the matrix of the responses, where p and q are the respective numbers of variables and n is the common sample size. The PLS approach seeks to identify the projection directions that maximise the covariance between {\mathbf{X}} and {\mathbf{Y}} . More precisely, letting r denote the dimension of the latent space shared by {\mathbf{X}} and {\mathbf{Y}} , the goal is to extract r pairs of directions ({\mathbf{u}}_{k},{\mathbf{v}}_{k}) that maximize the sample covariance between the projected variables {\mathbf{X}}{\mathbf{u}}_{k} and {\mathbf{Y}}{\mathbf{v}}_{k} . At each iteration, the matrices {\mathbf{X}} and {\mathbf{Y}} (or their cross-covariance) are updated before computing the next pair of directions. For the first component, all PLS variants reduce to the same optimization problem: \displaystyle\argmax_{{\mathbf{u}}_{1},{\mathbf{v}}_{1}}\penalty\ \displaystyle\left({\mathbf{X}}{\mathbf{u}}_{1}\right)^{\top}{\mathbf{Y}}{\mathbf{v}}_{1}\equiv\argmax_{{\mathbf{u}}_{1},{\mathbf{v}}_{1}}\penalty\ {\mathbf{u}}_{1}^{\top}\mathbf{S}_{XY}{\mathbf{v}}_{1} s.t. \displaystyle\lVert{\mathbf{u}}_{1}\rVert=1\text{ and }\lVert{\mathbf{v}}_{1}\rVert=1, where \mathbf{S}_{XY}=\frac{1}{\sqrt{pq}}{\mathbf{X}}^{\top}{\mathbf{Y}} is the normalized sample cross-covariance matrix. This problem admits a unique solution for ({\mathbf{u}}_{1},{\mathbf{v}}_{1}) , which corresponds to the leading left and right singular vectors of \mathbf{S}_{XY} , as guaranteed by the Eckart–Young–Mirsky theorem. After extracting the first component, the subsequent steps depend on the specific PLS variant. Some algorithms perform separate deflation of {\mathbf{X}} and {\mathbf{Y}} , removing the part explained by the previously obtained components; others deflate the cross-product {\mathbf{X}}^{\top}{\mathbf{Y}} directly. In addition, each variant imposes orthogonality constraints to ensure identifiability of the successive directions. These constraints differ across PLS formulations and lead to distinct properties (typically geared toward prediction or modeling). At iteration k>1 , the next pair ({\mathbf{u}}_{k},{\mathbf{v}}_{k}) is obtained by solving \displaystyle\argmax_{{\mathbf{u}}_{k},{\mathbf{v}}_{k}} \displaystyle\penalty\ {\mathbf{u}}_{k}^{\top}\mathbf{S}_{XY}^{(k)}{\mathbf{v}}_{k} \displaystyle\textrm{s.t.}\penalty\ \lVert{\mathbf{u}}_{k}\rVert \displaystyle=1\text{ and }\lVert{\mathbf{v}}_{k}\rVert=1,\ \text{and orthogonality constraints}, where \mathbf{S}_{XY}^{(k)} denotes the updated version of \mathbf{S}_{XY} at step k . Because numerous algorithmic variants exist, our goal is not to survey all implementations but rather to highlight the behavior and the fundamental challenges faced by PLS in high-dimensional settings. For this purpose, we focus on the PLS-SVD variant ( 31 ) , which imposes that each new direction satisfies {\mathbf{u}}_{k}\perp\operatorname{span}\{{\mathbf{u}}_{1},\ldots,{\mathbf{u}}_{k-1}\},\qquad{\mathbf{v}}_{k}\perp\operatorname{span}\{{\mathbf{v}}_{1},\ldots,{\mathbf{v}}_{k-1}\}. Under these constraints, the Eckart–Young–Mirsky theorem ensures that all components can be obtained simultaneously through the singular value decomposition (SVD) of \mathbf{S}_{XY} , thereby avoiding any iterative deflation step. Our analysis therefore focuses on the spectral properties of the square symmetric matrices \displaystyle{\mathbf{K}}\equiv \displaystyle\frac{1}{pq}{\mathbf{Y}}^{\top}{\mathbf{X}}{\mathbf{X}}^{\top}{\mathbf{Y}}=\mathbf{S}_{XY}^{\top}\mathbf{S}_{XY}\in\mathbb{R}^{q\times q}, (1) \displaystyle\tilde{{\mathbf{K}}}\equiv \displaystyle\frac{1}{pq}{\mathbf{X}}^{\top}{\mathbf{Y}}{\mathbf{Y}}^{\top}{\mathbf{X}}=\mathbf{S}_{XY}\mathbf{S}_{XY}^{\top}\in\mathbb{R}^{p\times p}, (2) whose nonzero eigenvalues coincide with the nonzero squared singular values of \mathbf{S}_{XY} , and whose eigenvectors correspond respectively to the right and left singular vectors of \mathbf{S}_{XY} . In this work, we further consider the following asymptotic regime. Assumptions A1 (High-dimensional asymptotic regime) . We assume that p,q,n\to\infty with the ratios of the sample size to the dimensions converging to finite positive constants: \displaystyle\frac{n}{p}\to\beta_{p}>0,\qquad\frac{n}{q}\to\beta_{q}>0,\qquad\beta\equiv\max(\beta_{p},\beta_{q}), (3) where \beta can be interpreted as the limit of the ratio \frac{n}{d} with d=\min(p,q) . This high-dimensional framework is particularly well suited for random matrix theory, which provides sharp characterizations of the limiting spectral distribution of the involved large matrices, the behavior of their largest eigenvalues, and the alignment of the corresponding eigenvectors. 1.1 Related Works The statistical properties of PLS have been studied from various perspectives over the past decades. Early theoretical work focused on establishing the algorithmic foundations and interpretations of PLS ( 14 ; 13 ) . Later theoretical developments analyzed the structure of PLS estimators and derived conditions for their consistency and asymptotic behavior in classical settings where the number of samples grows while dimensions remain fixed ( 24 ; 15 ) . However, these results do not directly apply to modern high-dimensional scenarios, and subsequent work has highlighted the limitations of PLS consistency when the number of variables is large relative to the sample size ( 7 ) . Random Matrix Theory and Spiked Models. The application of random matrix theory (RMT) to understand high-dimensional statistical methods has gained considerable attention. Foundational results include the Marčenko–Pastur law ( 22 ) , which characterizes the limiting spectral distribution of an isotropic sample covariance matrices, and the Baik–Ben Arous–Péché (BBP) phase transition ( 2 ) for spiked covariance models, which specify the threshold above which signals yield isolated eigenvalues. These tools have been successfully applied to principal component analysis ( 26 ; 21 ; 4 ) , as well as to computationally efficient implementations ( 9 ) , providing precise characterizations of eigenvalue behavior and eigenvector consistency in high-dimensional regimes. Multimodal data have also been studied through the asymptotic analysis of low-rank tensor decompositions; see, for example, ( 19 ) . Our work extends these techniques to the PLS setting, where the presence of two coupled data matrices creates additional complexity. PLS in High Dimensions. Despite its widespread use across numerous scientific domains, PLS remains comparatively less understood from a theoretical standpoint in high-dimensional settings. Recent efforts have begun addressing this gap. 7 proposed sparse PLS methods to regularize the solutions, while 8 analyzed the asymptotic prediction behavior of PLS regression across