Exponential Convergence of CAVI for Bayesian PCA
Arghya Datta, Philippe Gagnon, Florian Maire · 2025
Read Exponential Convergence of CAVI for Bayesian PCA by Arghya Datta, Philippe Gagnon, Florian Maire (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16145.txt.
Introduction
Arghya Datta, Philippe Gagnon, and Florian Maire prove how fast a common learning algorithm settles for Bayesian principal component analysis. The Bayesian form of PCA shrinks data while tracking uncertainty, a plus over the classic form. Its parameters are usually learned by CAVI, an algorithm that climbs one coordinate at a time. How fast it converges was unknown until now.
For a single component, the team proves exponential convergence through a link to the classic power iteration method. Classic PCA turns out to be the point estimate of the Bayesian answer. For any number of components they prove the same speed with fresh tools, plus a new bound on a divergence measure for bell curves.
Catalog record
| CREATOR | Arghya Datta, Philippe Gagnon, Florian Maire |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 116 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-16145
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Exponential Convergence of CAVI for Bayesian PCA Arghya Datta, Philippe Gagnon, Florian Maire Abstract Probabilistic principal component analysis (PCA) and its Bayesian variant (BPCA) are widely used for dimension reduction in machine learning and statistics. The main advantage of probabilistic PCA over the traditional formulation is allowing uncertainty quantification. The parameters of BPCA are typically learned using mean-field variational inference, and in particular, the coordinate ascent variational inference (CAVI) algorithm. So far, the convergence speed of CAVI for BPCA has not been characterized. In our paper, we fill this gap in the literature. Firstly, we prove a precise exponential convergence result in the case where the model uses a single principal component (PC). Interestingly, this result is established through a connection with the classical power iteration algorithm and it indicates that traditional PCA is retrieved as points estimates of the BPCA parameters. Secondly, we leverage recent tools to prove exponential convergence of CAVI for the model with any number of PCs, thus leading to a more general result, but one that is of a slightly different flavor. To prove the latter result, we additionally needed to introduce a novel lower bound for the symmetric Kullback–Leibler divergence between two multivariate normal distributions, which, we believe, is of independent interest in information theory. Keywords: coordinate ascent variational inference, principal component analysis, dynamical systems, power iteration algorithm, mean-field variational inference, information theoretic lower bounds. 1 Introduction 1.1 Probabilistic principal component analysis Principal component analysis (PCA) is a ubiquitous dimension reduction tool. It has a broad spectrum of applications ranging from computer vision ( Rousseau et al.,, 2024 ) to finance ( Veronesi,, 2016 ; Swanson et al.,, 2020 ) . In Veronesi, (2016) , it is explained that PCA is commonly used in a context of interest rate modeling to produce a low dimensional input to the model. More precisely, PCA is applied to yield curve data and three principal components (PCs), roughly corresponding to the level, slope and curvature of the yield curves, serve as inputs to the model. When the PCs have an interesting interpretation and serve as inputs to a model like in this example, uncertainty quantification is of importance. However, PCA is not a statistical model and thus uncertainty quantification is not available. Probabilistic principal component analysis (PPCA) is a model introduced by Tipping and Bishop, (1999) to remedy the situation. It has been widely used for uncertainty quantification in machine learning and statistics (see, e.g., Ilin and Raiko, (2010) and Wu et al., (2021) ). The PPCA model is defined as: x_{i}=\textbf{W}z_{i}+\tau^{-1/2}\epsilon_{i},\quad i=1,\ldots,n, (1) where \textbf{W}\in\mathbb{R}^{d\times k} is a loading matrix, z_{i}\in\mathbb{R}^{k} a standard normal latent variable, \tau>0 a precision parameter, and \epsilon_{i}\in\mathbb{R}^{d} a standardized normal error vector. Thus, x_{i}\in\mathbb{R}^{d} plays the role of a centered data vector, and it is assumed that n\geqslant d . The columns of the matrix Z , with lines given by z_{1},\ldots,z_{n} , play an analogous role as the PCs in PCA. The PPCA model is complex by, among others, its non-identifiability. A Bayesian version, henceforth referred to as BPCA, has been introduced in Bishop, (1998) . In this Bayesian version, W has a prior given by a matrix normal distribution ( Gupta and Nagar,, 1999 ) with mean \mathbf{0} and covariance parameter I_{d}\otimes\Lambda^{-1} , where I_{d} is the identity matrix of size d and \Lambda is a diagonal matrix of precision parameters. This covariance structure implies that the rows of W are independent with a shared covariance of \Lambda^{-1} . The prior distribution on \tau is a gamma. Despite the widespread use of such models, their theoretical properties remain largely unexplored. Recently, however, there has been a growing interest in establishing rigorous guarantees. For example, the consistency of the maximum likelihood estimator has been proved in Datta and Chakrabarty, (2023) . On the Bayesian side, a selection procedure for the number of PCs has been proposed and established to be consistent in Chérief-Abdellatif, (2019) . In this work, we establish exponential convergence of CAVI (coordinate ascent variational inference, Bishop and Nasrabadi, (2006) ) for mean-field variational inference of BPCA. Variational inference is used when Bayesian inference using Markov chain Monte Carlo methods ( Hastings,, 1970 ; Neal et al.,, 2011 ) are inefficient. In Bishop, (1999) , it has been highlighted to be the case for BPCA given the high-dimensional nature of the model (the number of unknowns is {dk+nk} ). In our theoretical analyses, \tau is assumed fixed and known to simplify, with a value equal to \tau_{0}>0 . 1.2 Contributions and organization of the paper In Section 2 , we present an overview of variational inference and the details of CAVI for BPCA, which will be useful to introduce our exponential convergence results for this numerical scheme. Recently, theoretical results about convergence speed of CAVI have been established in several contexts. In Zhang and Zhou, (2020) , the context is community detection, whereas in Plummer et al., (2020) and Goplerud et al., (2024) the contexts are 2D Ising models and generalized linear mixed models, respectively. More general results are presented under a log-concave framework in Lavenant and Zanella, (2024) and Arnese and Lacker, (2024) , and under a general abstract framework in Bhattacharya et al., (2025) . The results in Lavenant and Zanella, (2024) and Arnese and Lacker, (2024) cannot be exploited here as BPCA does not fall under the log-concave framework. For one of our theoretical results, we leverage recent tools provided in Bhattacharya et al., (2025) . By doing so, we highlight the strengths and limitations of such abstract tools. We now present our main contributions. • In Section 3 , we prove a precise exponential convergence result in the case k=1 , meaning that the model includes a single PC. To achieve this, we establish a novel connection between CAVI and the classical power iteration algorithm ( Trefethen and Bau,, 2022 , chapter 25) coming from the field of numerical linear algebra whose purpose is to compute the leading singular vectors of a matrix ( Theorem 1 ). • In Section 4 , we provide the first general (for any k ) theoretical result on the exponential convergence of CAVI for BPCA ( Theorem 2 ). This result being more general than that of Section 3 , it requires different tools and is of a slightly different flavor. • A key ingredient of the proof of Theorem 2 is a novel lower bound on the symmetric Kullback–Leibler (KL) divergence between two multivariate normal distributions ( Lemma 3 ). We believe this result is of independent interest in information theory as traditional bounds based on Pinsker’s inequality are often too loose to be useful in practice. 2 Variational inference and CAVI In variational inference, we resort to find a “ good enough ” approximation to the posterior distribution, that we denote by \pi , hoping to capture its key statistical properties. We begin by identifying a family \mathcal{Q} of parametric probability distributions for the approximation. Assume, as will be the case in our context of BPCA, that \pi and all distributions in \mathcal{Q} are absolutely continuous with respect to the Lebesgue measure. We then pick the best candidate from the family \mathcal{Q} , in the sense of