Information-Geometric Decomposition of Generalization Error in Unsupervised Learning
Gilhan Kim · 2026
Read Information-Geometric Decomposition of Generalization Error in Unsupervised Learning by Gilhan Kim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-12340.txt.
Introduction
Kim decomposes the Kullback-Leibler generalization error of unsupervised learning into three non-negative parts: model error, data bias, and variance. The split holds exactly for any e-flat model class, following from the generalized Pythagorean theorem and a dual e-mixture variance identity. Applied to regularized principal component analysis with rank truncation and a noise floor, the optimal rank keeps exactly those eigenvalues above the floor. A boundary comparison yields a three-regime phase diagram of retain-all, interior, and collapse.
Catalog record
| CREATOR | Gilhan Kim |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68995 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.12340
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Information-Geometric Decomposition of Generalization Error in Unsupervised Learning Gilhan Kim Abstract We decompose the Kullback–Leibler generalization error (GE)—the expected KL divergence from the data distribution to the trained model—of unsupervised learning into three non-negative components: model error, data bias, and variance. The decomposition is exact for any e-flat model class and follows from two identities of information geometry: the generalized Pythagorean theorem and a dual e-mixture variance identity. As an analytically tractable demonstration, we apply the framework to ϵ \epsilon -PCA, a regularized principal component analysis in which the empirical covariance is truncated at rank N K N_{K} and discarded directions are pinned at a fixed noise floor ϵ \epsilon . Although rank-constrained ϵ \epsilon -PCA is not itself e-flat, it admits a technical reformulation with the same total GE on isotropic Gaussian data, under which each component of the decomposition takes closed form. The optimal rank emerges as the cutoff λ cut ∗ = ϵ \lambda_{\mathrm{cut}}^{*}=\epsilon —the model retains exactly those empirical eigenvalues exceeding the noise floor—with the cutoff reflecting a marginal-rate balance between model-error gain and data-bias cost. A boundary comparison further yields a three-regime phase diagram—retain-all, interior, and collapse—separated by the lower Marchenko–Pastur edge and an analytically computable collapse threshold ϵ ∗ ( α ) \epsilon_{*}(\alpha) , where α \alpha is the dimension-to-sample-size ratio. All claims are verified numerically. Keywords: Generalization error; Kullback–Leibler divergence; information geometry; generalized Pythagorean theorem; unsupervised learning; random matrix theory; Marchenko–Pastur distribution; principal component analysis; phase transitions. 1 Introduction Identifying the model complexity that minimizes the generalization error (GE) is a central problem of statistical learning. For supervised learning the standard guide is the bias–variance tradeoff: increasing the model complexity reduces the bias of the estimator at the cost of inflating its variance, and the optimal complexity is reached at the balance between these two competing contributions. For unsupervised learning, where the goal is to estimate an entire probability distribution rather than a conditional mean, an analogous decomposition was missing until recently. A natural setting in which to ask the question rigorously is that of fully visible generative models—models in which every random variable is directly observed, with no latent or hidden variables marginalised out. This class includes fully visible Boltzmann machines, multivariate Gaussian models such as \epsilon -PCA, naive-Bayes-type models, and any visible-only Markov random field, and excludes RBMs, variational autoencoders, and most modern deep generative models. The strongest form of the decomposition holds when the model manifold is e-flat (an exponential family in its natural parameters); when the parametric constraint is non-linear in those parameters—as for the rank-constrained \epsilon -PCA class—the framework still defines the same three components but their non-negativity is in general lost (Proposition 1 ). For \epsilon -PCA itself we circumvent this obstruction by a technical e-flat reformulation (Lemma 1 ) that has the same total GE on isotropic data, and apply the decomposition to that reformulation. In a previous joint work [ 15 ] , the present author and collaborators proposed that the GE of unsupervised learning likewise admits a two-component tradeoff, \mathrm{GE}\;=\;\mathrm{ME}\;+\;\mathrm{DE}, (1) between a model error ME, which decreases as the model becomes more expressive, and a data error DE, which grows because a finite training set is an imperfect representative of the true distribution. The decomposition was corroborated numerically for restricted Boltzmann machines (RBMs) trained on the two-dimensional Ising model and on the totally asymmetric simple exclusion process (TASEP), and the resulting U-shaped GE curves yielded a non-trivial optimal model size that grows with the complexity of the data. The decomposition ( 1 ), however, was an empirical observation. Two questions were left unanswered: (Q1) Is the data error DE itself decomposable into more elementary contributions, in particular into a part that measures finite-sample bias and a part that measures stochasticity of training? (Q2) Is there a class of models in which the decomposition can be derived from first principles, and in which the optimal model complexity can be computed in closed form? In this paper we answer both questions affirmatively, by placing the problem inside the framework of information geometry [ 1 , 2 ] and combining it with classical random matrix theory. The information-geometric analysis of statistical learning has a long history: the generalized Pythagorean theorem and dual flatness underlie the modern understanding of KL-divergence minimisation [ 8 ] and of maximum-likelihood estimation in exponential families [ 2 ] , while singular learning theory and the widely applicable information criterion (WAIC) [ 21 ] provide a parallel asymptotic decomposition of the Bayes generalization error in terms of the real log-canonical threshold of the model. On the random-matrix side, the limiting empirical spectrum of sample covariance matrices is governed by the Marchenko–Pastur law [ 16 , 4 , 9 ] , and the optimal threshold for hard truncation of singular values in low-rank denoising is the celebrated 4/\sqrt{3} rule of [ 10 ] , complemented in the spiked-covariance setting by the BBP transition of [ 5 , 17 , 6 ] . Our decomposition is closer in spirit to the bias–variance picture of frequentist learning than to the Bayes-loss decomposition of [ 21 ] , while our \epsilon -PCA closed-form result is the natural KL-divergence analogue, in a generative setting, of the spectral truncation rules of [ 10 , 20 ] . Our main contributions are: (i) Closed-form optimal rank for \epsilon -PCA. We introduce \epsilon -PCA, a regularized principal component analysis of zero-mean Gaussian data: the empirical covariance is truncated at rank N_{K} and the N_{V}-N_{K} discarded directions are pinned at a fixed noise floor \epsilon>0 . For isotropic true covariance, the Marchenko–Pastur spectral law identifies (Theorem 3 ) the unique interior local minimum of GE, given by the remarkably simple cutoff condition \lambda_{\mathrm{cut}}^{*}\;=\;\epsilon; (2) that is, the optimal model retains precisely those empirical covariance eigenvalues that exceed the intrinsic noise floor \epsilon . The corresponding optimal rank N_{K}^{*}/N_{V}=\int_{\epsilon}^{\lambda_{+}(\alpha)}p_{\mathrm{MP}}(\lambda;\alpha)\,d\lambda depends explicitly on \epsilon and on the aspect ratio \alpha=N_{V}/D . (ii) Three-regime phase diagram of the optimum. Comparing the interior local minimum of (i) with the boundary value at zero rank, we obtain (Proposition 2 ) a sharp three-regime phase diagram for the global optimum: a retain-all phase ( N_{K}^{*}=N_{V} ) for \epsilon below the lower Marchenko–Pastur edge, an interior phase in which N_{K}^{*} decreases monotonically in \epsilon , and a collapse phase ( N_{K}^{*}=0 ) above an analytically computable threshold \epsilon_{*}(\alpha) . The closed-form prediction of (i) is verified numerically against direct brute-force optimization in Section 4.6 . (iii) An information-geometric three-component decomposition that supplies the proof. The closed-form results of (i) and (ii) arise from a general decomposition of the unsupervised GE that we develop in parallel and which is of independent interest. Theorem 2 states that whenever the model manifold \mathcal{M} is an e-flat submanifold of the space of probability distributions, the GE satisfies the exact identity \displaystyle\big\langle D_{\mathrm{KL}}(P\,\|\,Q^{m})\big\rangle_{m}\;=\; \displaystyle\underbrace{D_{\mathrm{KL}}(P\,\|\,Q^{0})}_{\text{ME}}\;+\;\underbrace{D_{\mathrm{KL}}(Q^{0}\,\|\,\bar{Q})}_{\text{Data bias}} \displaystyle\;+\;\underbrace{\big\langle D_{\mathrm{KL}}(\bar{Q}\,\|\,Q^{m})\big\rangle_{m}}_{\text{Variance}}, (3) with each of the three terms a non-negative KL-type quantity; Q^{0} is the m-projection of P onto \mathcal{M} and \bar{Q} is the e-mixture of the trained models \{Q^{m}\}_{m} . The proof is by two successive applications of identities from information geometry: the generalized Pythagorean theorem [ 2 , Thm. 3.8] and a dual e-mixture