Beyond Diagonal Covariance: Flexible Posterior VAEs via Free-Form Injective Flows

Peter Sorrenson, Lukas Lührs, Hans Olischläger, Ullrich Köthe · 2025

Read Beyond Diagonal Covariance: Flexible Posterior VAEs via Free-Form Injective Flows by Peter Sorrenson, Lukas Lührs, Hans Olischläger, Ullrich Köthe (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01522.txt.

Introduction

Peter Sorrenson and co-authors present a 2025 paper on loosening a straitjacket inside variational autoencoders, a family of generative models. These models squeeze data into a compact code and learn to rebuild it. For speed, the code's uncertainty is usually forced into a simple axis-aligned shape, which the authors show limits what the model can express.

Their fix reframes a recently proposed flow model as an autoencoder with a far more flexible, implicit uncertainty shape. With the right penalty added, that shape equals a full Gaussian in one special case but can bend beyond it in general. Small worked examples in the paper make the limits of the simple shape concrete.

Catalog record

CREATORPeter Sorrenson, Lukas Lührs, Hans Olischläger, Ullrich Köthe
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-01522

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Diagonal Covariance: Flexible Posterior VAEs via Free-Form Injective Flows Peter Sorrenson, Lukas Lührs, Hans Olischläger, Ullrich Köthe Abstract Variational Autoencoders (VAEs) are powerful generative models widely used for learning interpretable latent spaces, quantifying uncertainty, and compressing data for downstream generative tasks. VAEs typically rely on diagonal Gaussian posteriors due to computational constraints. Using arguments grounded in differential geometry, we demonstrate inherent limitations in the representational capacity of diagonal covariance VAEs, as illustrated by explicit low-dimensional examples. In response, we show that a regularized variant of the recently introduced Free-form Injective Flow (FIF) can be interpreted as a VAE featuring a highly flexible, implicitly defined posterior. Crucially, this regularization yields a posterior equivalent to a full Gaussian covariance distribution, yet maintains computational costs comparable to standard diagonal covariance VAEs. Experiments on image datasets validate our approach, demonstrating that incorporating full covariance substantially improves model likelihood. 1 Introduction Deep generative models have reshaped the way we represent, compress, and create high–dimensional data such as images, audio, or text. Among them, Variational Autoencoders (VAEs) ( Kingma and Welling 2014 ; Rezende et al. 2014 ) stand out for their elegant combination of amortized inference and latent-variable modeling, enabling smooth, semantically meaningful latent spaces and principled uncertainty estimates. Yet, in practical VAE implementations the approximate posterior is almost invariably chosen to be a diagonal Gaussian. This design choice is driven by the simplicity of the reparameterization trick and a closed-form Kullback-Leibler (KL) term, but it severely limits the posterior’s capacity to capture correlations across latent dimensions and thus constrains the kinds of data manifolds the model can learn. Beyond diagonal Gaussians. Several research lines have tried to lift the diagonal constraint. Hierarchical VAEs ( Sønderby et al. 2016 ; Maaløe et al. 2019 ) and sophisticated priors such as VampPrior ( Tomczak and Welling 2018 ) enrich the latent structure enrich the latent structure while still using diagonal Gaussians at the level of each latent block. Normalizing-flow posteriors ( Rezende and Mohamed 2015 ; Kingma et al. 2016 ) provide a strictly more expressive family at the price of stacking many flow steps whose Jacobians must be triangular or specially constructed for tractability. Full-covariance Gaussians can be learned directly via Cholesky factors ( Kingma et al. 2019 ) or by Householder transformations ( Tomczak and Welling 2016 ) , yet both scale quadratically in latent dimension. The Variational Laplace Autoencoder (VLAE) ( Park et al. 2019 ) solves an optimization problem to estimate the mean and (potentially non-diagonal) covariance of the posterior, but involves expensive Jacobian evaluation and matrix inversion which does not scale well with latent dimension. Injective flows as manifold learners. A complementary line of work adapts exact likelihood normalizing flows ( Rezende and Mohamed 2015 ; Kobyzev et al. 2021 ; Papamakarios et al. 2021 ) to learn a manifold embedded in the data space, trading a bijectivity constraint for injectivity. Such injective normalizing flows simultaneously learn and maximize likelihood on a manifold by jointly optimizing an encoder-decoder pair together with the change-of-variables term ( Brehmer and Cranmer 2020 ; Caterini et al. 2021 ) . Very recently, Free-form Injective Flows (FIFs) ( Sorrenson et al. 2024 ; Draxler et al. 2024 ) removed the architectural constraints of injective normalizing flows by introducing an efficient Hutchinson-style estimator for the gradient of the log-determinant. FIFs learn both manifold and density simultaneously, but they provide no direct latent-variable interpretation and no tractable posterior. This paper. We observe that, under a mild Laplace approximation, the geometry of FIFs naturally induces a full-covariance Gaussian posterior whose covariance is given implicitly through encoder and decoder Jacobians. Building on this insight, we introduce FIVE, a F ree-form I njective-Flow-based V ariational Auto E ncoder that: 1. Leverages FIFs to furnish VAEs with a highly flexible posterior that is computationally only slightly more expensive than the diagonal case. 2. Provides a differential-geometric analysis revealing when and why diagonal posteriors fail to represent curved data manifolds. 3. Proposes a curvature-aware regularization that stabilizes training and yields an approximate KL term without computing determinants explicitly. 4. Matches or surpasses the log-likelihood of both VAEs and standalone FIFs on benchmark image datasets. Taken together, our results indicate that moving beyond diagonal covariance is not only theoretically well-motivated but also practically attainable at scale. We hope that FIVE will serve as a drop-in replacement for standard VAEs when richer posteriors estimates are desired. 2 Variational Autoencoders and their limitations Variational Autoencoders (VAEs) are latent-variable models characterized by a prior distribution over latent variables p(z) (typically standard normal) and a conditionally normal distribution over data points given latent variables of the form: p(x|z)=\mathcal{N}(x;g(z),\sigma^{2}I), (1) where g(z) is a neural network decoder, and \sigma^{2} is a scalar controlling the variance of the likelihood. The marginal distribution p(x) is obtained by integrating over latent variables: p(x)=\int p(x|z)p(z)\,dz. (2) This integral is typically intractable, making direct optimization of the KL divergence D_{\text{KL}}(q(x)\|p(x)) between the training distribution q(x) and the model distribution p(x) difficult. 2.1 Introducing the variational distribution To circumvent the computational difficulty, VAEs introduce an auxiliary distribution, called the variational posterior, q(z|x) . By incorporating this distribution, we have two distributions defined over the joint latent-data space (x,z) : \displaystyle p(x,z) \displaystyle=p(z)p(x|z), (3) \displaystyle q(x,z) \displaystyle=q(x)q(z|x). (4) Instead of minimizing D_{\text{KL}}(q(x)\|p(x)) directly, we minimize the divergence between these two joint distributions: D_{\text{KL}}(q(x,z)\|p(x,z))=D_{\text{KL}}(q(x)\|p(x))+\mathbb{E}_{q(x)}\left[D_{\text{KL}}(q(z|x)\|p(z|x))\right]. (5) This divergence provides an upper bound on D_{\text{KL}}(q(x)\|p(x)) due to the non-negativity of KL divergence. Explicitly, we have: D_{\text{KL}}(q(x)\|p(x))\leq D_{\text{KL}}(q(x,z)\|p(x,z)), (6) with the gap precisely given by: \mathbb{E}_{q(x)}[D_{\text{KL}}(q(z|x)\|p(z|x))]. (7) 2.2 Evidence Lower Bound (ELBO) Typically, VAEs assume that the variational distribution q(z|x) is a diagonal Gaussian: q(z|x)=\mathcal{N}(z;\mu(x),\text{diag}(\sigma^{2}(x))), (8) for computational tractability. If we discard the unknown entropy of q(x) , we can rewrite eq. 5 as the well-known Evidence Lower Bound (ELBO): \text{ELBO}(x)=\mathbb{E}_{q(z|x)}[\log p(x|z)]-D_{\text{KL}}(q(z|x)\|p(z)). (9) Since typically p(z) is the standard normal, we can evaluate the KL-divergence via the analytical expression: D_{\text{KL}}(\mathcal{N}(\mu,\Sigma)\|\mathcal{N}(0,I_{d}))=\frac{1}{2}\left(\|\mu\|^{2}+\operatorname{tr}(\Sigma)-d-\log\det(\Sigma)\right) (10) which is easy to evaluate for diagonal \Sigma . Optimization then becomes a maximization of the ELBO, here equivalently written as a minimization of the negative ELBO: \mathcal{L}(x)=\mathbb{E}_{q(z|x)}[-\log p(x|z)]+D_{\text{KL}}(q(z|x)\|p(z)). (11) 2.3 Form of the posterior and its implications Since q(z|x) is assumed diagonal Gaussian, the optimal posterior p(z|x) which minimizes D_{\text{KL}}(q(z|x)\|p(z|x)) will also be approximately diagonal Gaussian. This assumption significantly constrains the decoder function g(z) . Specifically, for a Gaussian posterior approximation, the following theorem applies: Theorem 1 . When optimizing the above described latent variable model (standard VAE) with diagonal Gaussian q(z|x) and injective decoder, the optimal decoder g has an approximately diagonal pull-back metric, i.e., g^{\prime}(z)^{\top}g^{\prime}(z) is approximately diagonal for all z , where g^{\prime}(z)=\frac{\partial g(z)}{\partial z} is the Jacobian of the decoder. The full proof is in the appendix. Proof sketch: If the loss is minimal, then due to the decomposition in eq. 5 , D_{\text{KL}}(q(z|x)\|p(z|x))=0 . This implies that p(z|x)=q(z|x) and hence p(z|x) is diagonal Gaussian. Strictly speaking, the requirement that p(z|x) is Gaussian implies that g is linear. In practice, g typically needs to be nonlinear due to the structure of the data, but we will find that g needs to “look linear” on the scale of \sigma . This is considered in more detail in the appendix, where we show that second

Read the complete text