Multivariate Variational Autoencoder
Mehmet Can Yavuz · 2025
Read Multivariate Variational Autoencoder by Mehmet Can Yavuz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07472.txt.
Introduction
Mehmet Can Yavuz rethinks the hidden space inside variational autoencoders, models that compress data into a small code. Standard versions force that code to follow a simple diagonal shape. The shortcut eases training but blocks linked uncertainty and often wastes dimensions.
The Multivariate Variational Autoencoder keeps a full covariance structure by combining per-sample scales with one shared coupling matrix. Correlated code factors become possible while the training objective stays in closed form. Sampling also stays simple. The goal is codes that are expressive, well shaped, and honestly calibrated. Better codes help generation, compression, and later analysis of what the model learned.
Catalog record
| CREATOR | Mehmet Can Yavuz |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-07472
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multivariate Variational Autoencoder Mehmet Can Yavuz Abstract Learning latent representations that are simultaneously expressive, geometrically well-structured, and reliably calibrated remains a central challenge for Variational Autoencoders (VAEs). Standard VAEs typically assume a diagonal Gaussian posterior, which simplifies optimization but rules out correlated uncertainty and often yields entangled or redundant latent dimensions. We introduce the Multivariate Variational Autoencoder (MVAE), a tractable full-covariance extension of the VAE that augments the encoder with sample-specific diagonal scales and a global coupling matrix. This induces a multivariate Gaussian posterior of the form 𝒩 ( μ ϕ ( x ) , C diag ( σ ϕ 2 ( x ) ) C ⊤ ) \mathcal{N}(\mu_{\phi}(x),C\operatorname{diag}(\sigma_{\phi}^{2}(x))C^{\top}) , enabling correlated latent factors while preserving a closed-form KL divergence and a simple reparameterization path. Beyond likelihood, we propose a multi-criterion evaluation protocol that jointly assesses reconstruction quality (MSE, ELBO), downstream discrimination (linear probes), probabilistic calibration (NLL, Brier, ECE), and unsupervised structure (NMI, ARI). Across Larochelle-style MNIST variants, Fashion-MNIST, and CIFAR-10/100, MVAE consistently matches or outperforms diagonal-covariance VAEs of comparable capacity, with particularly notable gains in calibration and clustering metrics at both low and high latent dimensions. Qualitative analyses further show smoother, more semantically coherent latent traversals and sharper reconstructions. All code, dataset splits, and evaluation utilities are released to facilitate reproducible comparison and future extensions of multivariate posterior models. I Introduction Learning compact yet expressive latent representations that support both high-fidelity generation and downstream discrimination remains a central objective in modern machine learning. Variational Autoencoders (VAEs) [ 8 , 13 ] address this goal by maximizing an evidence lower bound (ELBO) that jointly encourages accurate reconstructions and a regularized approximate posterior over latent variables. However, the conventional VAE parameterizes the encoder posterior with a diagonal covariance, implicitly assuming conditional independence among latent coordinates. This factorization simplifies optimization but constrains the geometry of the latent manifold, often producing entangled or redundant dimensions and exacerbating posterior collapse in high-capacity decoders. We address these limitations with a Multivariate Variational Autoencoder (MVAE) that augments the encoder with (i) a sample-specific diagonal scale and (ii) a global coupling matrix that induces persistent correlations in the latent space. Formally, each posterior is expressed as q_{\phi}(\mathbf{z}\mid\mathbf{x})=\mathcal{N}\!\big(\boldsymbol{\mu}_{\phi}(\mathbf{x}),\,\mathbf{C}\,\mathrm{diag}(\boldsymbol{\sigma}^{2}_{\phi}(\mathbf{x}))\,\mathbf{C}^{\top}\big), where the learnable matrix \mathbf{C} captures dataset-level dependencies while \boldsymbol{\sigma}_{\phi}(\mathbf{x}) modulates per-sample uncertainty. This decomposition yields a structured full-covariance family with analytic KL divergence, maintaining computational tractability while substantially enriching posterior geometry. In addition, the mean is linearly coupled through \mathbf{C} , enabling anisotropic yet interpretable transformations across latent dimensions. Empirically, across Larochelle-style MNIST variants [ 10 , 18 ] , Fashion-MNIST [ 20 ] , and CIFAR-10/100 [ 9 ] , MVAE consistently improves over diagonal-covariance VAEs of matched capacity in terms of ELBO (↑), reconstruction error (↓), calibration quality (NLL, Brier, ECE; ↓), and unsupervised structure (NMI/ARI; ↑). These gains confirm that incorporating correlated uncertainty and mean coupling leads to more robust and informative latent spaces. Beyond generative quality, we explicitly evaluate the usefulness of learned representations through linear probing, clustering, and probabilistic calibration. Together, these complementary criteria—generation, discrimination, calibration, and structure—provide a holistic view of latent-space quality that better reflects practical deployment. To facilitate reproducibility, we release complete training and evaluation pipelines, including metrics, dataset splits, and sweep utilities. Contributions. (1) We propose the Multivariate Variational Autoencoder (MVAE), which integrates a global coupling matrix and per-sample scales to realize a full-covariance Gaussian posterior with closed-form KL divergence. (2) We introduce a principled, multi-criterion evaluation framework combining ELBO, reconstruction fidelity, linear probing, calibration, and clustering under standardized seeds and data splits. (3) We release an open, fully reproducible implementation with scripts, logs, and evaluation utilities to enable fair and transparent comparisons. Fig. 1 : Representative samples from all benchmark datasets employed in our study. Each row (from top to bottom) corresponds to MNIST Basic, MNIST Rotated, MNIST with Random Backgrounds, MNIST with Background Images, CIFAR-10, CIFAR-100, and Fashion-MNIST. These examples illustrate the diversity in visual complexity, background texture, and object categories across datasets. II Related Work VAEs and posterior expressiveness. The Variational Autoencoder (VAE) framework [ 8 , 13 ] popularized amortized variational inference via the reparameterization trick, typically under a diagonal Gaussian posterior. Subsequent research has sought to enhance posterior flexibility while retaining tractable objectives. Normalizing flows transform simple posteriors into richer distributions through invertible mappings [ 14 , 7 ] , while auxiliary-variable and hierarchical extensions further expand expressiveness [ 12 , 16 ] . Mixture-based or learned prior formulations such as VampPrior [ 17 ] enrich the generative model by increasing prior diversity. Our approach is orthogonal and complementary: we retain a Gaussian posterior but remove the diagonal restriction by learning a full covariance. Full-covariance variational families. Full-covariance posteriors in deep VAEs have been explored through structured covariance parameterizations—e.g., low-rank plus diagonal decompositions, precision (information) parameterizations, or Kronecker/sparse structures—to balance flexibility and computational cost [ 15 , 19 ] . The proposed coupling-matrix formulation falls within this line of work, but emphasizes a global, data-shared correlation structure that remains amortized and lightweight. Disentanglement, structure, and robustness. Disentanglement-oriented VAEs, such as \beta -VAE [ 5 ] , FactorVAE [ 6 ] , and \beta -TCVAE [ 3 ] , promote statistically independent latent factors but often trade off reconstruction fidelity and are sensitive to hyperparameter tuning. In contrast, our coupling matrix introduces controlled anisotropy in the latent mean, improving linear separability and clustering without relying on mutual-information surrogates or adversarial penalties. Complementary robustness analyses—particularly those using Larochelle-style MNIST variants with structured and unstructured corruptions [ 10 , 18 ] —suggest that correlated posteriors generalize better under geometric and background perturbations. Evaluation beyond likelihood. ELBO or likelihood-based metrics alone may not fully reflect representation quality. Linear probes [ 1 ] , clustering indices, and calibration metrics [ 4 , 2 ] provide complementary perspectives on separability, structure, and uncertainty reliability. Accordingly, we report a unified set of (↑/↓)-oriented metrics—ELBO, per-pixel MSE, probe accuracy/NLL/Brier/ECE, and NMI/ARI—to provide a more faithful and multidimensional evaluation of model quality under consistent capacity and training budgets. Positioning. Compared to flow-augmented VAEs [ 14 , 7 ] , our Multivariate VAE (MVAE) maintains a simple and computationally efficient encoder–decoder architecture while explicitly modeling latent correlations through a global coupling matrix. Relative to disentanglement-penalized approaches [ 5 , 6 , 3 ] , we rely on geometric structure rather than additional regularization terms. These choices yield consistent performance gains across diverse visual domains (digits, fashion, and natural images) with minimal engineering overhead and seamless integration into existing VAE pipelines. III Datasets To evaluate the robustness and generalization capacity of our proposed method, we conducted experiments on a diverse suite of classification datasets that differ in background composition, rotation, visual complexity, and synthetic constraints. The datasets are derived from canonical benchmarks introduced by Larochelle et al. [ 10 ] and Vincent et al. [ 18 ] , with most being extensions or variants of the MNIST handwritten digit dataset [ 11 ] . Each dataset is briefly summarized below with details on its structure, preprocessing, and data splits. All datasets and accompanying code are publicly available. 1 1 1 HuggingFace Dataset 2 2 2 GitHub Repository • MNIST Basic: This dataset consists of 28\times 28 grayscale images of handwritten