Density-Informed VAE (DiVAE): Reliable Log-Prior Probability via Density Alignment Regularization
Michele Alessi, Alessio Ansuini, Alex Rodríguez, [0.8em] Department of Mathematics, Informatics, Geosciences, University of Trieste, Italy · 2025
Read Density-Informed VAE (DiVAE): Reliable Log-Prior Probability via Density Alignment Regularization by Michele Alessi, Alessio Ansuini, Alex Rodríguez, [0.8em] Department of Mathematics, Informatics, Geosciences, University of Trieste, Italy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-03928.txt.
Introduction
Variational autoencoders compress data into a latent space shaped like a simple prior, but real data rarely spreads itself so neatly. Michele Alessi, Alessio Ansuini, and Alex Rodríguez introduce DiVAE in a 2025 machine learning preprint. Their addition nudges the encoder to place latent points in proportion to how dense the original data is in each region.
The method adds one lightweight penalty term to the usual training objective, comparing the model's log-prior probability with a log-density estimated from data. A precision weighting keeps the penalty robust, and the extra computing cost is negligible. If the prior itself can learn, it drifts toward high-density regions too. On synthetic datasets, DiVAE matched latent densities to ground truth better and calibrated out-of-distribution uncertainty more honestly. On handwritten digits, it improved both interpretability and detection of out-of-distribution inputs with learnable priors.
Catalog record
| CREATOR | Michele Alessi, Alessio Ansuini, Alex Rodríguez, [0.8em] Department of Mathematics, Informatics, Geosciences, University of Trieste, Italy |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-03928
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Density-Informed VAE (DiVAE): Reliable Log-Prior Probability via Density Alignment Regularization Michele Alessi, Alessio Ansuini, Alex Rodríguez, [0.8em] Department of Mathematics, Informatics, Geosciences, University of Trieste, Italy Abstract We introduce Density-Informed VAE (DiVAE) , a lightweight, data-driven regularizer that aligns the VAE log-prior probability log p Z ( z ) \log p_{Z}(z) with a log-density estimated from data. Standard VAEs match latents to a simple prior, overlooking density structure in the data-space. DiVAE encourages the encoder to allocate posterior mass in proportion to data-space density and, when the prior is learnable, nudges the prior toward high-density regions. This is realized by adding a robust, precision-weighted penalty to the ELBO, incurring negligible computational overhead. On synthetic datasets, DiVAE (i) improves distributional alignment of latent log-densities to its ground truth counterpart, (ii) improves prior coverage, and (iii) yields better OOD uncertainty calibration. On MNIST , DiVAE improves alignment of the prior with external estimates of the density, providing better interpretability, and improves OOD detection for learnable priors. 1 Introduction Variational autoencoders (VAEs) typically assume a simple prior p_{Z} (e.g., \mathcal{N}(0,I) ) 6 . This choice encourages the variational posterior q_{\phi}(z\mid x) to place latent codes near the origin and to distribute mass according to the prior rather than the data’s empirical density . As a result, the log-prior probability s(z)=\log p_{Z}(z) may be poorly aligned with how samples concentrate in data space, reducing its usefulness for anomaly detection, out-of-distribution (OOD) scoring, and uncertainty estimation. It also contributes to well-known issues such as the hole problem –regions of high prior mass unsupported by the theoretically optimal prior: the aggregated variational posterior 18 ; 14 . To address the limitations of a simple, fixed prior described above, we introduce Density-Informed VAE (DiVAE) , which injects a per-sample density signal \rho –an estimate of log-density measured in a data-derived projection– into training via a regularization term. Acting as a density teacher , DiVAE adds an alignment penalty that pushes the encoder to place latent codes in proportion to \rho . Consequently, even with a non-learnable prior (e.g., \mathcal{N}(0,I) ), the latent space recovers the data-space density structure, improving the usefulness of s(z) without changing the prior itself. For a learnable prior (e.g., a Gaussian mixture model, GMM), the same density signal simultaneously (i) guides the encoder toward regions whose mass reflects the estimate and (ii) pulls the prior parameters toward those high-density regions, thereby improving agreement between prior mass and empirical structure. Concretely, we augment the ELBO with a robust, precision-weighted alignment loss (Section 2 ) that incurs negligible computational overhead (Table 12 ) and requires no changes to the encoder or decoder architectures. We implement this alignment in two forms: a direct variant that matches the model’s latent log-prior to the external log-density estimate, and a flow-corrected variant that learns a normalizing flow to account for the change-of-variables (Jacobian) between latent and data-derived projections. We show that on synthetic datasets (in which we know the ground truth), DiVAE (Sec. 3 ): (i) improves distributional alignment of latent log-densities to the ground truth log-density, shifting the mean toward the true scale; (ii) improves prior coverage, and (iii) yields better OOD uncertainty calibration—stronger separation, higher posterior entropy. These gains hold under both standard-normal and learnable GMM priors; flow attains the strongest separation, whereas direct provides the best calibration trade-off. In MNIST, where the ground truth is unknown, we find stable improvements in alignment between several types of prior and an external estimate of the density, with latent codes that are significantly most typical under the learned prior (see Table 2 ); for the OOD, when the prior is learnable, there are consistent improvements in adopting a density regularizer ( 4 ) . Related Work A major line of work improves VAE priors to better approximate the (theoretically optimal) aggregated posterior. Building on the observation that the aggregate is typically a gaussian mixture 18 , VampPrior introduces learnable pseudo-inputs so that the prior becomes a mixture of variational posteriors that explicitly targets this distribution 19 . Flow– and autoregressive–based priors further increase flexibility and improve likelihoods/compression 13 ; 9 ; 1 ; 4 ; 12 . In this work, rather than only enriching the prior, we directly align the density around data points –measured in the PCA-reduced space– with the latent density, enforcing local agreement between empirical structure and prior mass. 2 Methods Setup Let x\in\mathcal{X} be an input, q_{\phi}(z\mid x)=\mathcal{N}\!\big(\mu_{\phi}(x),\mathrm{diag}(\sigma_{\phi}^{2}(x))\big) is the encoder, p_{\theta}(x\mid z) the decoder, and p_{Z}(z)=p_{Z}(z;\psi) a prior (fixed or learnable) with parameters \psi (for a standard Gaussian prior, \psi=\varnothing ). We apply a dimensionality-reduction step used only for density estimation and for comparing densities between data and latent spaces. Specifically, we project the data from the original space \mathcal{X} onto the linear subspace \mathcal{U}\subseteq\mathcal{X} spanned by the top d principal components, where d=\dim(\mathcal{Z}) matches the latent dimensionality. Let P:\mathcal{X}\!\to\!\mathcal{U} denote the PCA projector; for each training point x_{i} , we write u_{i}=P\,x_{i} for its projection. We assume that we have access to a external (i.e., model-independent) log-density estimate \rho_{i} in the \mathcal{U} space of the PCA projected data, with a standard error \sigma_{i} expressing the estimator’s uncertainty at u_{i} . In practice, this estimate will be performed with standard techniques such as Kernel-Density Estimation (KDE), or Point Adaptive k-Nearest Neighbors (PAk) 16 ; 15 . We define the log-prior probability for x_{i} , expressing the log probability assigned by the prior to the encoding of a data point x_{i} , as s_{i}=\mathbb{E}_{z\sim q_{\phi}(\cdot\mid x_{i})}\!\big[\log p_{Z}(z)\big] , and estimate it with Monte Carlo sampling, where in practice we use one-sample estimates. The central idea of our work is to nudge the log-prior probabilities \left\{s_{i}\right\} towards \left\{\rho_{i}\right\} using a penalty, which we call the alignment loss . This loss comes in two flavours, depending on the aligner type: 1) a direct aligner and 2) a flow-corrected aligner. Direct aligner The direct aligner operates with a robust, precision-weighted loss term that acts as a regularizer, and compares the model’s prior log-densities \left\{s_{i}\right\} to the external log-densities \left\{\rho_{i}\right\} . We define precision weights from the estimator’s uncertainties w_{i}=\sigma_{i}^{-2}. Given a mini–batch \{x_{i}\}_{i=1}^{B} the direct alignment loss is written as: \mathcal{L}_{\text{direct-align}}=\sqrt{\frac{1}{B}\sum_{i=1}^{B}w_{i}\,\Gamma_{\delta}\!\big(s_{i}-\rho_{i}\big)}, (1) where \Gamma_{\delta} is the Huber loss. The Huber loss 5 behaves quadratically within a range \delta>0 , and linearly outside this range, to mitigate the influence of outliers (see D ). In our experiments we set \delta=1 , corresponding to \approx\frac{1}{2} probability for the residual to fall in the linear regime. Training objective We optimize the (negative) ELBO plus the alignment term \mathcal{L}=-\,\mathcal{L}_{\text{ELBO}}+\gamma_{t}\,\mathcal{L}_{\text{direct-align}} , with a warm-up weight \gamma_{t} , practically chosen as a linear warm-up from 0 to a 1 over training epochs. Unless stated otherwise, we do not detach encoder gradients in ( 1 ); this allows the regularizer to shape the variational posterior q_{\phi} . Flow-corrected aligner The direct aligner supervises the log-prior probability s(z)=\log p_{Z}(z) but, by itself, cannot account for the change of variables between the latent space \mathcal{Z} and \mathcal{U} , i.e., it omits the log-determinant Jacobian term. We therefore learn an invertible map f:\mathcal{Z}\!\to\!\mathcal{U} , parameterized as a normalizing flow 12 . The change of variables formula implies \log p_{U}(u)=\log p_{Z}\!\big(f^{-1}(u)\big)+\log\big|\det J_{f^{-1}}(u)\big|. We train f by maximum