Quasi Monte Carlo methods enable extremely low-dimensional deep generative models
Miles Martinez, Alex H. Williams · 2026
Read Quasi Monte Carlo methods enable extremely low-dimensional deep generative models by Miles Martinez, Alex H. Williams (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-18676.txt.
Introduction
Miles Martinez and Alex Williams build deep learn tools that find a very low, clear code for rich data sets. Most such tools learn a code by a guess-and-check path that needs a trained guide net. That path can blur the code and make it hard to read. Their plan skips the guide net. It works out the true score of each code by a near-dice sum with a smart twist.
The twist uses points spread with more care than pure chance, so the sum is firm with fewer points. With a firm score in hand, the tool can hunt for codes that are both small and true. A small, clear code for a rich set could show its deep form at a glance. That would help in fields from gene maps to shot sets.
Catalog record
| CREATOR | Miles Martinez, Alex H. Williams |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-18676
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Quasi Monte Carlo methods enable extremely low-dimensional deep generative models Miles Martinez, Alex H. Williams Abstract This paper introduces quasi-Monte Carlo latent variable models (QLVMs): a class of deep generative models that are specialized for finding extremely low-dimensional and interpretable embeddings of high-dimensional datasets. Unlike standard approaches, which rely on a learned encoder and variational lower bounds, QLVMs directly approximate the marginal likelihood by randomized quasi-Monte Carlo integration. While this brute force approach has drawbacks in higher-dimensional spaces, we find that it excels in fitting one, two, and three dimensional deep latent variable models. Empirical results on a range of datasets show that QLVMs consistently outperform conventional variational autoencoders (VAEs) and importance weighted autoencoders (IWAEs) with matched latent dimensionality. The resulting embeddings enable transparent visualization and post hoc analyses such as nonparametric density estimation, clustering, and geodesic path computation, which are nontrivial to validate in higher-dimensional spaces. While our approach is compute-intensive and struggles to generate fine-scale details in complex datasets, it offers a compelling solution for applications prioritizing interpretability and latent space analysis. 1 Introduction Deep generative models are a mainstay of contemporary machine learning. In addition to synthesizing complex images, sounds, and sentences, many researchers have sought to use these models to identify human-interpretable representations of complex data distributions. For example, variational autoencoders (VAEs) have been applied in scientific analysis of motor kinematics ( 35 ) , vocal communication patterns ( 18 ) , single cell gene expression ( 15 ; 20 ; 60 ) , neural population dynamics ( 45 ) , and extracellular spike waveforms ( 4 ) . Reducing dimensionality is seen as a key step towards interpretability in these models. VAEs achieve this by introducing a bottleneck layer. The size of this layer is a hyperparameter and often it is chosen to be modestly large. For VAEs, typical values are in the range of 32-128 latent dimensions and reconstruction quality is reported to drop off for networks with fewer than ~ 10 dimensions ( 37 ) . As a result, it is virtually universal practice to employ further dimensionality reduction methods like UMAP or t-SNE to visualize the latent space of a VAE. Many papers also apply analyses such as clustering to identify human-interpretable structure in the latent space ( 18 ; 35 ; 46 ; 30 , see e.g.,) . Even in moderately high dimensions, these post hoc analyses are non-trivial to tune and validate. Here, we demonstrate that in many cases two or three dimensional latent spaces are sufficient for performant deep generative models. Contrary to conventional wisdom ( 16 , see, e.g.,) , we find that numerical integration can be a viable strategy to marginalize over the latent variables. We use randomized lattice integration rules ( 14 ) to approximate the marginal likelihood. We call the resulting model a quasi-Monte Carlo latent variable model , or QLVM. Although our method requires multiple passes through the decoder on each training iteration, it completely eliminates variational approximations that can be challenging to tune and optimize (see 47 ; 22 ; 13 ; 29 , and fig. 3 in this paper) . Ultimately, we find that this approach decisively outperforms VAEs with comparable architectures and can even rival the performance of VAEs with higher capacity latent spaces. Despite the limitations of our approach (see § 4.2 ), there are many advantages to learning extremely low-dimensional embeddings of high-dimensional datasets. We can, for example, use kernel density estimators—which suffer a curse of dimensionality in higher dimensions—to precisely characterize the aggregate posterior density over the latent space. We can also exhaustively characterize the smoothness of the embedding by querying the Jacobian of the decoder network over a dense grid of points. We demonstrate how this information can be used to aid clustering analyses in § 3.3 , but the potential applications extend much more broadly to other topological analyses ( 58 ) . Altogether, our results highlight a simple, but widely overlooked, approach to building extremely low-dimensional deep latent variable models for exploratory analysis. 2 Methodology 2.1 Background on Deep Latent Variable Models Let \boldsymbol{x}_{i}\in\mathbb{R}^{D} for i\in\{1,\dots,n\} denote a sequence of high-dimensional, observed data points. Deep latent variable models seek to explain such datasets through the following probabilistic model: \boldsymbol{x}_{i}\sim p_{\theta}(\boldsymbol{x}_{i}~|~\boldsymbol{z}_{i})\qquad\text{where}\qquad\boldsymbol{z}_{i}\sim p(\boldsymbol{z}) (1) independently for i\in\{1,\dots,n\} . Here, a d -dimensional latent variable vector \boldsymbol{z}_{i} is drawn from a prior distribution p(\boldsymbol{z}) . 1 1 1 Following common practice in this literature, we use the same notation for both the probability distribution of a random variable and its associated density function. We also refer to the two interchangeably. The likelihood of each observation is parameterized by a deep network, called the decoder , with trainable weights \theta . We use f_{\theta}(\boldsymbol{z}_{i})=\mathbb{E}[\boldsymbol{x}_{i}~|~\boldsymbol{z}_{i}] to denote the mean output of the decoder. Ideally, the decoder weights would be trained to maximize the marginal likelihood: p_{\theta}(\boldsymbol{x}_{1},\dots,\boldsymbol{x}_{n})=\prod_{i=1}^{n}p_{\theta}(\boldsymbol{x}_{i})=\prod_{i=1}^{n}\int p_{\theta}(\boldsymbol{x}_{i}~|~\boldsymbol{z}_{i})p(\boldsymbol{z}_{i})\textrm{d}\boldsymbol{z}_{i} (2) Computing the marginal likelihood of a datapoint requires integrating over all possible values of \boldsymbol{z}_{i} . This integration is challenging in high dimensions and the simple-minded Monte Carlo estimate, p_{\theta}(\boldsymbol{x}_{i})=\int p_{\theta}(\boldsymbol{x}_{i}~|~\boldsymbol{z}_{i})p(\boldsymbol{z}_{i})\text{d}\boldsymbol{z}_{i}\approx\frac{1}{m}\sum_{j=1}^{m}p_{\theta}(\boldsymbol{x}_{i}~|~\boldsymbol{z}_{i}\!=\!\widetilde{\boldsymbol{z}}_{j}) (3) where \widetilde{\boldsymbol{z}}_{1},\dots,\widetilde{\boldsymbol{z}}_{m} are independent samples from p(\boldsymbol{z}) , is therefore typically dismissed ( 16 ; 39 ; 27 ) . However, the quality of this approximation only deteriorates in proportion to the variance of p_{\theta}(\boldsymbol{x}~|~\boldsymbol{z}) for \boldsymbol{z}\sim p(\boldsymbol{z}) ( 43 , see) . Thus, for sufficiently simple datasets, it is plausible that eq. 3 may suffice as a criterion to train \theta . To our knowledge, this possibility has not been seriously explored in deep latent variable models. Since computing p_{\theta}(\boldsymbol{x}_{i}) is often difficult, Variational Autoencoders (VAEs; 28 ) instead target a lower bound on \log p_{\theta}(\boldsymbol{x}_{i}) . To do this, VAEs introduce a second deep neural network with parameters \phi called the encoder . For each datapoint the encoder outputs a conditional distribution q_{\phi}(\boldsymbol{z}~|~\boldsymbol{x}_{i}) , which acts as a variational approximation to the posterior distribution of \boldsymbol{z}_{i} given \boldsymbol{x}_{i} . As a training objective, one considers: \mathcal{L}_{\textrm{ELBO}}^{(i)}(\theta,\phi)=\frac{1}{m}\sum_{j=1}^{m}\log p_{\theta}(\boldsymbol{x}_{i}~|~\widetilde{\boldsymbol{z}}_{j},\theta)-\textrm{D}\!\left(q_{\phi}(\boldsymbol{z}~|~\boldsymbol{x}_{i})\;\|\;p(\boldsymbol{z})\right) (4) where \widetilde{\boldsymbol{z}}_{1},\dots,\widetilde{\boldsymbol{z}}_{m} are independent samples from q_{\phi}(\boldsymbol{z}~|~\boldsymbol{x}_{i}) and \textrm{D}\!\left(\cdot\!\;\|\;\!\cdot\right) denotes the Kullback-Leibler (KL) divergence. In the limit that m\!\rightarrow\!\infty , this converges to a lower bound on \log p_{\theta}(\boldsymbol{x}_{i}) called the evidence lower bound (ELBO). Typically, m=1 sample is used to obtain an unbiased estimate of the lower bound. For further background see 16 ; 27 . 2.2 Quasi-Monte Carlo Latent Variable Models (QLVMs) VAEs target a lower bound on the marginal log likelihood, which is only tight when q_{\phi}(\boldsymbol{z}~|~\boldsymbol{x}_{i}) matches the true posterior distribution of \boldsymbol{z} given \boldsymbol{x}_{i} . Commonly, q_{\phi} is assumed to follow a simple form—e.g. Gaussian with diagonal covariance. In very low-dimensional latent spaces, we will see that this assumption can be violated and, even if it approximately holds, the true posterior can be so narrow that it is difficult for the encoder network to learn (see fig. 3 ). We therefore sought to understand whether the simple Monte Carlo approach shown in eq. 3 can be practical in situations of interest. By taking logarithms on both sides of eq. 3 , we obtain: \mathcal{L}_{\textrm{MC}}^{(i)}(\theta)=\log p_{\theta}(\boldsymbol{x}_{i})=\log\left[\frac{1}{m}\sum_{j=1}^{m}p_{\theta}(\boldsymbol{x}_{i}~|~\widetilde{\boldsymbol{z}}_{j})\right] (5) where \boldsymbol{x}_{i} is any datapoint and \widetilde{\boldsymbol{z}}_{1},\dots,\widetilde{\boldsymbol{z}}_{m} are sampled points in the latent space. This is a lower bound on the marginal log likelihood in expectation. Indeed, by Jensen’s