Distillation of a tractable model from the VQ-VAE
Armin Hadžić, Milan Papež, Tomáš Pevný · 2025
Read Distillation of a tractable model from the VQ-VAE by Armin Hadžić, Milan Papež, Tomáš Pevný (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-01400.txt.
Introduction
The VQ-VAE is a generative model that stores data in a discrete latent space, giving strong generation but inference that is widely seen as intractable, too hard to compute exactly. Armin Hadzic, Milan Papez, and Tomas Pevny challenge that view. They show the model can be distilled, compressed, into a tractable form by keeping only latent variables with high probability.
The distilled model is framed as a probabilistic circuit, a structure built for exact inference. It keeps the expressiveness of the original while allowing probabilities to be computed directly. The simple strategy works especially well because VQ-VAEs often underuse their latent space, leaving much of it safe to drop. Experiments show competitive results in density estimation and conditional generation, suggesting a large intractable model can hide a smaller exact one inside it.
Catalog record
| CREATOR | Armin Hadžić, Milan Papež, Tomáš Pevný |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-01400
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Distillation of a tractable model from the VQ-VAE Armin Hadžić, Milan Papež, Tomáš Pevný Abstract Deep generative models with discrete latent space, such as the Vector-Quantized Variational Autoencoder (VQ-VAE), offer excellent data generation capabilities, but, due to the large size of their latent space, their probabilistic inference is deemed intractable. We demonstrate that the VQ-VAE can be distilled into a tractable model by selecting a subset of latent variables with high probabilities. This simple strategy is particularly efficient, especially if the VQ-VAE underutilizes its latent space, which is, indeed, very often the case. We frame the distilled model as a probabilistic circuit, and show that it preserves expressiveness of the VQ-VAE while providing tractable probabilistic inference. Experiments illustrate competitive performance in density estimation and conditional generation tasks, challenging the view of the VQ-VAE as an inherently intractable model. 1 Introduction 10^{0} 10^{1} 10^{2} 10^{3} 10^{4} 10^{5} 10^{6} 10^{7} 10^{8} 0 0.2 0.4 0.6 0.8 1 0.9 0.99 Rank of \mathbf{z}_{(i)} in decreasing p(\mathbf{z}) F(i) PixelCNN prior Uniform prior Figure 1: Cumulative distribution functions (CDFs) of different prior distributions, p(\mathbf{z}) , in the VQ-VAE. The CDF is defined as F(i)=\sum_{j=1}^{i}p(\mathbf{z}_{(j)}) , where \mathbf{z}_{(j)} is the j\mathrm{\penalty\ th} latent variable in a decreasing order which is obtained based on p(\mathbf{z}) values. The VQ-VAE’s latent space size is |\mathcal{Z}|=96^{4}\approx 85\mathrm{M} . We repeated the experiment with five independently trained models. The solid line is the mean and the shaded area is the \pm 1 standard deviation. The blue and green colors correspond to the PixelCNN and the uniform prior, respectively. The red vertical lines are sizes of the latent space at which F reaches 90\% and 99\% of its total mass. Specifically, these lines correspond to approximately 5\% and 25\% of the latent space. This result shows that only a small fraction of the vast VQ-VAE’s latent space is actually utilized. Deep generative models provide a framework for learning complex patterns across diverse data modalities, such as images, language, and graphs [ Alaniz et al. 2022 , Tian et al. 2024 , Bachmann et al. 2024 ] . They excel at generating new samples; however, they cannot answer even the most basic inference tasks without approximations, making them intractable probabilistic models . This issue arises because deep neural networks make analytical integration, which is crucial for many inference tasks, infeasible [ Nguyen and Goulet 2021 , Rezende et al. 2014 ] . Probabilistic circuits (PCs) [ Choi et al. 2020 , Peharz et al. 2020 , Vergari et al. 2015 ] are tractable probabilistic models that provide closed-form solutions to a wide range of inference tasks, including marginalization, conditioning, and expectation. Recent work has begun exploring hybrid frameworks that combine the efficient inference of PCs with the expressive power of deep neural networks [ Correia et al. 2023 , Liu et al. 2023 , Sidheekh et al. 2023 ] . This paper extends this line of work by investigating the tractability of the vector-quantized variational autoencoder (VQ-VAE). The VQ-VAE [ van den Oord et al. 2017 ] is a deep generative model that compresses input data into a discrete latent representation, while preserving structural patterns (e.g., spatial coherence in images [ van den Oord et al. 2017 ] , temporal consistency in audio and video [ Dhariwal et al. 2020 , Zeghidour et al. 2021 , Yan et al. 2021 ] ). However, the latent space of the VQ-VAE is exponentially large, making probabilistic inference tasks intractable [ Correia et al. 2020 , Zhao et al. 2016 , Peharz et al. 2016 ] . Moreover, the vanilla VQ-VAE is known to suffer from the index collapse issue, which manifests as a poor utilization of the latent space. This underutilization means that only a (relatively) small collection of latent states contributes to the model outputs (e.g., its likelihood) [ Guo et al. 2024 , Huh et al. 2023 ] . Various techniques have been explored to mitigate this problem, namely replacement policies [ Zeghidour et al. 2021 , Dhariwal et al. 2020 ] , codebook resets [ Łańcucki et al. 2020 , Zeghidour et al. 2021 , Dhariwal et al. 2020 ] , stochastically-quantized VAE [ Takida et al. 2022 , Roy et al. 2018 ] , exponential moving average [ van den Oord et al. 2017 ] , and product-quantized VAE [ Guo et al. 2024 ] . Notwithstanding that, we advocate that the latent space underutilization, caused by the index collapse, is actually beneficial for the tractability of the VQ-VAE, i.e., the limited use of the latent space makes many inference tasks tractable. We propose a novel approach to transform the VQ-VAE into a tractable mixture model (i.e., a PC) by distilling a subset of the most relevant latent variables. The subset is identified using two complementary approaches: (i) random sampling, which requires enumeration of all latent variables, which is exhaustive and computationally expensive; and (ii) a beam search, which trades off optimality for computational efficiency. We demonstrate that the distilled model based on the beam search delivers a competitive performance to other tractable probabilistic models in the context of learning a probability distribution of images. 2 Background Discrete latent variable models. A complete probabilistic description of a discrete latent variable model is given by the following marginal probability distribution: p(\mathbf{x})=\sum_{\mathbf{z}\in\mathcal{Z}}p(\mathbf{x}|\mathbf{z})p(\mathbf{z}),\vskip-4.0pt (1) where \mathbf{x}\coloneqq\{x_{1},\ldots,x_{S}\}\in\mathcal{X} denotes observations with S elements (e.g., image pixels), and \mathbf{z}\coloneqq\{z_{1},\ldots,z_{M}\}\in\mathcal{Z} are discrete latent variables. The prior p(\mathbf{z}) is a probability mass function over discrete latent variables, describing the significance of each value of \mathbf{z} . The conditional distribution p(\mathbf{x}|\mathbf{z}) models \mathbf{x} conditionally on a different set of parameters depending on \mathbf{z} , i.e., p(\mathbf{x}|\mathbf{z})\coloneqq p(\mathbf{x}|\theta_{\mathbf{z}}) . The latent space, i.e., the set of all latent variable configurations, is a finite structured set with exponential complexity [ Correia et al. 2020 , Zhao et al. 2016 , Peharz et al. 2016 ] . Probabilistic circuits. A probabilistic circuit is a directed, acyclic, parametrized computational graph encoding a non-negative function over observations, p(\mathbf{x}) [ Vergari et al. 2019 , Choi et al. 2020 ] . The graph composes of three types of computational units : input , product , and sum . Sum and product units receive the outputs of other units as inputs. We denote the set of inputs of a unit u as \mathrm{in}(u) . Each unit u encodes a function p_{u} over a subset of random variables \mathbf{x}_{u}\subseteq\mathbf{x} , referred to as scope . The input unit p_{u}(\mathbf{x}_{u}) computes a pre-defined, parametrized probability distribution. The sum unit computes the weighted sum of its inputs p_{u}(\mathbf{x}_{u})\coloneqq\sum_{i\in\mathrm{in}(u)}w_{i}p_{i}(\mathbf{x}_{i}) , where w_{i}\in\mathbb{R} are the weight parameters, and the product unit computes the product of its inputs, p_{u}(\mathbf{x}_{u})\coloneqq\prod_{i\in\mathrm{in}(n)}p_{i}(\mathbf{x}_{i}) . The scope of any sum or product unit is the union of its input scopes, \mathbf{x}_{u}=\bigcup_{i\in\mathrm{in}(u)}\mathbf{x}_{i} [ Vergari et al. 2019 , Choi et al. 2020 ] . To achieve tractable inference, the children of each sum unit have to be defined over the same scopes (smoothness), and the children of each product unit have to be defined over pairwise disjoint scopes (decomposability). Additionally, the input units have to be tractable probability distributions (e.g., a member of an exponential family) [ Darwiche and Marquis 2002 ] . PCs are an example of the