Generalization in VAE and Diffusion Models: A Unified Information-Theoretic Analysis

Qi Chen, Jierui Zhu, & Florian Shkurti · 2025

Read Generalization in VAE and Diffusion Models: A Unified Information-Theoretic Analysis by Qi Chen, Jierui Zhu, & Florian Shkurti (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00849.txt.

Introduction

Qi Chen, Jierui Zhu, and Florian Shkurti present a machine learning theory paper, published in 2025, on why generative models work on data they never saw. Variational autoencoders and diffusion models both pair an encoder with a generator, yet earlier theory often covered only half of that pair. The authors build one information based framework that treats both halves as randomized maps and bounds the error of each.

For autoencoders the new analysis finally includes the generator's own generalization gap. For diffusion models it exposes a trade off governed by the total diffusion time, plus bounds computed from training data alone that can pick the best time setting. Tests on synthetic and real data supported the theory's predictions.

Catalog record

CREATORQi Chen, Jierui Zhu, & Florian Shkurti
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE103 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-00849

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Generalization in VAE and Diffusion Models: A Unified Information-Theoretic Analysis Qi Chen, Jierui Zhu, & Florian Shkurti Abstract Despite the empirical success of Diffusion Models (DMs) and Variational Autoencoders (VAEs), their generalization performance remains theoretically underexplored, especially lacking a full consideration of the shared encoder-generator structure. Leveraging recent information-theoretic tools, we propose a unified theoretical framework that provides guarantees for the generalization of both the encoder and generator by treating them as randomized mappings. This framework further enables (1) a refined analysis for VAEs, accounting for the generator’s generalization, which was previously overlooked; (2) illustrating an explicit trade-off in generalization terms for DMs that depends on the diffusion time T T ; and (3) providing computable bounds for DMs based solely on the training data, allowing the selection of the optimal T T and the integration of such bounds into the optimization process to improve model performance. Empirical results on both synthetic and real datasets illustrate the validity of the proposed theory. 1 Introduction Modeling complex data distributions with generative models has become a key focus in machine learning, driving transformative applications in various domains such as computer vision ( Rombach et al. 2022 ) , natural language processing ( Brown et al. 2020 ) , and scientific discovery ( Hoogeboom et al. 2022 ) . Over the past decade, this field has seen rapid advancement with the emerging of frameworks such as Variational Auto-Encoders (VAEs) ( Kingma & Welling 2013 ; Makhzani et al. 2015 ; Tolstikhin et al. 2017 ) , Generative Adversarial Networks (GANs) ( Goodfellow et al. 2014 ; Nowozin et al. 2016 ; Arjovsky et al. 2017 ) , Diffusion Models (DMs) ( Song & Ermon 2019 ; Song et al. 2020b ; Song et al. 2021 ) , as well as energy-based and auto-regressive models ( Larochelle & Murray 2011 ; Van den Oord et al. 2016 ) . Notably, deep latent diffusion models that combine VAEs and DMs have recently demonstrated exceptional success in generating high-resolution images ( Rombach et al. 2022 ) and videos ( Peebles & Xie 2023 ) . The empirical success and appeal of these models lie in their ability to generate new data rather than merely memorizing and reproducing training samples. However, recent studies have demonstrated memorization problems in GANs ( Feng et al. 2021 ; Meehan et al. 2020 ) , VAEs ( Van den Burg & Williams 2021 ) , DMs ( Carlini et al. 2023 ) , and auto-regressive Large Language Models (LLMs) ( Carlini et al. 2022 ) , raising significant privacy and copyright concerns. Consequently, understanding the generalization ability of generative models in producing diverse, novel samples becomes critical and urgent. While various generalization theories have been developed for GANs ( Arora et al. 2017 ; Biau et al. 2021 ; Mbacke et al. 2023 ; Yang & Weinan 2022 ; Ji et al. 2021 ) by adapting different theoretical tools originated in supervised learning ( e.g. , VC-dimension ( Vapnik et al. 1994 ) , Rademacher ( Bartlett & Mendelson 2002 ) , PAC-Bayes ( Shawe-Taylor & Williamson 1997 ; McAllester 1998 ) ), their counterparts for VAEs and DMs remain comparatively underexplored. In this paper, we aim to provide a novel generalization theory for VAEs and DMs. In comparison to previous works, our main contributions are as follows: 1. Unified information-theoretic framework. VAEs employ a probabilistic encoder-generator pair, while DMs can be viewed as an infinite-length composition of such encoder-generator sequences ( Tzen & Raginsky 2019 ; Huang et al. 2021 ) . We thereby model the encoder and generator as randomized mappings and develop a novel and unified theory with information-theoretic learning tools, avoiding traditional methods designed for deterministic mappings, which are unsuitable for our analysis. The general theoretical results are presented in Sec. 4 , where the proposed bounds are algorithm- and data-dependent under the sub-Gaussian assumption. 2. Improved analysis and tighter bounds for VAEs. To the best of our knowledge, we are the first to consider the generalization properties of both the encoder and generator in VAEs, whereas Chérief-Abdellatif et al. 2022 only consider guarantees for reconstruction loss and Mbacke et al. 2024 prove bounds for a fixed generator, ignoring its generalization. Moreover, compared to Mbacke et al. 2024 , we provide a tighter generalization bound for the encoder by directly bounding the generation error (defined in Sec. 3.2 ), removing the unnecessary Wasserstein-2 distance. 3. Computable bounds for diffusion models. We provide non-vacuous upper bounds for DMs to measure the divergence between the generated data and the original data, which can be tractably estimated, as detailed in Theorems 6.2 and 6.3 . Through these bounds, we show that: (a) There is an explicit trade-off between the generalization terms of both the encoder and generator that depends on the diffusion time T . As presented in Theorem 6.2 , when T approaches infinity, the encoder’s generalization term vanishes, while the generator’s term remains non-zero. Conversely, for small T , the encoder’s generalization term dominates. Empirical validation on both synthetic and real datasets verifies this phenomenon. Notably, this result implies that longer diffusion time does not necessarily lead to better generalization. To the best of our knowledge, this is the first explicit theoretical formulation of this trade-off in the context of generalization theory. (b) The proposed bound provides practical guidance for hyperparameter selection, where previous methods fall short due to the difficulty of accurately estimating the divergences between generated and test data, as shown in Fig. 3 . Additionally, the bound can be estimated using only training data, providing a practical and sample-efficient way to select the optimal diffusion time T or integrate the bounds into optimization for better model performance. 2 Related Work In this section, we only discuss the most relevant related works, while other related works on diffusion models and convergence theory are presented in Appendix I . Theories for VAEs. Some previous works ( Bozkurt et al. 2019 ; Huang et al. 2020 ; Bae et al. 2022 ) analyze VAEs with rate-distortion theory. Another approach involves deriving exact formulae under specific data distributions and high-dimensional limits. Assuming sample size m=\infty , Refinetti & Goldt 2022 examines the test error for nonlinear two-layer autoencoders as d\to\infty . Focusing on linear \beta -VAEs, Ichikawa & Hukushima 2024 analyzes generalization error with SGD dynamics, where fixed-point analysis reveals posterior collapse when \beta exceeds some threshold, suggesting appropriate KL annealing to accelerate convergence. Ichikawa & Hukushima 2023 uses the Replica method to derive asymptotic generalization error for \alpha=\frac{m}{d}=\Theta(1) , showing a peak in error at small \beta . This disappears after \beta exceeds some threshold, leading to posterior collapse regardless of the sample size. Husain et al. 2019 build a connection between GAN and WAE, where the generalization is analyzed based on the concentration result of Weed & Bach 2019 . Chérief-Abdellatif et al. 2022 applied PAC-Bayes theory to derive the generalization bound for the reconstruction loss. Recently, Mbacke et al. 2024 proved statistical guarantees with PAC-Bayes theory. However, their bounds only consider the generalization properties of the encoder. Instead, we provide tighter bounds for the encoder and use information-theoretic tools to derive generalization bounds for both the

Read the complete text