Deep Generative Models: Complexity, Dimensionality, and Approximation
Kevin Wang, Hongqian Niu, Yixin Wang, Didong Li, Department of Biostatistics, University of North Carolina at Chapel Hill, Department of Statistics, University of Michigan · 2025
Read Deep Generative Models: Complexity, Dimensionality, and Approximation by Kevin Wang, Hongqian Niu, Yixin Wang, Didong Li, Department of Biostatistics, University of North Carolina at Chapel Hill, Department of Statistics, University of Michigan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-00820.txt.
Introduction
Kevin Wang, Hongqian Niu, Yixin Wang, and Didong Li study deep generative models in a 2025 theory paper. Note that the catalog creator field also lists department names, so the byline in the text governs here. Such models learn complex data patterns and build new high-dimensional samples.
A common belief says the hidden input must be at least as large as the data's true shape dimension. The authors show that belief is wrong, drawing on space-filling curves. Inputs of any size can work, they prove. The cost is a sharp rise in network size, which reveals a trade-off among error, dimension, and model complexity.
Catalog record
| CREATOR | Kevin Wang, Hongqian Niu, Yixin Wang, Didong Li, Department of Biostatistics, University of North Carolina at Chapel Hill, Department of Statistics, University of Michigan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 85 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-00820
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Deep Generative Models: Complexity, Dimensionality, and Approximation Kevin Wang, Hongqian Niu, Yixin Wang, Didong Li, Department of Biostatistics, University of North Carolina at Chapel Hill, Department of Statistics, University of Michigan Abstract Generative networks have shown remarkable success in learning complex data distributions, particularly in generating high-dimensional data from lower-dimensional inputs. While this capability is well-documented empirically, its theoretical underpinning remains unclear. One common theoretical explanation appeals to the widely accepted manifold hypothesis, which suggests that many real-world datasets, such as images and signals, often possess intrinsic low-dimensional geometric structures. Under this manifold hypothesis, it is widely believed that to approximate a distribution on a d d -dimensional Riemannian manifold, the latent dimension needs to be at least d d or d + 1 d+1 . In this work, we show that this requirement on the latent dimension is not necessary by demonstrating that generative networks can approximate distributions on d d -dimensional Riemannian manifolds from inputs of any arbitrary dimension, even lower than d d , taking inspiration from the concept of space-filling curves. This approach, in turn, leads to a super-exponential complexity bound of the deep neural networks through expanded neurons. Our findings thus challenge the conventional belief on the relationship between input dimensionality and the ability of generative networks to model data distributions. This novel insight not only corroborates the practical effectiveness of generative networks in handling complex data structures, but also underscores a critical trade-off between approximation error, dimensionality, and model complexity. Keywords : Approximation theory, generative AI, manifold hypothesis, space-filling curve. 1 Introduction Generative models, such as generative adversarial networks (GANs, Goodfellow et al. 2014 ) and variational auto-encoders (VAEs, Kingma and Welling 2013 ), have become a central topic in machine learning. These models are versatile, addressing various problems ranging from image generation ( Bao et al. 2017 ; Han et al. 2018 ) , style transfer ( Karras et al. 2019 ) , anomaly detection ( Xia et al. 2022 ) , to data augmentation ( Tran et al. 2021 ) , achieving increasingly accurate results across disciplines. Despite these advances, GANs, in particular, face challenges such as mode collapse, vanishing gradients, and training instability, particularly when distributions are not continuous or have disjoint supports ( Arjovsky and Bottou 2017 ) . While traditional GANs and VAEs primarily relied on the Kullback–Leibler divergence (KL) to measure distances between input and output distributions, more recent approaches have greatly improved training performance. The shift from KL to Wasserstein distance ( Villani et al. 2009 ) is a notable example, as exemplified in the Wasserstein GAN (WGAN, Arjovsky et al. 2017 ), and further extended to Wasserstein VAE (WVAE, Ambrogioni et al. 2018 ) and Wasserstein Auto Encoders ( Tolstikhin et al. 2017 ) . Beyond this, there have been substantial enhancements in model architecture, loss functions, and regularization techniques. Advances such as conditional GANs (cGAN, Mirza and Osindero 2014 ) for targeted image generation, auxiliary classifier GANs (AC-GANs, Odena et al. 2017 ), and self-attention GANs (SA-GANs, Zhang et al. 2019 ) have enriched the versatility and effectiveness of GANs. Similarly, VAEs have seen improvements with techniques such as hierarchical latent variables ( Vahdat and Kautz 2020 ; Sønderby et al. 2016 ) and incorporation of normalizing flows ( Kingma et al. 2016 ) , refining their ability to model complex distributions. These developments collectively contribute to the generation of high-quality, high-dimensional data from simpler, lower-dimensional distributions, with notable impact in fields such as photorealistic image generation ( Wang et al. 2018 ; Sarkar et al. 2021 ) . The theoretical advancements in deep generative models, paralleling their practical applications, have been substantial. In particular, the depth and width of network architectures in these models have been theoretically shown to significantly impact their approximation capabilities, as detailed in the works focused on neural network expressiveness ( Eldan and Shamir 2016 ) . Furthermore, the challenges of mode collapse in GANs and the propensity of VAEs to produce overly smooth outputs have catalyzed research into improving their architectural designs and training methodologies ( Goodfellow et al. 2014 ; Kingma and Welling 2013 ) . Collectively, these theoretical advancements have not only deepened our understanding of the mechanisms behind deep generative models but also guided the development of more refined and capable generative architectures. However, the theoretical investigation of the manifold hypothesis in deep generative models, particularly its implications in approximation theory, remains less developed. The manifold hypothesis, widely accepted as a crucial reason behind the success of these models, posits that real-world high-dimensional data often reside on lower-dimensional manifolds ( Bronstein et al. 2017 ) . This concept is critical because it suggests a fundamental reason why models such as GANs and VAEs, which operate in low-dimensional spaces, are able to capture complex data distributions effectively. Although empirical evidence supports this hypothesis, a deeper theoretical understanding of how these models approximate and learn distributions on lower-dimensional manifolds within high-dimensional spaces is crucial. Further theoretical work in this area is essential, not only to validate the manifold hypothesis but also to enhance the design and efficiency of generative models in handling complex, high-dimensional data. Building on the previous discussion of the manifold hypothesis, Dahal et al. 2022 offers a noteworthy theoretical contribution in the field of deep generative models. Their work effectively addresses fundamental questions in approximation theory, demonstrating that distributions on a d -dimensional Riemannian manifold can be approximated by a deep generative network through the pushforward measure of an easily sampled distribution in d+1 dimensions, such as a uniform distribution in a cube [0,1]^{d+1} . Importantly, they provide an upper bound on the complexity of these models, revealing that the required number of layers and neurons is determined by both the approximation error and the manifold dimension d . This insight offers a direct theoretical justification for the models’ abilities to learn diverse probability distributions and suggesting avenues for efficiency improvements in practical applications. However, the research by Dahal et al. 2022 also brings to light some important, yet unanswered questions. Primarily, the necessity of using a d+1 dimensional input space to approximate a d dimensional manifold warrants further investigation. This approach, integral for assembling local patches, raises the possibility of achieving similar outcomes with an input dimension that matches d . Furthermore, in real-world applications where d is not readily known, the choice of input dimension becomes a critical decision. If the chosen dimensionality is either greater or lesser than the actual d , how would it impact the model’s performance and efficiency? Answering these questions is crucial for advancing the development of deep generative models, potentially leading to more adaptable and efficient solutions for complex data representation. In this work, we aim to understand how the input dimension relates to the manifold’s intrinsic dimension in deep generative models. We adapt the space-filling curve theory to demonstrate a novel aspect of deep generative models: their ability to learn a target data distribution from easy-to-sample distributions of arbitrary dimensions. This includes the capacity to approximate these distributions even from an input as low as a one-dimensional uniform distribution on the unit interval [0,1] (see Figure 1 for examples of curves filling 2-dimensional manifolds). Intuitively, if