Enhancing Variational Autoencoders with Smooth Robust Latent Encoding
Hyomin Lee, Minseon Kim, Sangwon Jang, Jongheon Jeong, Sung Ju Hwang · 2025
Read Enhancing Variational Autoencoders with Smooth Robust Latent Encoding by Hyomin Lee, Minseon Kim, Sangwon Jang, Jongheon Jeong, Sung Ju Hwang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-17219.txt.
Introduction
Hyomin Lee and four co-authors strengthen the autoencoders that sit inside image generators like Stable Diffusion. Their paper appeared in two thousand twenty-five. Variational autoencoders compress images into a latent code and rebuild them. Their robustness has been little studied. Adversarial training boosts robustness in classifiers, but builders feared it would dull generation quality. The team challenges that worry.
Their SRL-VAE framework attacks the latent code with crafted perturbations during training. That pressure smooths the latent space itself. Smoother codes generalize better to odd inputs. A regularizer keeps the code faithful to the original image. The result improves both generation quality and robustness at once. The trade-off the field assumed turns out to be avoidable.
Catalog record
| CREATOR | Hyomin Lee, Minseon Kim, Sangwon Jang, Jongheon Jeong, Sung Ju Hwang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-17219
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Enhancing Variational Autoencoders with Smooth Robust Latent Encoding Hyomin Lee, Minseon Kim, Sangwon Jang, Jongheon Jeong, Sung Ju Hwang Abstract Variational Autoencoders (VAEs) have played a key role in scaling up diffusion-based generative models, as in Stable Diffusion, yet questions regarding their robustness remain largely underexplored. Although adversarial training has been an established technique for enhancing robustness in predictive models, it has been overlooked for generative models due to concerns about potential fidelity degradation by the nature of trade-offs between performance and robustness. In this work, we challenge this presumption, introducing Smooth Robust Latent VAE (SRL-VAE), a novel adversarial training framework that boosts both generation quality and robustness. In contrast to conventional adversarial training, which focuses on robustness only, our approach smooths the latent space via adversarial perturbations, promoting more generalizable representations while regularizing with originality representation to sustain original fidelity. Applied as a post-training step on pre-trained VAEs, SRL-VAE improves image robustness and fidelity with minimal computational overhead. Experiments show that SRL-VAE improves both generation quality, in image reconstruction and text-guided image editing, and robustness, against Nightshade attacks and image editing attacks. These results establish a new paradigm, showing that adversarial training, once thought to be detrimental to generative models, can instead enhance both fidelity and robustness. 1 Introduction Variational Autoencoders (VAEs) [ 12 ] have been employed as a compressor in the success of latent generative models [ 24 , 22 , 19 , 15 ] , which have demonstrated surprising capabilities in generating high-quality images. The VAEs compress high-dimensional images into a latent space that retains semantic and structural information, which is continuous [ 24 ] or discrete [ 4 ] space for high-quality generative modeling. Despite their effectiveness as a compressor in generative models, prior work has largely overlooked the representational role of VAEs, primarily focusing on generative aspects to improve performance by proposing architectures [ 22 ] , training objectives [ 37 , 19 ] , or regularization [ 38 ] . However, obtaining an effective compressor is one of the key components to achieving higher-quality generations while also ensuring robustness with efficient computational costs. (a) Diffusion-based generative process of clean image (b) Diffusion-based generative process of perturbed image Figure 1 : Concept figure of SRL-VAE. Compared to SD-VAE, SRL-VAE maintains similar representations for clean examples while achieving robust representation against perturbed examples. To obtain effective VAEs for both higher fidelity and better robustness, representation space of VAEs needs to be robust and capable of capturing better structural latent. We were inspired by adversarial training [ 20 , 39 ] to build robust representations. Adversarial training [ 20 , 39 ] has initially been recognized as an effective method for improving adversarial robustness in predictive models against adversarial attacks, particularly in classification tasks. In fact, adversarial training builds robust and smooth representations [ 14 ] so that it could have better generalization [ 36 , 10 ] from leveraging a min-max formulation. However, adversarial training in generative modeling remains largely unexplored, partly because generative models have traditionally prioritized fidelity and diversity over robustness. Moreover, adversarial training is challenging to optimize and often leads to large performance degradation where trade-offs are clear [ 39 , 2 ] . As a result, the adoption of adversarial training in VAEs and similar generative models has been limited, as it is frequently perceived as a significant trade-off. Figure 2 : Training objective of Smooth Robust Latent Variational Autoencoders (ours). A novel adversarial training approach in the latent space of VAE with originality regularization. In this work, we introduce a novel adversarial training framework for VAEs that enhances both generation quality and robustness by constructing smooth latent space. Unlike conventional adversarial training, which primarily targets predictive robustness, our approach leverages adversarial perturbations to smooth the latent space and promote more generalizable representations of VAEs (Figure 1 ). Our approach consists of two key steps: (1) maximizing the VAE loss to introduce adversarial perturbations in the latent space, exposing the model to challenging variations, and (2) minimizing both the VAE loss and an originality loss to preserve the original representation structure while building smooth latent space, ensuring a stable training and robustness as shown in Figure 2 . Furthermore, our approach is applied as a post-training step on pre-trained VAEs, requiring only a small amount of additional computational resources, making it an efficient and practical solution for improving generative models. By bridging adversarial training and generative modeling, our work introduces a new perspective on the importance of obtaining robust, high-quality representations of VAEs in generative models. Our method enables VAEs to generate outputs of comparable or, in some cases, even higher quality (Figure 1(a) ) while being extremely effective against various types of adversarial attacks during the generation process (Figure 1(b) ). This demonstrates that adversarial training is not merely a defensive mechanism that sacrifices performance but rather a powerful strategy for enhancing generative models with robustness, paving the way for future advancements in robust generative learning. The main contributions can be summarized as follows: • Unlike prior adversarial training methods, which predominantly focus on classification models, we introduce the first adversarial training approach tailored for VAEs, demonstrating its ability to improve both generation quality and robustness simultaneously. • We show that adversarial training , when combined with an originality loss, fosters a smoother latent space, leading to more stable and generalizable representations, which enhance image fidelity and deliver surprisingly strong performance against various types of attacks. • Through extensive experiments, we demonstrate significant improvements in both fidelity and robustness across multiple tasks. Specifically, we evaluate image quality through image reconstruction and generation, and assess robustness by evaluation against adversarial attacks on text-guided image editing and adversarial poisoning attacks, establishing the effectiveness of our approach in enhancing both generative performance and robustness. 2 Related Works Latent Generative Models Variational Autoencoders (VAEs) [ 12 ] are generative models that learn compact latent representations by regularizing the latent distribution through a Kullback–Leibler (KL) divergence term. While VAEs enable smooth interpolation and efficient sampling, they often produce blurry outputs due to limitations in the latent space. To address this, Vector Quantized VAE (VQ-VAE) [ 33 ] introduces a discrete codebook of embeddings, preventing latent space collapse and improving reconstruction quality. VQ-GAN [ 4 ] further enhances this framework with adversarial training, guiding the decoder towards sharper and more realistic outputs. Building on these approaches, Latent Diffusion Models (LDMs) [ 24 ] apply diffusion processes in a compressed latent space obtained from a pretrained autoencoder, significantly reducing computational costs while preserving high fidelity. Cross-attention mechanisms enable flexible conditional generation from various inputs, as demonstrated in Stable Diffusion [ 24 ] . Recent work also explores enhancing the interaction between autoencoders and diffusion models, addressing spectral properties of latent spaces [ 31 ] and introducing equivariance regularization for better generative performance [ 13 ] . However, prior works have largely overlooked the quality of latent representations in VAEs in terms of fidelity and robustness. We address this by applying adversarial training to enhance the latent space, improving both generation quality and robustness. Adversarial Training Szegedy et al. [32] first revealed the vulnerability of deep neural networks (DNNs) to adversarial attacks, showing that imperceptible perturbations could mislead