Scalable GANs with Transformers
Sangeek Hyun, MinKyu Lee, Jae-Pil Heo · 2025
Read Scalable GANs with Transformers by Sangeek Hyun, MinKyu Lee, Jae-Pil Heo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-24935.txt.
Introduction
Sangeek Hyun, MinKyu Lee, and Jae-Pil Heo ask whether adversarial image generators scale as gracefully as other generative models, and they rebuild the design around two proven choices. Their generator and discriminator are pure transformers, and training happens inside the compressed latent space of a variational autoencoder, where computation is cheaper and image quality survives. Writing in twenty twenty-five, they first diagnose what breaks when the design is naively enlarged: early generator layers sit underused, and optimization grows unstable at larger widths.
Two light fixes follow: extra supervision partway through the generator, and learning rates adjusted for width. With them, models from small to extra-large trained reliably. The largest reached state-of-the-art one-step class-conditional scores on the standard ImageNet test in far fewer epochs than strong baselines, by the authors report. A project page hosts examples.
Catalog record
| CREATOR | Sangeek Hyun, MinKyu Lee, Jae-Pil Heo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-24935
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Scalable GANs with Transformers Sangeek Hyun, MinKyu Lee, Jae-Pil Heo Abstract Scalability has driven recent advances in generative modeling, yet its principles remain underexplored for adversarial learning. We investigate the scalability of Generative Adversarial Networks (GANs) through two design choices that have proven to be effective in other types of generative models: training in a compact Variational Autoencoder latent space and adopting purely transformer-based generators and discriminators. Training in latent space enables efficient computation while preserving perceptual fidelity, and this efficiency pairs naturally with plain transformers, whose performance scales with computational budget. Building on these choices, we analyze failure modes that emerge when naively scaling GANs. Specifically, we find issues as underutilization of early layers in the generator and optimization instability as the network scales. Accordingly, we provide simple and scale-friendly solutions as lightweight intermediate supervision and width-aware learning-rate adjustment. Our experiments show that GAT, a purely transformer-based and latent-space GANs, can be easily trained reliably across a wide range of capacities (S through XL). Moreover, GAT-XL/2 achieves state-of-the-art single-step, class-conditional generation performance (FID of 2.96) on ImageNet-256 in just 40 epochs, 6 × \times fewer epochs than strong baselines. 1 Introduction Figure 1: Curated examples of GAT-XL/2 on ImageNet-256. GAT-XL/2 exhibits strong generation capability (FID 2.96) within 40 epochs, 6 \times fewer than 1-NFE baselines (FID 3.43), while keeping the characteristics of GANs such as latent interpolation (bottom two rows). Recent breakthroughs in generative modeling have become a central driver of progress across core areas of computer vision. These developments have accelerated in recent years, enabling capabilities that were previously out of reach: state-of-the-art systems now support text-to-image ( Rombach et al. 2022 ; Podell et al. 2023 ; Esser et al. 2024 ; Han et al. 2024 ) and text-to-video synthesis ( Yang et al. 2024 ; Chen et al. 2024 ; Bar-Tal et al. 2024 ) , demonstrate practical applications ( Brooks et al. 2024 ; Google DeepMind 2025c ; Google DeepMind 2025b ) , and further enable the creation of 3D content ( Zhao et al. 2025 ) and large-scale world simulation models ( Google DeepMind 2025a ) . At the core of this advance is scalability: enlarging model capacity and data coverage reliably improves performance, often near-monotonically. When pushed to sufficiently large regimes, these trends yield marked gains in fidelity, coverage, and controllability. Crucially, these benefits depend on scale-friendly choices, including architectures that maintain stable signal flow, training recipes that remain well-behaved as width, depth, and batch size grow, and computational efficiency. Such scaling behavior has already been demonstrated in certain types of generative models such as autoregressive and diffusion families ( Tian et al. 2024 ; Peebles & Xie 2023 ; Liang et al. 2024 ) . By contrast, the scalability of Generative Adversarial Networks (GANs) has not been discussed yet, despite its attractive single-step sampling efficiency and interesting property of semantic latent space. While there have been attempts to train GANs at large scale ( Kang et al. 2023 ; Zhu et al. 2025 ; Sauer et al. 2023 ) , these efforts typically focus on a single high-capacity model with extensive, task-specific tuning, and thus do not constitute evidence of genuine scalability. In this work, we revisit GANs in the aspect of scalability. We focus on two ingredients that have proven central to the success of scalable generative models. First, these models are typically trained in a low-dimensional latent space, enabling a dramatic reduction of the computational burden of both learning and inference while preserving high perceptual fidelity. Second, they employ transformer architectures, which are known for their scalability against width, depth, data, and compute. Inspired by these two crucial factors, we combine these two elements to build a novel, scalable GAN framework: we construct a pure transformer-based GAN that operates in a compact latent space and study its behavior across substantial capacity ranges. We aim to assess the scalability of this design and to clarify the architectural and optimization choices. Accordingly, we pinpoint the hurdles that hinder adversarial training at scale. In detail, we identify the two key problems: (1) the early layers of the generator become inactive, leading to marginal contribution in image synthesis and (2) naïvely increasing depth and width with identical configuration leads to failures in convergence. To address the first issue, we propose Multi-level Noise-perturbed image Guidance (MNG), which provides supervision at multiple intermediate layers of the generator. Specifically, we leverage a noise hierarchy: the synthesized images from earlier stages are trained to resemble the real data perturbed by a stronger image-level Gaussian, and the noise level monotonically decreases with depth. They serve as direct supervision for the generator’s intermediate layers, restoring early-layer influence and improving layer-wise utilization throughout the network. For the second issue, we focus on the fact that both the static initialization and optimization scheme amplify output magnitudes as the model grows deeper and wider. Specifically, as model size increases, the entire network tends to exhibit more rapid changes in its outputs per optimization step. This phenomenon implies that the training speed changes proportionally to the model scale, potentially causing instability in GAN training dynamics. Thus, we devise a simple scaling rule for adjusting the hyperparameters, especially the learning rate, to preserve the constant magnitude of changes in network output regardless of scale. We experimentally validate that our framework, Generative Adversarial Transformers (GAT), is successfully trained on various scales of model (GAT-S to GAT-XL) and achieves FID of 2.96, which is the state-of-the-art performance in a one-step generation task on the class-conditional generation in ImageNet-256 dataset only within 40 epochs of training, while keeping the advantages of GAN, such as a single inference step or latent space manipulation (Fig. 1 , more examples are available in Appendix). 2 Proposed method 2.1 Preliminaries Generative Adversarial Networks Generative Adversarial Networks (GAN) ( Goodfellow et al. 2014 ) is an adversarial learning framework between two networks, the generator G(z,c) and discriminator D(I,c) . Specifically, for a given randomly sampled latent code z\in\mathbb{R}^{d_{z}}\sim p_{z} and condition c , the generator G(z,c) synthesizes a fake image \hat{x}\in\mathbb{R}^{H\times W\times 3} and the discriminator learns to distinguish the real image x\in\mathbb{R}^{H\times W\times 3} and the fake image \hat{x} , while the generator learns to deceive the discriminator. GAN has several interesting properties compared to other types of generative models, diffusion and AR models. For example, it offers extremely low dimensional latent space (e.g. d_{z}=64 ) and semantic latent space which is suitable for image manipulation. Moreover, its generation process requires only a single inference step, making inference highly efficient. Despite these advantages, GAN has not been explored in terms of scalability, which is one of the main cause of the success of other generative model. In this paper, we study how to scale GAN using on the transformer architecture that is already verified its scalability across various tasks. 2.2 Generative Adversarial Transformers We introduce Generative Adversarial Transformers (GAT), a transformer-based GAN framework at the latent space of VAE, for the first time. Our primary goal is to preserve the design of transformer as much as possible to keep its scalability. Basically, we build GAT on the latent space of VAE ( Rombach et al. 2022 ) , following the recent advances