Cross-scale Aligned Supervision for Training GANs

Sangeek Hyun, MinKyu Lee, Jae-Pil Heo · 2026

Read Cross-scale Aligned Supervision for Training GANs by Sangeek Hyun, MinKyu Lee, Jae-Pil Heo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26449.txt.

Introduction

Sangeek Hyun, MinKyu Lee, and Jae-Pil Heo challenge staged GAN training as a coarse-to-fine hierarchy. Each stage chases realism at its own size, but stages need not show the same sample. No stage guides the next, so later stages can drift to another sample instead of refining it. They call this cross-scale trajectory misalignment. Their CAT transformer keeps scale-wise discriminators and adds a consistency penalty tying intermediates to the last output. On ImageNet-256, CAT-H/2 scores FID-50K 1.56 after 60 one-step epochs.

Catalog record

CREATORSangeek Hyun, MinKyu Lee, Jae-Pil Heo
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50571
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.26449

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Cross-scale Aligned Supervision for Training GANs Sangeek Hyun, MinKyu Lee, Jae-Pil Heo Abstract Modern GANs often introduce adversarial supervision on intermediate generator outputs and interpret the resulting multi-stage synthesis as coarse-to-fine hierarchical generation. In this work, we challenge this interpretation. We argue that standard scale-wise adversarial supervision does not construct a proper coarse-to-fine hierarchy: each intermediate image is independently pushed toward the real distribution at its own resolution, but this scale-wise realism does not ensure that outputs across stages represent the identical generated sample. Moreover, the scale-specific image produced at each stage is not used as an explicit refinement target for the subsequent stage. Therefore, its adversarial loss can improve a scale-specific output without constraining later stages to preserve the same sample trajectory, allowing them to move toward a different sample rather than refine the previous output. We refer to this problem as a cross-scale trajectory misalignment problem. To resolve it, we propose CAT, a Cross-scale Aligned Transformer for multi-scale adversarial generation. CAT keeps the discriminator scale-wise, so each intermediate output is evaluated at its own resolution, while adding a simple generator-side consistency regularization that aligns intermediate outputs with the final output. On class-conditional ImageNet-256, CAT-H/2 achieves an FID-50K of 1.56 with one-step inference after only 60 training epochs, outperforming strong one-step GAN and diffusion/flow baselines. Our analyses further show that CAT reduces cross-scale discrepancy, decreases inter-stage rewriting, and improves alignment with the final refinement direction. 1 Introduction Figure 1 : Method overview and comparison with 1-step baselines. (Left) Our method combines generator-side consistency regularization with a scale-wise discriminator. (Right) Our method achieves strong FID with substantially fewer training epochs in one-step generation in ImageNet-256. (a) Training dynamics of scale-wise supervision (b) Possible generator-side trajectories Figure 2 : Failure modes of standard scale-wise adversarial supervision. (a) Since each scale-specific image is independently supervised by the discriminator at its own resolution, adversarial gradients can push different stages toward different realistic modes. This enforces scale-wise realism, but not sample-wise cross-scale alignment. (b) At each stage, x_{k} is produced for adversarial supervision, while the subsequent stage continues from the generator feature f_{k} rather than taking x_{k} itself as input. Thus, under inconsistent supervision, later stages can follow a different sample trajectory rather than explicitly refine the previous output ( x_{k} ). In summary, adversarial training with scale-wise supervision does not by itself construct a proper coarse-to-fine hierarchy. Recent generative models have achieved remarkable progress in image synthesis. A common principle behind many of these advances is to decompose generation into intermediate stages, so that the model solves a sequence of simpler prediction problems rather than synthesizing a complete image at once. Diffusion models [ 11 , 30 , 23 , 27 ] follow this principle through iterative denoising, while autoregressive and masked prediction models [ 4 , 1 , 36 , 19 ] factorize image generation into a sequence of prediction steps. In these paradigms, intermediate states are not merely auxiliary predictions; they actively participate in the generation process and progressively guide the model toward the final sample. Generative Adversarial Networks (GANs) have also pursued hierarchical generation through multi-stage synthesis. Since GANs generate samples in a single forward pass, prior work has introduced adversarial supervision on intermediate generator outputs [ 16 , 17 , 20 , 15 , 13 ] . In modern GAN architectures [ 15 , 44 ] , this idea is commonly instantiated as scale-wise adversarial supervision, where each generator-stage output is converted into a scale-specific image and the discriminator evaluates each resolution independently. This design is usually interpreted as coarse-to-fine generation, where early stages form global structure and later stages refine details. In this work, we challenge this interpretation. We argue that standard scale-wise adversarial supervision does not construct a proper coarse-to-fine hierarchy: it independently optimizes each intermediate image as a scale-specific supervised output, without constraining how outputs across stages relate to one another. As a result, intermediate images can become realistic at their own resolutions, but need not form progressively refined states of the same generated sample. This failure originates from the supervision objective itself, as illustrated in Fig. 2(a) . In the basic scale-wise formulation, each intermediate image is compared with real images at its corresponding resolution. Such supervision provides direct scale-wise realism feedback, but it only matches per-scale distributions. Since each scale is judged independently, the adversarial gradient at one stage can push its output toward a realistic mode that differs from the mode selected at another stage. Therefore, outputs from different stages can become realistic at their own resolutions while failing to represent the same sample. This breaks sample-wise cross-scale alignment, which is necessary for a proper coarse-to-fine hierarchy. This issue is further reinforced by how intermediate outputs are used in multi-stage generators, as illustrated in Fig. 2(b) . At each stage, the scale-specific image x_{k} is optimized for adversarial supervision, but it is not enforced as the image-level refinement target of the next stage. Subsequent synthesis proceeds through the generator feature f_{k} , so later outputs can deviate from x_{k} when scale-wise objectives provide inconsistent signals. Thus, later stages may follow a different sample trajectory rather than refine the previous output. Together, the supervision objective and the generator-side usage explain why standard scale-wise adversarial supervision can produce realistic intermediate images without constructing a coherent coarse-to-fine generation hierarchy. Motivated by this, we propose CAT, a Cross-scale Aligned Transformer for multi-scale adversarial generation. CAT keeps the discriminator scale-wise, preserving direct adversarial feedback for each generated image. At the same time, it introduces a simple generator-side consistency regularization that aligns intermediate outputs with the final output. This design applies scale-wise adversarial feedback to coordinated intermediate targets, allowing intermediate supervision to support final-stage synthesis rather than optimizing disconnected side predictions. On ImageNet-256, CAT-H/2 achieves an FID-50K of 1.56 with one-step inference after only 60 training epochs, outperforming strong one-step GAN and diffusion/flow baselines with up to \sim 13\times fewer training epochs. These results suggest that, when hierarchical adversarial supervision is properly organized, transformer-based GANs can serve as highly competitive one-step generative models. 2 Preliminary Generative Adversarial Networks. Generative Adversarial Networks (GANs) [ 8 ] formulate image generation as an adversarial game between a generator G(z,c) and a discriminator D(x,c) . Here, x\in\mathbb{R}^{H\times W\times 3} denotes an image sample that can come either from the real data distribution p_{\mathrm{data}}(x\mid c) or from the generated distribution p_{G}(x\mid c) induced by G(z,c) with z\sim p_{z} , where z and c are random noise and condition, respectively. The discriminator distinguishes real and generated samples, while the generator learns to fool it. Multi-stage adversarial generation in GANs. Rather than supervising only the final generator output, several GAN frameworks expose intermediate images at multiple generator stages and apply adversarial feedback to them [ 16 , 17 , 15 , 13 ] . This design is often motivated as hierarchical or coarse-to-fine generation, where earlier stages provide coarse predictions and later stages refine them. Such intermediate supervision can be realized through multi-scale images [ 16 , 15 ] or multi-level noise perturbations [ 13 ] ; in this work, we focus on the multi-scale image formulation. We denote by x_{k} the image used for adversarial supervision at stage or scale k , where k=0,\ldots,K and

Read the complete text