NitroFusion: High-Fidelity Single-Step Diffusion through Dynamic Adversarial Training

Dar-Yen Chen Hmrishav Bandyopadhyay Kai Zou Yi-Zhe Song · 2024

Read NitroFusion: High-Fidelity Single-Step Diffusion through Dynamic Adversarial Training by Dar-Yen Chen Hmrishav Bandyopadhyay Kai Zou Yi-Zhe Song (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-02030.txt.

Introduction

Image generators that need many steps are slow, and a 2024 paper by Dar-Yen Chen and co-authors gets top quality in a single step. Their system, NitroFusion, trains the generator against a crowd of critics rather than by slow distillation from a bigger model.

Each critic head in the pool watches a different aspect of quality at a different noise level, like art judges split by specialty. Refresh rules keep the critics from memorizing and going stale. Paired global and local critics judge both the whole image and its details, and training mixes prompted and unprompted runs. Users can also add refinement steps at run time if they want more polish. The design aims to end the usual trade between speed and fidelity.

Catalog record

CREATORDar-Yen Chen Hmrishav Bandyopadhyay Kai Zou Yi-Zhe Song
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-02030

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

NitroFusion: High-Fidelity Single-Step Diffusion through Dynamic Adversarial Training Dar-Yen Chen Hmrishav Bandyopadhyay Kai Zou Yi-Zhe Song Abstract We introduce NitroFusion, a fundamentally different approach to single-step diffusion that achieves high-quality generation through a dynamic adversarial framework. While one-step methods offer dramatic speed advantages, they typically suffer from quality degradation compared to their multi-step counterparts. Just as a panel of art critics provides comprehensive feedback by specializing in different aspects like composition, color, and technique, our approach maintains a large pool of specialized discriminator heads that collectively guide the generation process. Each discriminator group develops expertise in specific quality aspects at different noise levels, providing diverse feedback that enables high-fidelity one-step generation. Our framework combines: (i) a dynamic discriminator pool with specialized discriminator groups to improve generation quality, (ii) strategic refresh mechanisms to prevent discriminator overfitting, and (iii) global-local discriminator heads for multi-scale quality assessment, and unconditional/conditional training for balanced generation. Additionally, our framework uniquely supports flexible deployment through bottom-up refinement, allowing users to dynamically choose between 1-4 denoising steps with the same model for direct quality-speed trade-offs. Through comprehensive experiments, we demonstrate that NitroFusion significantly outperforms existing single-step methods across multiple evaluation metrics, particularly excelling in preserving fine details and global consistency. Figure 1 : Our one-step diffusion pipeline generates vibrant and photorealistic images with exceptional detail in a single inference step, broadening the potential for text-to-image synthesis in applications like real-time interactive systems. 1 Introduction Recent advances in accelerated diffusion models [ 27 , 14 , 29 , 21 , 15 , 51 , 49 , 58 ] have demonstrated that high-quality image generation is possible with dramatically reduced step counts. While several approaches now achieve one-step generation [ 42 , 56 , 32 , 23 , 37 , 53 , 52 ] , they face significant challenges in matching the quality of multi-step methods, particularly in preserving fine details and ensuring global coherence. This quality gap has limited the practical adoption of single-step methods, especially in applications requiring both speed and high fidelity. The core challenge in single-step diffusion lies in compressing an entire denoising trajectory [ 57 , 25 ] into a single transformation. Traditional approaches based on distillation [ 39 , 46 ] struggle because they attempt to directly match intermediate states or distributions, leading to blurry outputs and loss of detail. Recent adversarial methods [ 13 , 42 , 43 , 52 ] show promise but face training instability and diversity collapse when pushed to single-step generation. NitroFusion introduces a fundamentally different approach to single-step diffusion through a dynamic adversarial framework. Consider how a panel of art critics evaluates a painting – each critic specializes in different aspects like composition, color, technique, and detail. Similarly, rather than relying on a single discriminator that can quickly become overconfident [ 12 , 31 , 30 , 8 ] , we maintain a large, dynamic pool of specialized discriminator groups that operate on top of a frozen UNet backbone [ 38 ] . Just as a diverse panel of critics provides more comprehensive feedback than a single judge, our ensemble of discriminators guides the generator toward high-quality outputs by providing specialized feedback at different noise levels [ 23 ] and spatial scales. Our framework implements this insight through three technical innovations: (i) a dynamic discriminator pool architecture where we leverage the teacher model’s UNet encoder as a frozen feature extractor, with multiple lightweight discriminator groups \mathcal{H}_{t^{*}} specialized for different noise levels t^{*} to improve generation quality, (ii) a strategic refresh mechanism that randomly re-initializes \sim 1% of discriminator heads while preserving the collective knowledge distribution across the pool to prevent discriminator overfitting – a common failure mode in GAN training – while maintaining stable adversarial feedback, and (iii) a multi-scale strategy with dual training objectives where global heads and local heads are compartmentalized in a 1:2 ratio, with global heads assessing overall image coherence at resolution H{\times}W and local heads examining fine-grained details in patches of size h{\times}w . These are further divided as unconditional and prompt-conditional discriminator heads (dual-training) effectively balancing prompt alignment with image coherence. These technical components work together to solve the fundamental challenges of single-step generation. The dynamic discriminator pool and refresh mechanism work in tandem to maintain a balanced feedback system throughout training – as established heads provide consistent feedback, the periodic introduction of new heads prevents the system from becoming too rigid or predictable. The multi-scale strategy then complements this dynamic feedback system, enabling our generator to achieve what previous approaches could not: transforming noise into high-quality images in a single step while avoiding the artifacts and quality degradation that typically plague fast generation methods. Notably, unlike existing approaches [ 23 , 37 , 52 ] that require separate models for different step counts, our framework uniquely supports flexible deployment through bottom-up refinement. While we optimize primarily for single-step generation, our model uniquely enables dynamic refinement – users can simply add steps (up to 4) on-demand if higher quality is desired, all with the same model weights. Through extensive experimentation, we demonstrate that NitroFusion consistently produces sharper, more detailed images than existing single-step methods. Our approach not only matches but often exceeds the quality metrics of recent fast diffusion models while maintaining the speed advantages of single-step generation. Human evaluation studies further confirm the superior visual quality of our results, particularly in challenging areas like face detail and texture preservation. Our key contributions include: (i) a dynamic discriminator pool with specialized discriminator groups to improve generation quality, (ii) strategic refresh mechanisms to prevent discriminator overfitting, and (iii) multi-scale strategy with dual training objectives to effectively balance prompt alignment and image coherence. Additionally, we uniquely enable flexible deployment by supporting 1-4 denoising steps with the same model weights. 2 Related Works 2.1 Timestep Distillation Timestep distillation accelerates inference in diffusion models by reducing the required sampling steps for high-quality output. Standard approaches [ 27 , 14 , 56 , 29 , 32 , 15 , 51 , 49 , 58 ] distil a multi-step teacher model into a student model with fewer steps. A common strategy is to approximate the sampling trajectory, modeled as an ordinary differential equation (ODE), of the teacher model in a reduced step count. This can be implemented by either preserving [ 57 ] the original ODE path at each timestep, or reformulating [ 25 , 42 ] and learning a more efficient trajectory directly from the final outputs. Recent works train a series of such student models that progressively lower sampling steps [ 39 , 28 ] , while enforcing self-consistency [ 46 , 21 ] . Hyper-SD [ 37 ] further combines ODE-preserving and -reformulating methods. However, these models often face quality degradation due to limited model fitting capacity. Different from flow-guided distillation, Distribution Matching Distillation (DMD) [ 53 , 52 ] minimizes the Kullback-Leibler (KL) divergence between generated and target distributions to directly match distributions on the sample domain. Despite these advancements, achieving high fidelity in one-step distillation remains challenging, as these models frequently struggle with degradation and instability in extreme low-step settings. 2.2 Adversarial Distillation Adversarial Diffusion Distillation [ 42 , 43 ] (ADD) incorporates GAN training to address the limitations of MSE-based distillation in

Read the complete text