TinyFusion: Diffusion Transformers Learned Shallow

Gongfan Fang, Kunjun Li, Xinyin Ma, Xinchao Wang · 2024

Read TinyFusion: Diffusion Transformers Learned Shallow by Gongfan Fang, Kunjun Li, Xinyin Ma, Xinchao Wang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01199.txt.

Introduction

Image-generating diffusion transformers are powerful but heavy, and Gongfan Fang and co-authors slim them by removing whole layers. Their method, TinyFusion, learns which layers are expendable instead of judging them by a fixed importance score. A differentiable sampling trick makes the cut itself trainable. An extra learned parameter rehearses the fine-tuning that will follow, so the method optimizes how good the model will be after recovery, not just right after cutting.

In tests, the approach beat importance-based and error-based rivals and transferred across several transformer families. On one large model it produced a shallow version at under seven percent of the original training cost. The slim model ran twice as fast while scoring better than competitors.

Catalog record

CREATORGongfan Fang, Kunjun Li, Xinyin Ma, Xinchao Wang
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-01199

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TinyFusion: Diffusion Transformers Learned Shallow Gongfan Fang, Kunjun Li, Xinyin Ma, Xinchao Wang Abstract Diffusion Transformers have demonstrated remarkable capabilities in image generation but often come with excessive parameterization, resulting in considerable inference overhead in real-world applications. In this work, we present TinyFusion, a depth pruning method designed to remove redundant layers from diffusion transformers via end-to-end learning. The core principle of our approach is to create a pruned model with high recoverability , allowing it to regain strong performance after fine-tuning. To accomplish this, we introduce a differentiable sampling technique to make pruning learnable, paired with a co-optimized parameter to simulate future fine-tuning. While prior works focus on minimizing loss or error after pruning, our method explicitly models and optimizes the post-fine-tuning performance of pruned models. Experimental results indicate that this learnable paradigm offers substantial benefits for layer pruning of diffusion transformers, surpassing existing importance-based and error-based methods. Additionally, TinyFusion exhibits strong generalization across diverse architectures, such as DiTs, MARs, and SiTs. Experiments with DiT-XL show that TinyFusion can craft a shallow diffusion transformer at less than 7% of the pre-training cost, achieving a 2 × \times speedup with an FID score of 2.86, outperforming competitors with comparable efficiency. Code is available at https://github.com/VainF/TinyFusion 1 Introduction Diffusion Transformers have emerged as a cornerstone architecture for generative tasks, achieving notable success in areas such as image Peebles and Xie 2023 ; Esser et al. 2024 ; Labs 2024 and video synthesis Zheng et al. 2024 ; Lab and etc. 2024 . This success has also led to the widespread availability of high-quality pre-trained models on the Internet, greatly accelerating and supporting the development of various downstream applications Yang et al. 2023 ; Chen et al. 2023 ; Gao et al. 2023 ; Yu et al. 2023 . However, pre-trained diffusion transformers usually come with considerable inference costs due to the huge parameter scale, which poses significant challenges for deployment. To resolve this problem, there has been growing interest from both the research community and industry in developing lightweight models Lin et al. 2024 ; Kim et al. 2023 ; Fang et al. 2023 ; Zhao et al. 2023 . Figure 1 : This work presents a learnable approach for pruning the depth of pre-trained diffusion transformers. Our method simultaneously optimizes a differentiable sampling process of layer masks and a weight update to identify a highly recoverable solution, ensuring that the pruned model maintains competitive performance after fine-tuning. The efficiency of diffusion models is typically influenced by various factors, including the number of sampling steps Song et al. 2020 ; Song et al. 2023 ; Lu et al. 2022 ; Salimans and Ho 2022 , operator design Teng et al. 2024 ; Dao et al. 2022 ; Xie et al. 2024 , computational precision Li et al. 2023 ; Shang et al. 2023 ; He et al. 2024 , network width Fang et al. 2023 ; Castells et al. 2024 and depth Kim et al. 2023 ; Daniel Verdú 2024 ; Men et al. 2024 . In this work, we focus on model compression through depth pruning Yu et al. 2022 ; Men et al. 2024 , which removes entire layers from the network to reduce the latency. Depth pruning offers a significant advantage in practice: it can achieve a linear acceleration ratio relative to the compression rate on both parallel and non-parallel devices. For example, as will be demonstrated in this work, while 50% width pruning Fang et al. 2023 only yields a 1.6× speedup, pruning 50% of the layers results in a 2× speedup. This makes depth pruning a flexible and practical method for model compression. This work follows a standard depth pruning framework: unimportant layers are first removed, and the pruned model is then fine-tuned for performance recovery. In the literature, depth pruning techniques designed for diffusion transformers or general transformers primarily focus on heuristic approaches, such as carefully designed importance scores Men et al. 2024 ; Daniel Verdú 2024 or manually configured pruning schemes Kim et al. 2023 ; Yu et al. 2022 . These methods adhere to a loss minimization principle Han et al. 2015 ; Molchanov et al. 2016 , aiming to identify solutions that maintain low loss or error after pruning. This paper investigates the effectiveness of this widely used principle in the context of depth compression. Through experiments, we examined the relationship between calibration loss observed post-pruning and the performance after fine-tuning. This is achieved by extensively sampling 100,000 models via random pruning, exhibiting different levels of calibration loss in the searching space. Based on this, we analyzed the effectiveness of existing pruning algorithms, such as the feature similarity Daniel Verdú 2024 ; Men et al. 2024 and sensitivity analysis Han et al. 2015 , which indeed achieve low calibration losses in the solution space. However, the performance of all these models after fine-tuning often falls short of expectations. This indicates that the loss minimization principle may not be well-suited for diffusion transformers. Building on these insights, we reassessed the underlying principles for effective layer pruning in diffusion transformers. Fine-tuning diffusion transformers is an extremely time-consuming process. Instead of searching for a model that minimizes loss immediately after pruning, we propose identifying candidate models with strong recoverability, enabling superior post-fine-tuning performance. Achieving this goal is particularly challenging, as it requires the integration of two distinct processes, pruning and fine-tuning, which involve non-differentiable operations and cannot be directly optimized via gradient descent. To this end, we propose a learnable depth pruning method that effectively integrates pruning and fine-tuning. As shown in Figure 1 , we model the pruning and fine-tuning of a diffusion transformer as a differentiable sampling process of layer masks Gumbel 1954 ; Jang et al. 2016 ; Fang et al. 2024 , combined with a co-optimized weight update to simulate future fine-tuning. Our objective is to iteratively refine this distribution so that networks with higher recoverability are more likely to be sampled. This is achieved through a straightforward strategy: if a sampled pruning decision results in strong recoverability, similar pruning patterns will have an increased probability of being sampled. This approach promotes the exploration of potentially valuable solutions while disregarding less effective ones. Additionally, the proposed method is highly efficient, and we demonstrate that a suitable solution can emerge within a few training steps. To evaluate the effectiveness of the proposed method, we conduct extensive experiments on various transformer-based diffusion models, including DiTs Peebles and Xie 2023 , MARs Li et al. 2024a , SiTs Ma et al. 2024a . The learnable approach is highly efficient. It is able to identify redundant layers in diffusion transformers with 1-epoch training on the dataset, which effectively crafts shallow diffusion transformers from pre-trained models with high recoverability. For instance, while the models pruned by TinyFusion initially exhibit relatively high calibration loss after removing 50% of layers, they recover quickly through fine-tuning, achieving a significantly more competitive FID score (5.73 vs. 22.28) compared to baseline methods that only minimize immediate loss, using just 1% of the pre-training cost. Additionally, we also explore the role of knowledge distillation in enhancing recoverability Hinton et al. 2015 ; Kim et al. 2023 by introducing a MaskedKD variant. MaskedKD

Read the complete text