Designing Parameter and Compute Efficient Diffusion Transformers using Distillation
Vignesh Sundaresha · 2025
Read Designing Parameter and Compute Efficient Diffusion Transformers using Distillation by Vignesh Sundaresha (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-14226.txt.
Introduction
Vignesh Sundaresha, writing alone, works on shrinking the AI models that generate images and video, in a 2025 paper. The leading models, built as diffusion transformers, hold billions of settings and need heavy computing. That size shuts them out of small devices such as smart glasses, where quick local generation would be prized.
The approach trains a small model to mimic a big one, a method called distillation. Sundaresha maps how choices like depth, width, and the number of attention heads trade performance against size and speed. Two new teaching schemes are proposed, one using a middle-sized helper model and one combining several helpers into one. Both are tested on real edge hardware, including a compact NVIDIA board, to show the small models can run where the giants cannot.
Catalog record
| CREATOR | Vignesh Sundaresha |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-14226
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Designing Parameter and Compute Efficient Diffusion Transformers using Distillation Vignesh Sundaresha Abstract Diffusion Transformers (DiTs) with billions of model parameters form the backbone of popular image and video generation models like DALL.E, Stable-Diffusion and SORA. Though these models are necessary in many low-latency applications like Augmented/Virtual Reality, they cannot be deployed on resource-constrained Edge devices (like Apple Vision Pro or Meta Ray-Ban glasses) due to their huge computational complexity. To overcome this, we turn to knowledge distillation and perform a thorough design-space exploration to achieve the best DiT for a given parameter size. In particular, we provide principles for how to choose design knobs such as depth, width, attention heads and distillation setup for a DiT. During the process, a three-way trade-off emerges between model performance, size and speed that is crucial for Edge implementation of diffusion. We also propose two distillation approaches - Teaching Assistant (TA) method and Multi-In-One (MI1) method - to perform feature distillation in the DiT context. Unlike existing solutions, we demonstrate and benchmark the efficacy of our approaches on practical Edge devices such as NVIDIA Jetson Orin Nano. 1 Introduction Diffusion Transformers (DiTs) ( Peebles & Xie 2023 ) have become the de facto method ( Dhariwal & Nichol 2021 ) for generating images and videos due to their high fidelity ( Ho et al. 2020 ) , generalizability ( Nichol & Dhariwal 2021 ) , ease of training ( Ho et al. 2020 ) and scalability ( Peebles & Xie 2023 ) . DiTs form the backbone of various practically deployed image and video generation models like DALL.E ( Betker et al. 2023 ) , StableDiffusion ( Esser et al. 2024 ) and Sora ( OpenAI 2024 ) . Due to the large parameter size and computational complexity of these models, one has to employ Cloud services to run them remotely. The significant latencies associated with such data transmissions from Cloud to the Edge cannot be afforded for high-frame-rate applications like Augmented/Virtual Reality (AR/VR) which need to be implemented on resource-constrained Edge devices ( Apple Inc. ; Meta Platforms, Inc. ) . The main challenge in directly implementing neural network inferences on Edge devices comes from the limited memory and energy capacity of the Edge hardware. To address this, we need to design parameter- and compute-efficient DiTs . Edge devices which typically hold on-chip memories in the range of few megabytes, thus requiring model sizes to be in the order of a million parameters as compared to existing practical models ( Peebles & Xie 2023 ; Crowson et al. 2024 ) which have billions of parameters. Prior works (explained in detail in Appendix A ) which focus on efficient DiTs optimize only specific layers ( Pu et al. 2024 ) or look at only precision ( Wu et al. 2024 ) or do not push the parameter-limit required to achieve the desired performance ( Geng et al. 2024 ) . The focus of our work is not about providing SOTA DiT models through novel algorithmic methods. Instead, our goal is to provide the best DiT model - in terms of performance and speed - at a given parameter size using principled design choices. Contributions: 1. We provide principles for designing an efficient SOTA (at the given model size) DiT model (DiT-Nano) by employing distillation. Through the process, we highlight a key trade-off that emerges between model performance (FID), size (#parameters) and speed (latency). In particular, we show the practical impact of our designed models on real-life Edge devices such as NVIDIA Jetson Orin Nano. This is our key contribution. 2. We propose two algorithms - Teaching Assistant (TA) method and the Multi-In-One (MI1) method, and explore the efficacy of these methods. 2 Designing Efficient DiTs Figure 1: Design space exploration of diffusion distillation. Design-Space Exploration : Among the several design knobs for distilling DiT models, we pick the following most relevant ones - depth, width, number of attention heads of the DiT model, and the setup (loss function and teacher models) for distillation. The former two knobs impact both the efficiency and performance, while the latter two only impact performance. We do not consider timesteps as a design knob despite its importance since extensive studies have been performed on that already ( Peebles & Xie 2023 ; Yin et al. 2024a ) (we do 1-step diffusion only). Below we propose two methods which explore new Distillation Setups for DiT. Teaching Assistant (TA) Method : This approach is inspired by the original TA paper ( Mirzadeh et al. 2020 ) for distilling convolutional networks. We explore the possibility of combined feature distillation (see Fig. 2 ) using the teacher and TA with LPIPS loss ( Zhang et al. 2018 ) . Multi-In-One (MI1) Method : This approach performs multiple diffusion timesteps in a single step by mapping the diffusion samples to specific layers of the student. The noise-image pair of the teacher model which does multi-step diffusion is used to calculate the intermediate noisy images by employing the forward diffusion probability flow ODE (shown in Section B.3 ). Figure 2: (Left) Baseline approach which performs regular knowledge distillation using offline teacher ( Geng et al. 2024 ) . (Center) Teaching Assistant (TA) which performs layer-wise distillation using an online teaching assistant and an offline teacher. (Right) Multi-In-One (MI1) approach which maps multiple diffusion steps into a single step. This is done by mapping different diffusion timesteps to different layers of a DiT. The training targets are obtained using forward diffusion of the probability flow ODE. See Appendix B for implementation details of TA and MI1. 3 Experiments In this section we mainly describe the principles for designing efficient DiTs and the experiments to corroborate them. The figure captions are provided with extensive detail and hence the text will only include points not mentioned in the captions. 3.1 Setup The comprehensive details of the training setup is provided in the Appendix C . The results below are shown for CIFAR-10 on DiTs distilled using EDM ( Karras et al. 2022 ) as teacher. We do not show results for other larger models or datasets since such a design-space exploration is computationally infeasible. Thus we provide guidelines from our study for such bigger and more practical implementations. We use FID ( Heusel et al. 2017 ) as the metric to evaluate the generated image performance while using model size and latency (instead of FLOPs) as the metrics for efficiency. Even though our results can be extended to other scenarios, we consider here mainly the scenario of offline distillation (noise-image pairs of teacher are generated before training) due to compute resources. 3.2 Design Space Exploration and Principles In this section, we look at the impact of various design knobs (mentioned in Section 2 ) on the performance and efficiency of the DiTs. Before sweeping the depth and width knobs, we first identify the best distillation setup based on existing methods and loss functions in Table 1 . The huge impact of using LPIPS loss ( Zhang et al. 2018 ) is obvious for both GET ( Geng et al. 2024 ) and DMD ( Yin et al. 2024b ) . The simpler training setup of GET provides better