Sparse-to-Sparse Training of Diffusion Models

Inês Cardoso Oliveira, Decebal Constantin Mocanu, Luis A. Leiva · 2025

Read Sparse-to-Sparse Training of Diffusion Models by Inês Cardoso Oliveira, Decebal Constantin Mocanu, Luis A. Leiva (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-21380.txt.

Introduction

Diffusion image makers are heavy to train, and Inês Oliveira, Decebal Mocanu, and Luis Leiva train them sparse from the first step, in work from twenty twenty-five. Three sparse training styles were tried on two model kinds across six data sets. Most weights stay zero through the whole run.

The sparse models matched their dense twins, and often beat them, while using far fewer weights and far less math. Safe sparsity levels are named for builders. The work moves thrift from the serving side to the training side, where the big bills sit. Labs with small clusters gain the most.

Catalog record

CREATORInês Cardoso Oliveira, Decebal Constantin Mocanu, Luis A. Leiva
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-21380

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Sparse-to-Sparse Training of Diffusion Models Inês Cardoso Oliveira, Decebal Constantin Mocanu, Luis A. Leiva Abstract Diffusion models (DMs) are a powerful type of generative models that have achieved state-of-the-art results in various image synthesis tasks and have shown potential in other domains, such as natural language processing and temporal data modeling. Despite their stable training dynamics and ability to produce diverse high-quality samples, DMs are notorious for requiring significant computational resources, both in the training and inference stages. Previous work has focused mostly on increasing the efficiency of model inference. This paper introduces, for the first time, the paradigm of sparse-to-sparse training to DMs, with the aim of improving both training and inference efficiency. We focus on unconditional generation and train sparse DMs from scratch (Latent Diffusion and ChiroDiff) on six datasets using three different methods (Static-DM, RigL-DM, and MagRan-DM) to study the effect of sparsity in model performance. Our experiments show that sparse DMs are able to match and often outperform their Dense counterparts, while substantially reducing the number of trainable parameters and FLOPs. We also identify safe and effective values to perform sparse-to-sparse training of DMs. 1 Introduction Diffusion models (DMs) are a class of deep generative models that exhibit extraordinary performance to produce diverse and high-quality data. DMs currently dominate the generative field in computer vision, having been applied to a wide range of tasks such as (un)conditional image generation [ Ho et al. 2020b , Rombach et al. 2021 , Nichol and Dhariwal 2021 , Dhariwal and Nichol 2021 , Nichol et al. 2022 , Blattmann et al. 2022 , Das et al. 2023 ] , image super-resolution [ Saharia et al. 2021 , Chung et al. 2022 ] , and image inpainting [ Nichol et al. 2022 , Chung et al. 2022 , Saharia et al. 2022 ] , among others. DMs have also shown incredible potential in other domains, including speech generation [ Liu et al. 2023a ] , text generation [ Li et al. 2022 , Gong et al. 2023 ] , and time-series prediction and imputation [ Rasul et al. 2021 , Tashiro et al. 2021 ] . Despite these advantages, DMs are notorious for their slow training, demanding significant computational resources and resulting in a considerable carbon footprint [ Strubell et al. 2020 ] . Due to the extensive number of diffusion timesteps required to produce a single sample (e.g., Rombach et al. 2021 mentioned up to 500 steps), DMs also suffer from slow sampling speed [ Song et al. 2021 ] . Even though progress has been made in improving inference speed, DMs are still considerably slower than other generative approaches such as GANs and VAEs [ Rombach et al. 2021 ] . This inefficiency impacts not only end users, but also the research community, by hindering further developments due to the lengthy process of model training and evaluation. Reducing the computational costs and memory requirements of DMs is a critical challenge for the broad implementation and adoption of these models, and an active field of research. Much of the existent literature has addressed this challenge through improvements to the inference stage [ Song et al. 2021 , Nichol and Dhariwal 2021 , Fang et al. 2023 , Shang et al. 2023 , Li et al. 2023 , Salimans and Ho 2022 , Meng et al. 2023 ] . Efforts have also been made in the direction of training efficiency, exploring different architectures and training strategies [ Wang et al. 2023 , Ding et al. 2023 , Rombach et al. 2021 , Phung et al. 2022 ] , but training DMs is still an extensive and costly process. In the last few years, sparse-to-sparse training has emerged as a promising approach to significantly reduce the computational cost of deep learning models, by training sparse networks from scratch [ Mocanu et al. 2018 , Bellec et al. 2018 , Dettmers and Zettlemoyer 2019 , Evci et al. 2020 , Zhang et al. 2024b ] . Interestingly, sparse neural networks have been shown to match, or even outperform, their Dense counterparts in classification tasks [ Mocanu et al. 2018 , Liu et al. 2021a ] , generative modeling using GANs [ Liu et al. 2023b ] , and Reinforcement Learning [ Sokar et al. 2022 ] , all while requiring less memory and reducing the number of floating-point operations (FLOPs). We should note that, currently, most sparse neural networks require roughly the same amount of time to train as their dense counterparts, since today’s hardware is optimized for dense matrix operations. However, growing interest in sparse models is reshaping the landscape; see Appendix A for a discussion in this regard. We propose to lower the computational cost of DMs by incorporating, for the first time, the paradigm of sparse-to-sparse training for unconditional generation. As such, we introduce three different methods, Static-DM (static strategy), RigL-DM, and MagRan-DM (both dynamic strategies), that can be easily integrated with existing DMs. Since our goal is to study the effect of these techniques on the performance of DMs, we experiment using two state-of-the-art DMs in two domains: Latent Diffusion [ Rombach et al. 2021 ] for image generation (continuous, pixel-level data) and ChiroDiff [ Das et al. 2023 ] for sketch generation (discrete, spatiotemporal sequence data). In sum, we make the following contributions: • We introduce sparse-to-sparse training to DMs, with both static and dynamic strategies. We consider various sparsity levels, two state-of-the-art models (Latent Diffusion and ChiroDiff), and six datasets in total. • Our experiments show great promise of sparse-to-sparse training for DMs, as we were able to train a sparse DM for each model/dataset case with comparable performance to their respective Dense counterpart, while significantly reducing the parameters count and FLOPs. In most cases, at least one sparse DM outperformed its Dense version. • We identify safe and effective values to perform sparse-to-sparse training of DMs. Higher performance is achieved using dynamic sparse training with 25–50% sparsity levels. For models with higher sparsity ratio, a conservative prune and regrowth ratio of 0.05 provides better results. 2 Background and Related Work 2.1 Diffusion Models DMs [ Sohl-Dickstein et al. 2015 , Ho et al. 2020a , Song et al. 2020 ] are probabilistic models designed to learn a data distribution q(x) through two processes: a forward noising process and a reverse denoising process. The forward process is defined as a Markov Chain of length T in which Gaussian noise is added at each timestep t , producing a sequence of increasingly noisier samples: q(x_{t}|x_{t-1})=\mathcal{N}(x_{t};\sqrt{1-\beta_{t}}x_{t-1},\beta_{t}\textbf{I}) (1) q(x_{1:T}|x_{0})=\prod_{t=1}^{T}q(x_{t}|x_{t-1}) (2) where x_{0} is the original data point, x_{t} is the data point at timestep t , and \beta_{t} is the pre-defined amount of noise added at timestep t . The reverse denoising process q(x_{t-1}|x_{t}) , attempts to recover the original data, but it is intractable as it depends on the entire data distribution q(x) . As such, we need to parameterize a neural network p_{\theta} to approximate it. This network p_{\theta} can be optimized by training with the simplified objective, \mathcal{L}=\mathbb{E}_{t\sim[1,T],x_{0},\epsilon\sim\mathcal{N}(0,1)}||\epsilon-\epsilon_{\theta}(x_{t},t)||^{2} , where x_{T} is a noisy version of input x at the final timestep T , and \epsilon_{\theta} the prediction of the neural network p_{\theta} . 2.2 Efficiency

Read the complete text