Efficient Fine-Tuning and Concept Suppression for Pruned Diffusion Models
Reza Shirkavand, Peiran Yu, Shangqian Gao, Gowthami Somepalli, Tom Goldstein, Heng Huang · 2024
Read Efficient Fine-Tuning and Concept Suppression for Pruned Diffusion Models by Reza Shirkavand, Peiran Yu, Shangqian Gao, Gowthami Somepalli, Tom Goldstein, Heng Huang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-15341.txt.
Introduction
Reza Shirkavand and five co-authors tackle a safety flaw in shrinking image generation models, in a paper from 2024. Pruning and distillation make diffusion models small enough for phones. Yet the process can keep unwanted behavior, such as copying protected styles or making unsafe images, even when the fine-tuning data are clean.
Their method joins fine-tuning and unlearning in one two-level optimization. The small model keeps the teacher's quality and style range while learning to suppress set concepts. The framework plugs into varied pruning and unlearning methods. The aim is fast, safe deployment of small image models in controlled settings.
Catalog record
| CREATOR | Reza Shirkavand, Peiran Yu, Shangqian Gao, Gowthami Somepalli, Tom Goldstein, Heng Huang |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-15341
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Efficient Fine-Tuning and Concept Suppression for Pruned Diffusion Models Reza Shirkavand, Peiran Yu, Shangqian Gao, Gowthami Somepalli, Tom Goldstein, Heng Huang Abstract Recent advances in diffusion generative models have yielded remarkable progress. While the quality of generated content continues to improve, these models have grown considerably in size and complexity. This increasing computational burden poses significant challenges, particularly in resource-constrained deployment scenarios such as mobile devices. The combination of model pruning and knowledge distillation has emerged as a promising solution to reduce computational demands while preserving generation quality. However, this technique inadvertently propagates undesirable behaviors, including the generation of copyrighted content and unsafe concepts, even when such instances are absent from the fine-tuning dataset. In this paper, we propose a novel bilevel optimization framework for pruned diffusion models that consolidates the fine-tuning and unlearning processes into a unified phase. Our approach maintains the principal advantages of distillation—namely, efficient convergence and style transfer capabilities—while selectively suppressing the generation of unwanted content. This plug-in framework is compatible with various pruning and concept unlearning methods, facilitating efficient, safe deployment of diffusion models in controlled environments. Code is available here . 1 Introduction The rapid advancement in generative models, particularly diffusion models [ 46 , 49 , 19 , 50 ] , has led to the development of powerful tools capable of generating high-quality synthetic images [ 40 , 42 , 37 ] . However, the parameter-heavy architectures and significant memory requirements of these models often make their deployment on smaller GPU clouds and edge devices challenging. To address this resource-intensive nature, various approaches have been proposed [ 61 , 22 , 54 , 28 , 27 ] , with model pruning [ 3 ] being a prominent technique aimed at reducing the computational demands of diffusion models to improve efficiency [ 6 , 56 , 12 ] . While pruning can significantly alleviate computational costs, retraining is typically needed to restore the pruned model’s performance. Previous methods [ 22 , 56 , 12 ] have utilized knowledge distillation [ 17 , 41 ] to fine-tune pruned models effectively. Figure 1 : Comparison of images generated in the styles of Claude Monet (top row) and Mary Cassatt (bottom row), both impressionist artists, using Stable Diffusion, a pruned model finetuned using standard knowledge distillation, and our proposed controlled fine-tuning method. While both the Stable Diffusion and distilled models generate images in Monet’s style, our method effectively unlearns Monet’s Impressionist style while preserving Cassatt’s distinct Impressionist features. This indicates our approach’s advantage in selectively suppressing specific styles while maintaining high fidelity to the other unremoved concepts and features. While distillation enhances convergence speed and preserves the expressive power of the original model, it introduces inherent risks. Diffusion models are known to generate copyrighted or inappropriate (e.g., NSFW) content [ 39 , 47 ] , prompting ongoing efforts to filter out such content [ 9 , 23 , 44 , 59 , 53 , 36 , 58 , 20 ] without the significant expense of retraining on a filtered dataset. It might seem intuitive that removing unsafe concepts from the fine-tuning dataset would naturally filter out such content. However, this assumption does not necessarily hold. Even when such content is excluded from the fine-tuning dataset, unsafe concepts still persist in a pruned diffusion model, particularly when distillation is used during the fine-tuning process (See Figs. 1 and 2 ). This poses a critical challenge for deploying diffusion models in controlled environments, where generating these types of outputs is unacceptable. A naive approach to tackle this issue is to use distillation to recover the generative capabilities of the base model first, followed by a concept unlearning method to remove undesirable content. However, this approach is inefficient and, as we hypothesize and empirically demonstrate, suboptimal in practice. It often leads to ineffective concept removal and degraded generation quality due to the interdependent optimization requirements for both retraining and unlearning processes. Specifically, the parameters optimal for retraining to restore model performance are not necessarily the best initialization point for effective concept unlearning. Conversely, the parameters chosen for unlearning can impact the model’s ability to regain its generative capabilities. This mutual dependency creates a circular optimization problem where the success of one step depends on the outcomes of the other, resulting in suboptimal trade-offs between preserving generative quality and removing unwanted concepts. To effectively tackle this circular dependency challenge during the fine-tuning phase, we propose a novel bilevel optimization framework for fine-tuning pruned diffusion models. Our approach retains the benefits of distillation—rapid convergence and effective style transfer—while selectively suppressing unwanted generative behaviors from the base model. As we will show, it is significantly superior to the two-stage approach (first fine-tune then forget). The lower-level optimization performs standard distillation and diffusion loss minimization on the fine-tuning dataset to restore the model’s generative capabilities. Meanwhile, the upper-level directs the model away from generating unwanted concepts. Our method is a plug-in technique that can be integrated with various pruning methods for fine-tuning. It can also incorporate any concept unlearning method in the upper level optimization step. To summarize our contributions: • We introduce a novel bilevel optimization framework for fine-tuning pruned diffusion models, effectively solving the interdependence problem between restoring generative quality and removing undesirable content. By integrating these tasks, our method avoids the inefficiencies and suboptimal results of sequential approaches, where the circular dependency between retraining and unlearning results in degraded model performance. • Our framework is designed to be adaptable, allowing the fine-tuning of the result of any pruning method. It can also incorporate any concept unlearning technique in the upper-level optimization. This flexibility broadens the applicability of our method to a wide range of resource-constrained settings and customization needs. • Through extensive evaluations of artist style and NSFW content removal tasks, we demonstrate that our bilevel method significantly outperforms the two-stage approach. Our results show superior concept suppression while retaining high generation quality, underscoring the effectiveness of our method in real-world controlled deployment environments. 2 Related Work Figure 2 : Comparison of generative quality and style adherence: Row 1: The original Stable Diffusion 2.1 model. Row 2: A pruned version fine-tuned with 20,000 iterations of combined DDPM and distillation loss. Row 3: A pruned version fine-tuned with 20,000 iterations of our proposed bilevel fine-tuning approach, removing styles of Van Gogh, Monet, and Picasso. Our bilevel method is successful in retaining generative quality and style diversity while suppressing undesirable concepts. See the Sec. C.3 for prompts used. 2.1 Efficient Diffusion Architectures Numerous studies have targeted efficient architecture design for diffusion models [ 61 , 54 , 28 ] . For instance, MobileDiffusion [ 61 ] applies empirical adjustments based on performance metrics from MS-COCO [ 29 ] , while SnapFusion [ 28 ] focuses on finding optimal architectures specifically for text-to-image (T2I) models. Pruning-based approaches [ 11 ] attempt to eliminate non-essential layers or blocks of a pre-trained diffusion model. SPDM [ 6 ] evaluates the importance of weights via Taylor expansion, removing lower-value weights, while BK-SDM [ 22 ] streamlines the U-Net by eliminating blocks that minimally impact generation quality. APTP [ 12 ] employs a dynamic, prompt-based pruning strategy for T2I models, adjusting resources according to prompt complexity in a