Sculpting Memory: Multi-Concept Forgetting in Diffusion Models via Dynamic Mask and Concept-Aware Optimization
Gen Li, Yang Xiao, Jie Ji, Kaiyuan Deng, Bo Hui, Linke Guo, Xiaolong Ma · 2025
Read Sculpting Memory: Multi-Concept Forgetting in Diffusion Models via Dynamic Mask and Concept-Aware Optimization by Gen Li, Yang Xiao, Jie Ji, Kaiyuan Deng, Bo Hui, Linke Guo, Xiaolong Ma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-09039.txt.
Introduction
Gen Li and co-authors teach image generators to forget in a 2025 paper. Text-to-image models memorize vast content, and owners sometimes must erase concepts, for copyright, bias, or safety. Erasing several concepts at once usually leaves residue, breeds instability, or spoils image quality.
Their method pairs a dynamic mask, which updates which weights may change as training proceeds, with a loss that keeps each erased concept aligned to a harmless parent class. A distillation guard keeps earlier erasures forgotten during later rounds. Tests show stronger forgetting with better fidelity and coherence than prior methods, especially across many concepts.
Catalog record
| CREATOR | Gen Li, Yang Xiao, Jie Ji, Kaiyuan Deng, Bo Hui, Linke Guo, Xiaolong Ma |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-09039
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Sculpting Memory: Multi-Concept Forgetting in Diffusion Models via Dynamic Mask and Concept-Aware Optimization Gen Li, Yang Xiao, Jie Ji, Kaiyuan Deng, Bo Hui, Linke Guo, Xiaolong Ma Abstract Text-to-image (T2I) diffusion models have achieved remarkable success in generating high-quality images from textual prompts. However, their ability to store vast amounts of knowledge raises concerns in scenarios where selective forgetting is necessary, such as removing copyrighted content, reducing biases, or eliminating harmful concepts. While existing unlearning methods can remove certain concepts, they struggle with multi-concept forgetting due to instability, residual knowledge persistence, and generation quality degradation. To address these challenges, we propose Dynamic Mask coupled with Concept-Aware Loss , a novel unlearning framework designed for multi-concept forgetting in diffusion models. Our Dynamic Mask mechanism adaptively updates gradient masks based on current optimization states, allowing selective weight modifications that prevent interference with unrelated knowledge. Additionally, our Concept-Aware Loss explicitly guides the unlearning process by enforcing semantic consistency through superclass alignment, while a regularization loss based on knowledge distillation ensures that previously unlearned concepts remain forgotten during sequential unlearning. We conduct extensive experiments to evaluate our approach. Results demonstrate that our method outperforms existing unlearning techniques in forgetting effectiveness, output fidelity, and semantic coherence, particularly in multi-concept scenarios. Our work provides a principled and flexible framework for stable and high-fidelity unlearning in generative models. Code available in https://github.com/coulsonlee/Sculpting-Memory-ICCV-2025 Figure 1 : We demonstrate the effectiveness of our proposed approach for multi-concept unlearning. When unlearning specific classes from ImageNet, our method removes the target classes while mapping the forgotten concepts to their corresponding superclasses, preserving prompt integrity. Similarly, for Not-Safe-for-Work (NSFW) content, our approach simultaneously unlearns both “naked” and “sexual” concepts while maintaining the meaning of text prompts and overall model performance. We used a black box to cover the sensitive areas. 1 Introduction Diffusion models have demonstrated exceptional capability in generating high-fidelity images conditioned on textual descriptions [ 40 , 39 , 18 , 17 , 44 ] . Models such as Stable Diffusion (SD) [ 35 ] , SiT-XL [ 27 ] , SDXL [ 33 ] , and PixArt- \alpha [ 5 ] leverage large-scale datasets to capture an extensive range of visual concepts, leading to highly creative and realistic image generation. However, this vast knowledge retention raises ethical and legal concerns, particularly in scenarios requiring the removal of copyrighted content, the mitigation of biases, or the elimination of harmful imagery. These concerns have spurred interest in machine unlearning [ 2 , 24 , 31 , 20 , 37 , 3 ] , which seeks to erase specific concepts while preserving the model’s ability to generate diverse and coherent images. A major challenge in this field is multi-concept unlearning , where multiple target concepts must be forgotten either sequentially or simultaneously. For example, a T2I model may need to remove harmful concepts like weapons and explicit content while preserving its ability to generate safe and diverse images. Most existing methods are designed for single-concept unlearning [ 45 , 8 , 7 , 19 , 4 , 9 , 13 ] . However, when applied iteratively to multiple concepts, these methods often lead to unintended consequences. Previously forgotten concepts may resurface due to residual knowledge, and overall generation quality may degrade, affecting unrelated content. A key issue is that unlearning is inherently dynamic, yet existing methods apply fixed update rules that uniformly modify parameters without considering interactions between concepts. This issue may create conflicts where forgetting one concept interferes with previously unlearned ones. As training progresses, these conflicts accumulate, leading to incomplete forgetting or instability, causing forgotten concepts to resurface or overall generation quality to degrade. Second, the multi-concept unlearning methods [ 25 , 26 , 46 ] struggle to balance forgetting effectiveness and content preservation, resulting in either incomplete forgetting or excessive degradation of generation quality (demonstrated in the Sec. 3 ). Third, many existing methods [ 8 , 7 , 45 , 4 , 19 , 9 , 25 , 26 , 46 ] fail to ensure semantic consistency after unlearning, often producing meaningless noise or artifacts instead of meaningful content replacement (visual results are shown in Figure 3 , 4 ). These challenges highlight the need for a more adaptive and stable unlearning mechanism that ensures consistent forgetting without compromising generation quality. To overcome these challenges, we propose Dynamic Mask coupled with Concept-Aware Loss , a novel unlearning framework for diffusion models. As shown in Figure 2 , the Dynamic Mask mechanism adaptively updates gradient masks based on current gradient information, allowing weight updates to be selectively frozen, unmasked, or replaced over time. This adaptive strategy ensures stable and effective multi-concept forgetting while minimizing interference with unrelated knowledge. The Concept-Aware Loss consists of two complementary objectives. The first ensures that post-unlearning generation remains semantically meaningful by aligning the model’s learned representations with higher-level conceptual categories, preventing degraded or arbitrary outputs. The second enhances the stability of multi-concept unlearning by preventing previously removed concepts from reappearing when new concepts are unlearned. With the guidance of the loss gradient, combined with Dynamic Mask , this formulation improves both the reliability and effectiveness of multi-concept forgetting. Our contribution can be summarized as follows: • Dynamic Mask : An adaptive mask mechanism that dynamically adjusts weight updates, stabilizing multi-concept unlearning across multiple training steps. • Concept-Aware Loss : A loss design that ensures meaningful replacement of forgotten concepts via superclass alignment while preventing their relearning, stabilizing the unlearning process. • Robust Multi-Concept Forgetting : The proposed approach effectively handles multiple sequential and simultaneous unlearning tasks while preserving overall model performance. • Comprehensive Evaluation : Extensive experiments demonstrate the effectiveness of our approach in terms of forgetting efficiency, output fidelity, and semantic consistency. Figure 2 : Overview of our proposed framework, illustrating the detailed training process and the working mechanism of dynamic mask through a diagram. 2 Related Works Machine unlearning was initially introduced for tasks requiring the removal or modification of specific data [ 11 , 12 , 2 ] , such as data privacy protection and model updates. With the rise of generative artificial intelligence, machine unlearning techniques have increasingly been applied to T2I diffusion models. Current machine unlearning approaches in diffusion models can be categorized based on the unlearning target. The first category, which most existing works focus on, is single-concept unlearning, where the goal is to forget a specific concept. The second category involves multi-concept unlearning, where multiple concepts are removed within the same model. Single-concept forgetting. SLD [ 36 ] incorporates a safety guidance mechanism to dynamically suppress inappropriate content. FMN [ 45 ] further refines this strategy by employing attention re-steering techniques for targeted concept removal. ESD [ 8 ] fine-tunes using conditioned and unconditioned scores from the frozen SD model to guide outputs away from the concepts being erased. In contrast, SA [ 13 ] adopts a continual learning framework with elastic weight consolidation and generative replay to achieve precise forgetting. Meanwhile, SalUn [ 7 ] utilizes gradient-based weight saliency to selectively update the parameters most sensitive to the target concept. AC [ 19 ] assigns an anchor concept to overwrite the target concept, ensuring that unlearning does not compromise the meaningfulness of the generated outputs. However,