Distill, Forget, Repeat: A Framework for Continual Unlearning in Text-to-Image Diffusion Models
Naveen George, Naoki Murata, Yuhta Takida, Konda Reddy Mopuri, Yuki Mitsufuji · 2025
Read Distill, Forget, Repeat: A Framework for Continual Unlearning in Text-to-Image Diffusion Models by Naveen George, Naoki Murata, Yuhta Takida, Konda Reddy Mopuri, Yuki Mitsufuji (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-02657.txt.
Introduction
Naveen George and four co-authors help image generators forget concepts they should no longer produce. Models trained on huge web datasets can clash with privacy and copyright law, including Europe's right to be forgotten. Retraining from scratch to remove one concept costs far too much.
Their Distill, Forget, Repeat framework unlearns concepts one after another in a continual way. Each cycle aims to erase the target while keeping other skills intact. Reliable unlearning could let model owners honor removal requests quickly, without rebuilding models or losing their wider abilities. A tool that can drop one face or one style on call is a tool rights desks can use at last.
Catalog record
| CREATOR | Naveen George, Naoki Murata, Yuhta Takida, Konda Reddy Mopuri, Yuki Mitsufuji |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 80 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-02657
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Distill, Forget, Repeat: A Framework for Continual Unlearning in Text-to-Image Diffusion Models Naveen George, Naoki Murata, Yuhta Takida, Konda Reddy Mopuri, Yuki Mitsufuji Abstract The recent rapid growth of visual generative models trained on vast web-scale datasets has created significant tension with data privacy regulations and copyright laws, such as GDPR’s “Right to be Forgotten.” This necessitates machine unlearning (MU) to remove specific concepts without the prohibitive cost of retraining. However, existing MU techniques are fundamentally ill-equipped for real-world scenarios where deletion requests arrive sequentially, a setting known as continual unlearning (CUL). Naively applying one-shot methods in a continual setting triggers a stability crisis, leading to a cascade of degradation characterized by retention collapse, compounding collateral damage to related concepts, and a sharp decline in generative quality. To address this critical challenge, we introduce a novel generative distillation based continual unlearning framework that ensures targeted and stable unlearning under sequences of deletion requests. By reframing each unlearning step as a multi-objective, teacher-student distillation process, the framework leverages principles from continual learning to maintain model integrity. Experiments on a 10-step sequential benchmark demonstrate that our method unlearns forget concepts with better fidelity and achieves this without significant interference to the performance on retain concepts or the overall image quality, substantially outperforming baselines. This framework provides a viable pathway for the responsible deployment and maintenance of large-scale generative models, enabling industries to comply with ongoing data removal requests in a practical and effective manner. Figure 1 : Qualitative comparison of our method (bottom row) against SOTA baselines on 10 sequential unlearning steps. This figure highlights the critical failure modes of existing methods in a continual setting. Baselines like ESD-x Gandikota et al. 2023 and MACE Lu et al. 2024 suffer from catastrophic “retention collapse”, where generative quality completely breaks down on retained concepts (right panel). Other methods like DUGE Thakral et al. 2025 avoid total collapse but still exhibit severe quality degradation and poor unlearning in later stages. In contrast, our method demonstrates a superior ability to both effectively unlearn target concepts (left panel) and preserve general knowledge (right panel), maintaining a good generative quality compared to the original SD model. $\dagger$ $\dagger$ footnotetext: Work done during an internship at Sony AI 1 Introduction Recent visual generative models such as SORA Brooks et al. 2024 , Gemini Team et al. 2023 , Imagen 3.0 Saharia et al. 2022 , and DALL·E 3 Betker et al. 2023 can synthesize photorealistic media at scale. These models are trained on massive web-scale datasets that often include sensitive personal data, NSFW material, and copyrighted content. Laws such as GDPR Regulation 2018 grant a “Right to be Forgotten” , but retraining foundation models from scratch is prohibitively expensive. The common workaround, post-generation filtering, is superficial; the model retains underlying knowledge, and filters can be bypassed by users Rando et al. 2022 ; Yang et al. 2024 . This motivates machine unlearning (MU): directly editing model weights to remove targeted concepts (objects, NSFW, artistic styles, identities) from generative behavior. However, deletion requests arrive sequentially rather than all at once. In principle, an ideal unlearning method should satisfy three desiderata: (I) Perfect unlearning , where forget concepts are not produced; (II) No Ripple Effects Amara et al. 2025 , so that related concepts remain intact (e.g., removing Brad Pitt should not distort Angelina Jolie or Leonardo DiCaprio; unlearning Van Gogh should not degrade other artistic styles); and (III) Quality Preservation , maintaining overall image quality. Current approaches map forget concepts to fixed placeholders (empty strings Gandikota et al. 2023 ; Wu et al. 2025 , general concepts Kumari et al. 2023 ; Gandikota et al. 2024 , or random concepts Fan et al. 2024 ), suppressing rather than truly unlearning them. This leads to ripple effects (collateral damage to related concepts Amara et al. 2025 ; Zhang et al. 2024d ). These challenges are significantly amplified in continual unlearning (CUL) Thakral et al. 2025 ; Gao et al. 2025 ; Wuerkaixi et al. 2025 , where deletion requests arrive sequentially over time. When existing one-shot methods are applied repeatedly, they trigger three catastrophic failures that compound across steps: (1) Retention collapse , where weak preservation mechanisms (simple KL divergence or anchor mapping Kumari et al. 2023 ; Gandikota et al. 2024 ; Heng and Soh 2023 ) cause cumulative knowledge loss, eventually leading to complete loss of general capabilities Zhang et al. 2024d ; Thakral et al. 2025 ; (2) Compounding ripple effects , where broad parameter updates Gandikota et al. 2023 ; Wu et al. 2025 and fixed placeholder mapping cause collateral damage that accumulates with each step, blurring semantic boundaries; and (3) Cumulative parameter drift , where parameters progressively drift from their original state without explicit regularization Kirkpatrick et al. 2017 , destabilizing the model and causing quality degradation. We formalize these as three failure modes in Section 3.2.3. These interacting failures also give rise to concept revival, where previously unlearned concepts spontaneously reappear after later updates George et al. 2025 . These problems fundamentally compound in CUL, as each step inherits and magnifies previous errors. To address these challenges, we propose a distillation-based framework grounded in continual learning principles. We formulate each unlearning step as teacher-student distillation, where the previous-step model serves as teacher providing reference for retained knowledge. Our framework consists of three synergistic components: (1) Generative replay with distillation counters retention collapse by explicitly distilling retain concepts using teacher-generated synthetic data, providing stronger knowledge preservation than simple KL constraints; (2) Parameter regularization counters drift through L2 penalties constraining deviation from the previous step, achieving stabilization similar to Elastic Weight Consolidation Kirkpatrick et al. 2017 ; and (3) Contextual trajectory re-steering achieves effective unlearning while mitigating ripple effects through context-preserving mapping strategies instead of fixed placeholders, surgically modifying generation trajectories to minimize collateral damage. We validate our approach on a challenging 10-step sequential unlearning benchmark. As shown in Figure 1 , while existing methods collapse after 3-5 steps with catastrophic quality degradation and concept revival, our method remains stable after 10 sequential steps, successfully unlearning forget concepts (Pikachu, Brad Pitt, Dog, Golf Ball, Van Gogh Style, Apple, Spiderman, Lionel Messi, Cartoon Style, Banana) while retaining general knowledge and image quality. We demonstrate clear superiority over state-of-the-art baselines across all evaluation metrics. Our contributions are: 1. Systematic analysis of CUL challenges: We elucidate how existing one-shot unlearning methods fail when applied sequentially, identifying retention collapse, ripple effects, and parameter drift as fundamental challenges. 2. Distillation-based framework: We propose a principled framework with three complementary components: generative replay, parameter regularization, and contextual trajectory re-steering, each addressing specific failure modes. 3. Evaluation protocol for CUL: We establish refined metrics and benchmark construction for evaluating continual unlearning. 4. Practical solution: We demonstrate a practical approach enabling industry deployment to handle sequential deletion requests while maintaining model stability and quality. 2 Related Works Unlearning in Text-to-Image Diffusion Models Machine unlearning (MU) has emerged as a critical post-hoc technique for aligning large-scale generative models with evolving safety, privacy, and copyright standards, obviating the need for complete model retraining. Initial approaches focused on re-mapping “forget” concepts to neutral anchors, as pioneered by ESD Gandikota et al. 2023 using classifier-free guidance and formalized