A Unified Framework for Diffusion Model Unlearning with f-Divergence
Nicola Novello, Federico Fontana, Luigi Cinque, Deniz Gündüz, Andrea M. Tonello · 2025
Read A Unified Framework for Diffusion Model Unlearning with f-Divergence by Nicola Novello, Federico Fontana, Luigi Cinque, Deniz Gündüz, Andrea M. Tonello (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-21167.txt.
Introduction
Deleting an unwanted concept from a text to image diffusion model can be framed in many mathematical ways. The work is by Nicola Novello and 4 co-authors. Diffusion models can make images of concepts that users want removed. Most removal methods shrink a squared error loss between outputs for a target and an anchor concept. That loss hides a known divergence measure between two Gaussians. The paper widens that view.
The authors extend the loss to any f-divergence, a family of ways to measure gaps between distributions. Some members give cheap closed form training goals. Others need a min max form, which they provide. They study how each choice changes gradient size and convergence. One closed form based on Hellinger distance beat the squared error baseline across many tests.
Catalog record
| CREATOR | Nicola Novello, Federico Fontana, Luigi Cinque, Deniz Gündüz, Andrea M. Tonello |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 131 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-21167
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Unified Framework for Diffusion Model Unlearning with f-Divergence Nicola Novello, Federico Fontana, Luigi Cinque, Deniz Gündüz, Andrea M. Tonello Abstract Machine unlearning aims to remove specific knowledge from a trained model. While diffusion models (DMs) have shown remarkable generative capabilities, existing unlearning methods for text-to-image (T2I) models often rely on minimizing the mean squared error (MSE) between the output distribution of a target and an anchor concept. We show that this MSE-based approach is a special case of a unified f f -divergence-based framework, in which any f f -divergence can be utilized. We analyze the benefits of using different f f -divergences, that mainly impact the convergence properties of the algorithm and the quality of unlearning. The proposed unified framework offers a flexible paradigm that allows to select the optimal divergence for a specific application, balancing different trade-offs between aggressive unlearning and concept preservation. 1 Introduction Deep learning algorithms for T2I generation based on DMs have demonstrated remarkable success and widespread adoption in recent years. Some notable examples are Stable Diffusion [ 1 ] , DALL \cdot E 2 [ 2 ] , and Imagen [ 3 ] , which find a large variety of applications in real-world scenarios. However, a significant challenge associated with these algorithms stems from the fact that they are trained over large scale datasets (such as LAION-5B [ 4 ] ) that are scraped from the Internet, and contain not safe for work (NSFW) content, including explicit material [ 5 ] , and copyrighted works, such as artistic styles and even personal information [ 6 , 7 ] . Since DMs are able to learn and memorize all these contents [ 8 ] , it is crucial to develop algorithms that erase specific concepts from the final trained models. To force a model to forget a concept, the trivial solution would be to re-train the model with a curated dataset where the specific concept has been removed. However, this is impractical, as it is resource-intensive and time-consuming, and in some cases the training dataset could have been deleted. Therefore, it is fundamental to develop techniques that would allow a given trained diffusion model to unlearn a specific concept. Figure 1: Unlearning a concept using different f -divergences yield different unlearning dynamics (top: mean squared error, middle: Hellinger closed-form, bottom: Hellinger variational). The inset in the bottom right corner of each image reports a not erased concept. Many algorithms proposed for DM unlearning are grounded on a shared idea: shifting the model’s prediction corresponding to the concept to be unlearned (referred to as "target") towards prediction of a substitute concept (referred to as "anchor"). The latter can be selected in many different ways, depending on the user’s goal: null concept, hyper-class, semantically close/distant to the concept to forget. Usually, this is achieved by minimizing an MSE-based loss [ 9 , 10 , 11 ] , derived from formulating the problem as the minimization of the Kullback-Leibler (KL) divergence between two Gaussian distributions, conditioned on the two concepts, target and anchor, characterizing the image generation process. In this paper, instead of minimizing the standard KL divergence, we propose minimizing the f -divergence between the two distributions conditioned on the target and anchor concept. For specific f -divergences, we derive the loss function for DM unlearning from the closed-form expression of the f -divergence between two Gaussian distributions. These loss functions, similarly to the standard MSE, benefit from a simple implementation and training complexity, while differing from the MSE in terms of convergence properties. When the closed-form expression does not exist, we use the variational representation of the f -divergence formulated as a min-max problem, which allows us to tackle the DM unlearning task with any f -divergence. We refer to the proposed method as f -divergence-based DM unlearning ( f -DMU). To the best of our knowledge, f -DMU is the first approach based on general f -divergences for the DM unlearning task. We theoretically study the gradients of the f -DMU closed-form objective functions and empirically verify the theoretical findings. Then, we theoretically prove the local convergence property of the proposed min-max formulation, and relate the choice of different f -divergences with the algorithm convergence speed. Finally, we perform an extensive numerical analysis to compare results with different f -divergences, and we demonstrate the superiority of the proposed unified framework with respect to the standard MSE approach. In particular, we empirically demonstrate that the deployment of f -divergences outperforms the MSE loss consistently across different unlearning scenarios, such as style ablation and object ablation. 2 Preliminaries 2.1 Related Work In this section, we present a condensed version of the related work in which we focus only on methods in the context of DMs, while we defer to Appendix A for an extended overview. Post-processing techniques Post-processing techniques target the elimination of unsafe generated images through the usage of filtering or inference guiding. Stable Diffusion (SD) [ 1 ] adopts a NSFW filter that removes all generated images with embeddings close to those of 17 pre-chosen nudity concepts [ 12 ] . Safe Latent Diffusion (SLD) [ 5 ] is applied during inference and acts by repelling the generation from unsafe contents. Another inference guidance-based approach is SAFREE [ 13 ] , which can be applied for both image and video generation. The main drawback of post-processing algorithms is that it is possible to remove them from the inference pipeline to allow the model to generate what should have been forgotten. Fine-tuning techniques Fine-tuning approaches modify the weights of a trained model. Many fine-tuning approaches share the common idea of aligning the target and anchor concepts by minimizing the KL divergence. Erased Stable Diffusion (ESD) [ 9 ] (and other methods derived from it [ 11 ] ) fine-tunes the model to align the probability distributions of the model’s output fed with a target concept and a neutral concept. Concept Ablation (CAbl) [ 10 ] minimizes the MSE between the model’s prediction corresponding to the target and anchor concept. With the goal of finding the best anchor concept, Bui et al. propose adversarial learning frameworks with a loss function comprising two MSE losses, and show that the best anchor concept should be semantically close to the target [ 14 , 15 ] . This knowledge is applied to design an MSE loss that shifts the target concept to semantically close concepts [ 16 ] . While the previous algorithms are limited in using loss functions based on the KL divergence between the corresponding probability distributions, another line of work proposes to update the model weights in closed-form, leading to faster computations [ 17 , 18 , 19 ] . These methods also target the minimization of the MSE, but the loss is formulated directly on the attention weights. However, these approaches are constrained in modifying cross-attention weights, while, in principle, distribution-based methods can update any network parameter and work for different models, such as flow-based generative models [ 20 ] . Domain Correction (DoCo) [ 21 ] extends CAbl by using a GAN-based [ 22 ] loss function. We show in Appendix A that DoCo and CAbl’s loss functions can be obtained as special cases