Unlearning in Diffusion Models: A Unified Framework with KL Divergence and Likelihood Constraints
Shervin Khalafi, Alejandro Ribeiro, Dongsheng Ding · 2026
Read Unlearning in Diffusion Models: A Unified Framework with KL Divergence and Likelihood Constraints by Shervin Khalafi, Alejandro Ribeiro, Dongsheng Ding (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30825.txt.
Introduction
Khalafi, Ribeiro, and Ding build a frame for making diffusion models forget chosen data while keeping other skills intact. Forgetting means staying close to the base model under strict split rules, using two KL forms and a likelihood rule. They prove strong duality for all three forms, name the best targets, build paired solvers, and beat weight-based rivals on keep-versus-forget trade-offs, with the likelihood form keeping held ideas best.
Catalog record
| CREATOR | Shervin Khalafi, Alejandro Ribeiro, Dongsheng Ding |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 106950 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.30825
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Unlearning in Diffusion Models: A Unified Framework with KL Divergence and Likelihood Constraints Shervin Khalafi, Alejandro Ribeiro, Dongsheng Ding Abstract Unlearning in diffusion models aims to remove undesirable data or concepts while preserving the utility of pretrained models—two fundamentally conflicting objectives. We propose a principled constrained optimization framework that formulates unlearning as minimizing the deviation from a pretrained model, subject to explicit separation constraints from the unlearning distributions. Specifically, we formulate three constrained optimization problems based on reverse and forward KL divergences, and likelihood constraints. The first two generalize existing approaches for concept and data unlearning, while the third offers a novel and natural formulation for unlearning. Despite the nonconvexity of the KL constraints, we establish strong duality for all three problems, enabling us to explicitly characterize their optimal solutions as unlearning targets and develop primal–dual algorithms for each formulation. Experimental results demonstrate that our KL-constrained approach achieves superior retention-unlearning tradeoffs compared to weight-based baselines for concept and data unlearning, and that our likelihood-based approach matches unlearning effectiveness while better preserving retained concepts compared to baselines. mnlargesymbols’164 mnlargesymbols’171 1 Introduction Generative diffusion models have emerged as an effective approach for synthesizing high-quality images that resemble training data and capture underlying concepts. However, this expressiveness also raises significant safety and ethical concerns, as such models may generate inappropriate content, including copyrighted material ( 5 ) or harmful content ( 43 ) . While policy-makers have made progress in regulating the removal of undesirable data or concepts from trained models ( 13 ; 20 ) , it remains an open problem to develop effective technical methods to achieve this goal. Recently, the classical concept of machine unlearning has been introduced in the context of diffusion models ( 4 ) . A standard approach aims to retain the utility of pretrained models, ensuring high-quality image generation, while simultaneously preventing the generation of specific harmful data or concepts—two inherently conflicting objectives. This principle has motivated recent empirical advances, including concept unlearning ( 17 ; 16 ) and data unlearning ( 39 ; 1 ) . Often, these works employ the simple practice of balancing two conflicting objectives by combining them with weighted sums. However, such weight-based methods are inherently ad hoc, as the weights are treated as hyperparameters, making the approach sensitive and poorly generalizable across scenarios. In this work, we take a principled constrained optimization approach to address the conflicting objectives in unlearning. To do so, we view retaining the utility of trained models as minimizing the distance between the trained model and the pretrained model, while preventing harmful generation is formulated as additional constraints that push the trained model away from specific data or concept distributions. As a result, the tradeoff between these objectives can be systematically characterized by a Lagrangian formulation. To measure the distance between two models, we employ reverse and forward KL divergences, which broadly correspond to concept and data unlearning, respectively. We summarize our key contributions in three aspects below. Figure 1 : Heat maps of a three-Gaussian mixture before/after unlearning one mode (Right up). From left to right: pretrained model, reverse KL-constrained unlearning, forward KL-constrained unlearning, and likelihood-constrained unlearning. (i) Problem Formulation . We formulate a distribution optimization problem that minimizes the reverse KL divergence to a pretrained model, while constraining the reverse KL divergences to the distributions associated with the concepts to be unlearned to remain above user-specified thresholds. Formulation ( RU ) corresponds to unlearning at the concept level. We then consider an analogous optimization problem based on the forward KL divergence ( FU ), which corresponds to unlearning at the data level. Finally, we formulate a distribution optimization problem ( LU ) that constrains the expected likelihood of the concepts to be unlearned to remain below user-specified thresholds. This constrained framework extends the unlearning objectives, thereby revealing their commonalities and differences; see Section 1.1 for exact formulations and discussion. (ii) Theoretical Analysis . Although the reverse and forward KL unlearning problems ( RU ) and ( FU ) are nonconvex , we exploit the convexity of the image of a non-atomic vector measure to prove that they are strongly dual, rather than relying on standard convex duality. In contrast, strong duality for the likelihood unlearning problem ( LU ) directly follows from its convexity. In all cases, we explicitly characterize the optimal solutions, which serve as principled unlearning targets; see Figure 1 for visualizations and discussion. A key implication of strong duality is that the unlearning problems can be solved equivalently in the dual domain. This enables our development of primal–dual algorithms instantiated with diffusion models for each formulation. (iii) Empirical Results . We demonstrate the effectiveness of our constrained optimization approach for concept and data unlearning in a text-to-image diffusion model. In concept unlearning, our method achieves the same degree of unlearning while deviating less from the pretrained model compared to equal-weights approaches. In data unlearning, assigning the same weight to every sample to be unlearned leads to excessive deviation from the pretrained model, whereas our method learns optimal weights that ensure unlearning while minimizing deviation. Furthermore, we show that our novel likelihood-based unlearning method achieves a better tradeoff between unlearning the target concept and preserving the retained concept. This allows us to perform strong unlearning without deviating significantly from the parts of the distribution we wish to retain. 1.1 Summary of our problem formulations Within the optimization framework over the probability measure space \Delta , we outline three principled unlearning targets and defer their exact characterizations to Section 2 . Here, q\in\Delta denotes the data distribution or the distribution induced by the pretrained model to be retained, and q_{\mathrm{u}}^{i}\in\Delta denotes the i th concept or data distribution to be unlearned. Reverse KL-Constrained Unlearning . We minimize the reverse KL divergence between the model p and the reference q to ensure that the model is close to the reference. To remove the m undesirable concepts, we impose m reverse KL divergence constraints, each encouraging the model to stay away from an undesirable concept encoded by q_{\text{u}}^{i} . To ensure this separation, we enforce a lower bound b_{i} on each KL divergence by formulating a reverse KL-constrained distribution optimization problem, \begin{array}[]{rl}\displaystyle\minimize_{p\,\in\,\Delta}&D_{\text{KL}}(p\,\|\,q)\\[5.69046pt] \subject&D_{\text{KL}}(p\,\|\,q_{\text{u}}^{i})\;\geq\;b_{i}\;\text{ for }\;i\in[m].\end{array} (RU) Problem ( RU ) aims to minimize deviation from the pretrained model q —thereby maximizing model utility—while forgetting undesirable concepts. In Section 2.1 , we capture a solution to Problem ( RU ) as a ratio of distributions, p_{\text{rev}}^{\star}(\cdot)\;\propto\;\frac{(q(\cdot))^{\alpha_{0}}}{\prod_{i\,=\,1}^{m}(q_{\text{u}}^{i}(\cdot))^{\alpha_{i}}} where \{\alpha_{i}\}_{i\,=\,0}^{m} is a set of non-negative exponents that balance the relative importance of each model. The subscript rev stands for reverse KL. The solution p_{\text{rev}}^{\star} reflects unlearning by reducing the probability of sampling in regions where q_{\text{u}}^{i}(\cdot) is large. This idea generalizes the concept unlearning target in concept erasing ( 17 ) and negation generation ( 12 ) . Forward KL-Constrained Unlearning . We employ the forward KL (rather than reverse KL) divergence to measure the closeness between two distributions. To do so, we obtain a variant of Problem ( RU ) in terms of forward KL, \begin{array}[]{rl}\displaystyle\minimize_{p\,\in\,\Delta}&D_{\text{KL}}(q\,\|\,p)\\[5.69046pt] \subject&D_{\text{KL}}(q_{\text{u}}^{i}\,\|\,p)\;\geq\;b_{i}\;\text{ for }\;i\in[m].\end{array} (FU) Since the forward KL divergence is an expectation over the data distribution, it essentially optimizes the log-likelihood of the data under the model—often approximated by the evidence