UnGuide: Learning to Forget with LoRA-Guided Diffusion Models
Agnieszka Polowczyk, Alicja Polowczyk, Dawid Malarz, Artur Kasymov, Marcin Mazur, Jacek Tabor, Przemysław Spurek · 2025
Read UnGuide: Learning to Forget with LoRA-Guided Diffusion Models by Agnieszka Polowczyk, Alicja Polowczyk, Dawid Malarz, Artur Kasymov, Marcin Mazur, Jacek Tabor, Przemysław Spurek (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-05755.txt.
Introduction
Agnieszka Polowczyk and co-authors present UnGuide, a 2025 method that teaches image diffusion models to forget a concept without smudging everything else. Low-rank adapters can steer a model away from a target, but they often warp unrelated content and dull realism. Their fix adds a dynamic guidance mechanism at generation time.
The system watches the first denoising steps for stability and sets the guidance strength accordingly. When a prompt asks for the erased concept, the adapter leads and the base model pushes back; for innocent prompts the base model rules, preserving fidelity. Across object erasure and explicit content removal tests, UnGuide beat earlier adapter-based methods at controlled removal. Clean forgetting matters as image models spread and regulators demand takedowns inside the model itself.
Catalog record
| CREATOR | Agnieszka Polowczyk, Alicja Polowczyk, Dawid Malarz, Artur Kasymov, Marcin Mazur, Jacek Tabor, Przemysław Spurek |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-05755
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
UnGuide: Learning to Forget with LoRA-Guided Diffusion Models Agnieszka Polowczyk, Alicja Polowczyk, Dawid Malarz, Artur Kasymov, Marcin Mazur, Jacek Tabor, Przemysław Spurek Abstract Recent advances in large-scale text-to-image diffusion models have heightened concerns about their potential misuse, especially in generating harmful or misleading content. This underscores the urgent need for effective machine unlearning, i.e., removing specific knowledge or concepts from pretrained models without compromising overall performance. One possible approach is Low-Rank Adaptation (LoRA), which offers an efficient means to fine-tune models for targeted unlearning. However, LoRA often inadvertently alters unrelated content, leading to diminished image fidelity and realism. To address this limitation, we introduce UnGuide—a novel approach which incorporates UnGuidance, a dynamic inference mechanism that leverages Classifier-Free Guidance (CFG) to exert precise control over the unlearning process. UnGuide modulates the guidance scale based on the stability of a few first steps of denoising processes, enabling selective unlearning by LoRA adapter. For prompts containing the erased concept, the LoRA module predominates and is counterbalanced by the base model; for unrelated prompts, the base model governs generation, preserving content fidelity. Empirical results demonstrate that UnGuide achieves controlled concept removal and retains the expressive power of diffusion models, outperforming existing LoRA-based methods in both object erasure and explicit content removal tasks. Code is available at https://github.com/gmum/UnGuide . Figure 1: We propose a novel unlearning model, UnGuide, which consists of two key components: a LoRA adapter and an UnGuidance mechanism. While the LoRA adapter is responsible for removing specific concepts, it may inadvertently generate out-of-distribution content for prompts containing erased concepts (e.g., “ cat ” in the figure). Additionally, it can alter generations for unrelated prompts (e.g., “ dog ”). To mitigate this, we introduce an adaptive guidance mechanism that adjusts the influence of the LoRA adapter. For prompts containing erased concepts, we primarily rely on the adapted LoRA model, whereas for other prompts, we favor the base model to preserve the original generation quality. Introduction Large-scale text-to-image (T2I) models ( Chang et al. 2023 ; Ding et al. 2022 ; Lu, Liu, and Kong 2023 ; Malarz et al. 2025 ) have demonstrated remarkable generative capabilities, but their broad expressivity poses significant challenges regarding safety, ethics, and legal compliance. Unlearning in this context refers to deliberately suppressing the model’s capacity to represent or generate particular concepts, especially those that are offensive. Low-Rank Adaptation (LoRA) ( Hu et al. 2022 ) , introduced to enhance T2I models with new concepts, has recently been repurposed to facilitate targeted forgetting ( Lu et al. 2024 ) . The MACE framework employs specialized LoRA modules. First, residual information is erased from surrounding or frequently co-occurring words. Then, separate LoRA modules are trained to remove the core information specific to each target concept. The architecture leverages carefully designed loss functions and segmentation tools such as Grounded-SAM ( Liu et al. 2024 ) to localize erasure within attention maps, achieving a balance between generality and specificity. However, this methodology necessitates recalibration of tokens and dependent segmentation pipelines, which increases complexity and external requirements. To overcome these limitations, we introduce UnGuide (see Fig. 1 ), a novel unlearning model that employs a standard LoRA framework, eschewing both prompt embedding modification and reliance on external segmentation. Our approach pioneers an UnGuidance mechanism, inspired by AutoGuidance ( Karras et al. 2024 ; Kasymov et al. 2024 ) , but specifically tailored for concept removal. While AutoGuidance typically guides higher-quality generation using a weaker or undertrained model’s version, UnGuidance interpolates dynamically between base and adapted models. Both models employ classifier-free guidance (CFG) at inference, and our method refines CFG itself rather than replacing it, enabling fine-grained, adaptive unlearning control. Our experiments show two key results. First, LoRA is very effective at removing specific concepts and generalizes well out of context. Second, unlearning can unintentionally distort unrelated concepts. This pushes them away from the natural data manifold, causing instability and semantic drift. The destabilization is profound during unlearning because the elimination of a concept can induce highly diverse and unconstrained generative outputs. Analogous to Tolstoy’s insight: while real data forms a coherent manifold (“all happy families are alike”), aggressive unlearning may result in diverse and unconstrained outputs (“each unhappy family is unhappy in its own way”). UnGuide addresses this challenge by deploying a dynamic, per-prompt guidance schedule. During generation, we adaptively modulate the influence of the base and LoRA-adapted models according to their response diversity. Specifically, by sampling sets of partially denoised images from each model, we measure the discrepancies in their outputs. When the LoRA-adapted model exhibits high variance (typically for prompts targeting the unlearned concept) we reduce reliance on the base model, thereby reinforcing the forgetting effect. Conversely, for stable and in-distribution outputs, stronger base model guidance ensures overall fidelity and prevents semantic drift. Thus, for prompts unrelated to the banned concepts, the model largely mirrors original behavior, with minimal bias introduced by the LoRA adapter, ensuring image quality and semantic integrity elsewhere. In summary, our principal contributions are as follows: • We present UnGuide, a framework that combines LoRA adaptation with an UnGuidance mechanism to enable effective and adaptive unlearning in text-to-image (T2I) models. • We demonstrate that UnGuide dynamically interpolates the outputs of baseline and unlearned models, leveraging an analysis of partially denoised images to optimize guidance for each prompt. • We validate UnGuide through extensive experiments, demonstrating that it consistently outperforms existing LoRA-based methods in both object erasure and explicit content removal tasks, Figure 2: The frozen model (with parameters \theta^{*} ) predicts the noise twice: once for the original prompt c and once for the prompt c_{m} , which specifies the mapped or neutral concept. The model with the LoRA adapter (with parameters \theta ) also predicts the noise, but only for the prompt c . The two predictions of the frozen model are linearly combined, and then the L2 loss is computed. This cost function causes LoRA to suppress features associated with the undesirable concept. Related Works Figure 3: Overview of the adaptive guidance mechanism in UnGuide. We quantify the LoRA adapter’s influence relative to the base model by comparing the norms of predicted noise for a target prompt c and a neutral prompt c_{0} . After a short initial denoising phase (typically t=10 steps), we perform several additional denoising steps ( N=10 ) to approximate the output distribution. The difference in norms between the base and LoRA model predictions informs adaptive adjustment of the guidance scale: for w\leq-1 , we prioritize the LoRA model to ensure concept erasure (e.g., removing the cat), while for w\geq 1 , we lean on the base model to preserve the original concept in generation. The concept and formal problem of machine unlearning were first articulated by Kurmanji et al. 2023 , originally within the context of data deletion and privacy. The standard approach, i.e., refining the training dataset and retraining the model, is both computationally intensive and inflexible when adapting to new constraints ( Carlini et al. 2022 ; O’Connor 2022 ) . Other strategies, such as post-generation filtering or inference-time guidance, tend to be ineffective, as they are often circumvented by users ( Rando et al. 2022 ; Schramowski et al. 2023 ) . Recent methods addressing unlearning in