EraseFlow : Learning Concept Erasure Policies via GFlowNet-Driven Alignment

Abhiram Kusumba 2 ◊ ∗ 2\lozenge\ast Maitreya Patel 1 ​ ◊ 1\lozenge, Kyle Min 3 † 3\dagger, Changhoon Kim 4 4, Chitta Baral 1 1, Yezhou Yang 1 1 1 Arizona State University, 2 Capital One, 3 Oracle, 4 Soongsil University ◊ Equal contribution. ∗ Work done while at Arizona State University. † Work parti · 2025

Read EraseFlow : Learning Concept Erasure Policies via GFlowNet-Driven Alignment by Abhiram Kusumba 2 ◊ ∗ 2\lozenge\ast Maitreya Patel 1 ​ ◊ 1\lozenge, Kyle Min 3 † 3\dagger, Changhoon Kim 4 4, Chitta Baral 1 1, Yezhou Yang 1 1 1 Arizona State University, 2 Capital One, 3 Oracle, 4 Soongsil University ◊ Equal contribution. ∗ Work done while at Arizona State University. † Work parti (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-00804.txt.

Introduction

Abhiram Kusumba and colleagues introduce EraseFlow, a method for removing unwanted concepts from text to image generators. Erasing harmful or proprietary content is now a basic safety need, yet older methods often wreck image quality, depend on fragile adversarial training, or demand costly retraining.

EraseFlow treats unlearning as exploration over the full denoising paths a diffusion model takes, not just final images. It trains a policy with GFlowNets, which sample whole paths in proportion to their reward, steering generation away from the target concept while keeping the model's prior intact. Because no hand built reward model is needed, the approach generalizes to concepts not seen in training and, the authors report, beats prior baselines on the erasure versus quality trade off.

Catalog record

CREATORAbhiram Kusumba 2 ◊ ∗ 2\lozenge\ast Maitreya Patel 1 ​ ◊ 1\lozenge, Kyle Min 3 † 3\dagger, Changhoon Kim 4 4, Chitta Baral 1 1, Yezhou Yang 1 1 1 Arizona State University, 2 Capital One, 3 Oracle, 4 Soongsil University ◊ Equal contribution. ∗ Work done while at Arizona State University. † Work parti
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-00804

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

EraseFlow : Learning Concept Erasure Policies via GFlowNet-Driven Alignment Abhiram Kusumba 2 ◊ ∗ 2\lozenge\ast Maitreya Patel 1 ​ ◊ 1\lozenge, Kyle Min 3 † 3\dagger, Changhoon Kim 4 4, Chitta Baral 1 1, Yezhou Yang 1 1 1 Arizona State University, 2 Capital One, 3 Oracle, 4 Soongsil University ◊ Equal contribution. ∗ Work done while at Arizona State University. † Work partially done while at Intel Labs. Link: Project Page | Model | Code Abstract Erasing harmful or proprietary concepts from powerful text‑to‑image generators is an emerging safety requirement, yet current “concept erasure” techniques either collapse image quality, rely on brittle adversarial losses, or demand prohibitive retraining cycles. We trace these limitations to a myopic view of the denoising trajectories that govern diffusion‑based generation. We introduce EraseFlow , the first framework that casts concept unlearning as exploration in the space of denoising paths and optimizes it with GFlowNets equipped with the trajectory‑balance objective. By sampling entire trajectories rather than single end states, EraseFlow learns a stochastic policy that steers generation away from target concepts while preserving the model’s prior. EraseFlow eliminates the need for carefully crafted reward models and by doing this, it generalizes effectively to unseen concepts and avoids hackable rewards while improving the performance. Extensive empirical results demonstrate that EraseFlow outperforms existing baselines and achieves an optimal trade-off between performance and prior preservation. Warning: This paper may contain content that may seem as offensive in nature. Figure 1 : EraseFlow (ours) achieves effective concept erasure when compared with various concept erasure methods across diverse tasks—removing NSFW content (top), artistic styles like “Van Gogh” (middle), and fine-grained elements such as the “Nike” logo from shoes (bottom)—while preserving image quality and fidelity. 1 Introduction Recent advances in diffusion models have led to remarkable improvements in text-to-image generative models, enabling unprecedented photorealism and widespread adoption across various domains [ 50 , 12 , 49 , 45 , 69 ] . However, because these models are typically trained on large-scale, unregulated internet data, concerns have grown regarding their potential misuse, including the unauthorized reproduction of copyrighted or harmful content [ 54 , 15 ] . Consequently, methods to erase or “unlearn” specific concepts from pretrained text-to-image generators have become critically important to ensure model’s safety and compliance [ 17 , 28 ] . Prior approaches addressing these challenges broadly include filtering [ 67 ] , training data attribution [ 60 ] , post-generation filtering [ 41 ] , and explicit concept unlearning [ 17 , 28 ] . While filtering-based methods are hard to scale on arbitrary concepts, unlearning-based methods have recently attracted significant attention due to their capability to intervene. Early unlearning techniques primarily relied on fine-tuning pretrained diffusion models by modifying cross-attention mechanisms [ 18 ] . These approaches, however, often degrade the generative model’s overall quality and are susceptible to adversarial reintroduction of erased concepts [ 71 ] . Reinforcement learning (RL)-based strategies were later introduced to improve alignment [ 55 , 43 ] , yet they similarly suffer from brittleness and susceptibility to adversarial attacks. More recently, adversarial unlearning methods have demonstrated improved robustness [ 27 , 70 ] , but at the cost of substantial computational overhead, typically requiring hours of compute to erase a single concept, thereby severely limiting scalability. Figure 2 : Comparison of EraseFlow variants and baselines on erasing the nudity concept in Stable Diffusion v1.4. Robustness is measured by adversarial attack success rate (↓) and utility by FID (↓). Lower is better on both axes. Circle size reflects training cost. We hypothesize that these limitations originate from inadequate control of the post-training alignment process. Specifically, the stochastic nature of diffusion models implies that early denoising steps have substantial uncertainty and can yield diverse distributions, while later steps become more deterministic. Despite this inherent property, most existing methods treat all denoising steps equally during fine-tuning, ignoring the critical role of evolving conditional marginal distributions. This oversight often leads to suboptimal unlearning outcomes [ 27 , 16 ] . Therefore, a fine-tuning strategy that explicitly accounts for these conditional marginal distributions is crucial for effective and efficient concept erasure. Motivated by recent advancements in generative flow networks (GFlowNets) [ 5 ] — probabilistic models capable of sampling from unnormalized distributions—we propose EraseFlow , a novel unlearning method that leverages complete denoising trajectories to dynamically adapt alignment based on evolving conditional marginal distributions through a trajectory balance (TB) formulation. Additionally, to remove the dependency on manually designed and potentially vulnerable reward models, we introduce a straightforward reward-free alignment strategy. Specifically, we theoretically prove that even a constant reward in combination with TB leads to more reliable erasing of semantic content. This enables generalization to arbitrary unseen concepts without explicit reward specification. Critically, our alignment strategy ensures the preservation of the pretrained model’s prior while effectively removing targeted concepts. We validate the robustness and efficacy of EraseFlow through comprehensive evaluations across diverse and challenging scenarios (see Figure 1 ), including nudity filtering, artistic style removal, and fine-grained realistic concept erasure (e.g., corporate logos such as Nike). Our method consistently outperforms existing baselines on the UDAtk [ 71 ] benchmark without requiring adversarial training. Integrating EraseFlow with orthogonal methods like SAFREE [ 67 ] and AdvUnlearn [ 70 ] further improves results, achieving state-of-the-art performance with only 1% failure rates compared to the previous best of 16% on NSFW concept erasure. Furthermore, prior-preservation metrics indicate that our method retains superior generative quality, achieving state-of-the-art Fréchet Inception Distance (FID) [ 22 ] as shown in Figure 2 . In fine-grained tasks, EraseFlow uniquely demonstrates the ability to remove specific logos without adversely affecting general prior distributions, unlike competing methods. In summary, our main contributions are: • Introducing EraseFlow , the first GFlowNet-based method specifically designed for efficient and robust concept erasure from pretrained text-to-image diffusion models. • Proposing a novel reward-free alignment strategy; enabling generalization to arbitrary, unseen concepts without explicit reward definitions. • EraseFlow achieves the SOTA-like performance across various concept erasure benchmarks, while significantly reducing computational overhead and preserving prior generative quality and robustness against adversarial reintroduction. • Lastly, we show that EraseFlow can easily be composed with adversarial and filtering-based methods to further boost the performance. 2 Related Works Concept Erasure for Text-to-Image Diffusion Models. The ability to remove specific concepts from diffusion models has become a critical requirement for ensuring the ethical and legal alignment of generative AI systems. Rather than relying on reactive mechanisms such as post hoc filtering or attribution-based tracing [ 26 , 14 , 42 , 25 ] , concept erasure methods proactively disable a model’s capacity to synthesize targeted content. Concept erasure methods can be broadly categorized into three families. Fine-tuning approaches modify internal weights—most commonly in cross-attention or denoising modules—to suppress representations of the target concept by aligning them with benign alternatives [ 17 , 28 , 23 ] . These methods offer strong control over generation but often incur high retraining costs and risk unintended degradation of unrelated content. In contrast, closed-form editing techniques directly compute parameter updates, typically within projection matrices, to enable scalable multi-concept removal with minimal training overhead [ 18 , 39 , 2 ] . Lastly, inference-time interventions preserve model weights and

Read the complete text