AEGIS: Adversarial Target-Guided Retention-Data-Free Robust Concept Erasure from Diffusion Models Fengpeng Li Kemou Li Qizhou Wang Bo Han Jiantao Zhou Abstract Concept erasure helps stop diffusion models (DMs) from generating harmful content; but current methods face robustness–retention trade-off. Robustness means the model fine-tuned by concept erasure methods resists reactivation of erased concepts, even under semantically related prompts. Retention means unrelated concepts are preserved so the model’s overall utility stays intact. Both are critical for concept erasure in practice, yet addressing them simultaneously is challenging, as existing works typically improve one factor while sacrificing the other. Prior work typically strengthens one while degrading the other—e.g., mapping a single erased prompt to a fixed safe target leaves class-level remnants exploitable by prompt attacks, whereas retention-oriented schemes underperform against adaptive adversaries. This paper introduces A dversarial E rasure with G radient- I nformed S ynergy (AEGIS), a retention-data-free framework that advances both robustness and retention. First, AEGIS replaces handpicked targets with an Adversarial Erasure Target (AET) optimized to approximate the semantic center of the erased concept class. By aligning the model’s prediction on the erased prompt to an AET-derived target in the shared text–image space, AEGIS increases predicted-noise distances not just for the instance but for semantically related variants, substantially hardening the DMs against state-of-the-art adversarial prompt attacks. Second, AEGIS preserves utility without auxiliary data via Gradient Regularization Projection (GRP), a conflict-aware gradient rectification that selectively projects away the destructive component of the retention update only when it opposes the erasure direction. This directional, data-free projection mitigates interference between erasure and retention, avoiding dataset bias and accidental relearning. Extensive experiments show that AEGIS markedly reduces attack success rates across various concepts while maintaining or improving FID/CLIP versus advanced baselines, effectively pushing beyond the prevailing robustness–retention trade-off. The source code is in https://github.com/Feng-peng-Li/AEGIS . 1 Introduction While DMs ( 35 ; 11 ) excel at text-to-image (T2I) generation, biased training data poses safety and reliability risks, including harmful outputs ( 19 ) . Concept erasure ( 7 ) , modifying the denoising UNet of DMs to remove undesirable concepts, has become a standard measure to obtain a reliable DMs. These methods are typically categorized as output-based, realigning concept representations, or attention-based, aiming at manipulating cross-attention scores ( 2 ) . Despite progress, these techniques face two critical challenges. First, as shown in Fig. 1 , erased concepts can be reactivated by adversarial or semantically related prompts not seen during training ( 5 ) . Second, improving robustness against adversarial prompt attacks (APAs) comes at the cost of degrading the generation quality for unrelated concepts, creating a harsh robustness-retention trade-off ( 55 ; 56 ; 9 ; 26 ) . Both goals are critical for concept erasure, while seldom works can handle them simultaneously. Consequently, achieving robust erasure without compromising retention remains a central, unsolved problem. Figure 1: Adversarial prompts can still result in DMs fine-tuned by concept erasure methods generating images with harmful information. Prevailing studies ( 2 ) largely focus on the impact of mapping target on the trade-off between concept erasure and the retention. A critical yet underexplored dimension, however, is the influence of the erasure target on the model’s robustness against APAs, mitigating the trade-off seems better. In most methods, only an instance for the concept, e.g., nudity , is adopted to perform concept erasure by maximizing the distance of noises induced by the instance between pre- and post-fine-tuning DMs. However, practical users input often involve semantically equivalent or metaphorical expressions, such as naked or sexual . Since DM can be interpreted as a classifier ( 4 ) , these synonymous terms are likely to be clustered within the same semantic class in the latent space. If the chosen instance lies too far to the semantic center of the nudity class, the concept information may not be fully eliminated. Under APAs like 56 close to the concept semantic center, the incompleteness of nudity class erasure becomes evident. Building upon these insights, we introduce AEGIS ( Adversarial Erasure with Gradient-Informed Synergy ), a novel framework that significantly enhances the robustness of concept erasure against APAs while preserving the retention for unrelated concepts. AEGIS achieves this through two key components: Adversarial Erasure Target (AET) and Gradient Regularization Projection (GRP). AET with an optimizable embedding is iteratively updated to create a potent adversarial target. The model is then fine-tuned to minimize the difference between the noise predicted for AET and the erased concept, effectively strengthening the erasure process. GRP is applied to maintain the model’s performance on unrelated concepts without requiring additional retention data, thus preventing negative impacts from the erasure process. Experiments across various concepts against a range of APAs confirm the superior effectiveness of AEGIS, which surpasses state-of-the-art methods in erasure robustness by large margins while preserving higher retention performance. Contribution Summary. The contributions of this work can be summarized as follows: • We demonstrate that the vulnerability of concept erasure stems from an inappropriately chosen learning target. In particular, if the target lies too close to the semantic center—formed by words semantically related to the erased concept—the concept information cannot be fully removed. • We propose a novel retention-data-free framework, AEGIS, which synthesizes an AET to guide the removal of class-level information associated with erased concepts. Additionally, we introduce a gradient projection strategy to better balance erasure robustness and retention performance. • Extensive experiments on multiple datasets validate that AEGIS markedly enhances erasure robustness against a variety of APAs. Notably, AEGIS decreases the attack success rate by \sim 5.31% on the nudity concept and \sim 24% on the Van Gogh style concept under P4D ( 5 ) and UnlearnDiffAtk ( 56 ) , while incurring little impact on retention performance. 2 Preliminaries Latent Diffusion Models. This work focuses on latent diffusion models (LDMs) ( 30 ) . Given a training dataset \mathcal{D}=\{(\mathbf{x}_{i},\mathbf{c}_{i})\}_{i=1}^{N} , each image \mathbf{x}_{i} is paired with a prompt \mathbf{c}_{i} . A noise predictor {\bm{\epsilon}}_{{\bm{\theta}}}(\cdot|\mathbf{c}) , parameterized by {\bm{\theta}} , is learned to estimate the Gaussian noise {\bm{\epsilon}} added to the latent representation \mathbf{z}_{t} at each diffusion step t\in\{1,\dots,T\} , conditioned on \mathbf{c} , optimized as: \underset{{\bm{\theta}}}{\min}\ \mathbb{E}_{(\mathbf{x},\mathbf{c})\sim\mathcal{D},t,{\bm{\epsilon}}\sim\mathcal{N}(\mathbf{0},\mathbf{I})}\big[\|{\bm{\epsilon}}-{\bm{\epsilon}}_{{\bm{\theta}}}(\mathbf{z}_{t}|\mathbf{c})\|_{2}^{2}\big]. (1) Concept Erasure in DMs. To remove specific undesirable concepts from DMs, {\bm{\epsilon}}_{{\bm{\theta}}} is fine-tuned. Let \mathbf{c}_{e} denote the prompt about the (to-be-)erased concept , e.g., “nudity”. Existing methods ( 30 ; 3 ; 2 ) map \mathbf{c}_{e} to an erasure target concept \tilde{\mathbf{c}} —a generic concept (e.g., “a photo”) or simply an empty prompt. In parallel, a retention dataset, \mathcal{D}_{r}=\{(\mathbf{x}_{r,i},\mathbf{c}_{r,i})\}_{i=1}^{N_{r}} where \mathbf{c}_{r,i}\in\mathcal{C}_{\textit{retain}} is unrelated to \mathbf{c}_{e} , is employed to retain the model utility. Let {\bm{\theta}}_{0} denote the original parameters, the erasing objective is defined as: \underset{{\bm{\theta}}}{\min}\ \Big[\underbrace{\mathbb{E}_{t}\big[\|{\bm{\epsilon}}_{\bm{\theta}}(\mathbf{z}_{t}|\mathbf{c}_{e})-{\bm{\epsilon}}_{{\bm{\theta}}_{0}}(\mathbf{z}_{t}|\tilde{\mathbf{c}})\|_{2}^{2}\big]}_{\text{Erasing Loss}}+\lambda\cdot\underbrace{\mathbb{E}_{\mathcal{D}_{\text{r}},t}\big[\|{\bm{\epsilon}}_{\bm{\theta}}(\mathbf{z}_{t}|\mathbf{c}_{r})-{\bm{\epsilon}}_{{\bm{\theta}}_{0}}(\mathbf{z}_{t}|\mathbf{c}_{r})\|_{2}^{2}\big]}_{\text{Retention Loss}}\Big], (2) where \lambda balances erasure and retention. The erasing loss encourages the DM to respond to \mathbf{c}_{e} as if it were \tilde{\mathbf{c}} , thereby reducing its ability to reconstruct \mathbf{c} . In general, {\bm{\epsilon}}_{{\bm{\theta}}_{0}}(\mathbf{z}_{t}|\tilde{\mathbf{c}}) is set as {\bm{\epsilon}}_{{\bm{\theta}}_{0}}(\mathbf{z}_{t})-\eta\cdot({\bm{\epsilon}}_{{\bm{\theta}}_{0}}(\mathbf{z}_{t}|\mathbf{c}_{e})-{\bm{\epsilon}}_{{\bm{\theta}}_{0}}(\mathbf{z}_{t})) , where {\bm{\epsilon}}_{{\bm{\theta}}_{0}}(\mathbf{z}_{t}) is unconditional generation and \eta controls the erasing strength. Adversarial Prompt Attacks and Robust Concept Erasure. Prior studies ( 56 ; 5 ) reveal that, even after applying the erasure objective in Eq. ( 2 ), the erased concept may still resurface under adversarial prompt attacks. Given a \mathbf{c}_{e} that has been erased, one can craft an adversarial prompt \mathbf{c}^{*} to provoke the concept-erased model into producing a response akin to that of the original model when queried with \mathbf{c}_{e} . Specifically, \mathbf{c}^{*} can be obtained by solving \mathbf{c}^{*}=\argmin_{\|\mathbf{c}-\mathbf{c}_{e}\|_{0}\leq m^{\prime}}\mathbb{E}_{t}\big[\|{\bm{\epsilon}}_{\bm{\theta}}(\mathbf{z}_{t}|\mathbf{c})-{\bm{\epsilon}}_{{\bm{\theta}}_{0}}(\mathbf{z}_{t}|\mathbf{c}_{e})\big\|_{2}^{2}\big], (3) subject to the constraint that m^{\prime}\leq m dimensions of \mathbf{c}\in\mathbb{R}^{m} are learnable during optimization. To strengthen erasure robustness against such attacks, inspired by adversarial training ( 24 ; 17 ; 18 ) , AdvUnlearn ( 55 ) substitutes the original erased concept \mathbf{c}_{e} in Eq. ( 2 ) with an adversarial prompt \mathbf{c}^{*} directly within the objective. This compels the model to suppress not only prompts explicitly tied to \mathbf{c}_{e} but also nearby adversarial variants, thereby improving robustness against prompt-space leakage. With the preliminaries of concept erasure established, we now examine its inherent vulnerabilities. 3 Motivation: Unveiling the Vulnerability in Concept Erasure This section explores a fundamental vulnerability in current approaches of concept erasure in DMs, motivating the design of our AEGIS. Specifically, we investigate the following central question: Can a single prompt entity suffice to effectively remove all information of a concept from DMs? To explore this, we conduct an initial study on erasing the concept nudity ( \mathbf{c}_{e}^{0} ) and assess its influence on semantically related concepts— naked, sexual, erotic, and impure ( \mathbf{c}_{e}^{i},i=1,2,3,4 ). We define three concept groups to facilitate our analysis: \mathcal{C}_{0}=\{\mathbf{c}_{e}^{i}\}_{i=0}^{4} , \mathcal{C}_{1}=\{\mathbf{c}_{e}^{0},\mathbf{c}_{e}^{1}\} , and \mathcal{C}_{2}=\{\mathbf{c}_{e}^{2},\mathbf{c}_{e}^{3},\mathbf{c}_{e}^{4}\} . To characterize how concept erasure impacts related information, inspired by 4 interpreting DMs as generative classifiers, we view the predicted noise {\bm{\epsilon}}_{{\bm{\theta}}}(\mathbf{z}_{t}|\mathbf{c}_{e}) at timestep t as the model’s internal representation of concept \mathbf{c}_{e} . At the final step T , this noise becomes deterministic that reflects the location of \mathbf{c}_{e} in the latent space, and can be interpreted as a class prototype. Intuitively, similar concept instances induce similar noises, thus forming clusters in the latent space. This perspective enables two types of measurement: 1) Intra-model distances, reflecting semantic separation between concept instances in the same DM; 2) Cross-model distances, quantifying erasure strength by comparing predictions from pre- and post-erasure models. The former is exactly what we discussed above. To theoretically justify the latter, we establish Prop. 3.1 , suggesting that successful erasure results in a significant shift in the predicted noise embedding of the erased concept. Proposition 3.1 (Deviation Lower Bound) . Let \mathcal{L}_{\text{erase}}({\bm{\theta}})=\mathbb{E}_{t}\big[\|{\bm{\epsilon}}_{\bm{\theta}}(\mathbf{z}_{t}|\mathbf{c}_{e}^{0})-{\bm{\epsilon}}_{{\bm{\theta}}_{0}}(\mathbf{z}_{t}|\tilde{\mathbf{c}})\|_{2}^{2}\big] , and \Delta_{T}=\|{\bm{\epsilon}}_{{\bm{\theta}}_{0}}(\mathbf{z}_{T}|\mathbf{c}_{e}^{0})-{\bm{\epsilon}}_{{\bm{\theta}}_{0}}(\mathbf{z}_{T}|\tilde{\mathbf{c}})\|_{2}^{2} . Assume optimizing Eq. ( 2 ) yields \mathcal{L}_{\text{erase}}({\bm{\theta}})\leq\delta<\Delta_{T} . Then, \big\|{\bm{\epsilon}}_{{\bm{\theta}}}(\mathbf{z}_{T}|\mathbf{c}_{e}^{0})-{\bm{\epsilon}}_{{\bm{\theta}}_{0}}(\mathbf{z}_{T}|\mathbf{c}_{e}^{0})\big\|_{2}^{2}\geq\big(\sqrt{\Delta_{T}}-\sqrt{\delta}\big)^{2}. (4) The proof is deferred to Appx. E.1 . Prop. 3.1 indicates that successful concept erasure leads to a significant deviation in predicted noises before and after fine-tuning the DM. Based on this theoretical insight, Def. 1 formally defines predicted noise distances to systematically evaluate erasure quality and robustness across different concepts and methods: d_{0} quantifies semantic similarity within the base model, d_{1} captures erasure effectiveness across pre- and post-erasure models, and d_{2} evaluates the distance between original and adversarial concept prompts within the fine-tuned DM, reflecting adversarial vulnerability. With these metrics, we in Fig. 2 empirically analyze the impact of concept erasure on nudity and its related concepts using ESD ( 30 ) and AdvUnlearn. (a) (b) (c) (d) (e) Figure 2: Predicted noise distances (10 {}^{\text{-4}} ) between nudity and its synonyms, measured using the original DM {\bm{\theta}}_{0} and concept-erased DMs {\bm{\theta}}_{\text{ESD}} and {\bm{\theta}}_{\text{AET}} . Definition 1 (Predicted Noise Distance) . Define the predicted noise distances d_{0},d_{1},d_{2} between any two concept instances \mathbf{c}_{e}^{i},\mathbf{c}_{e}^{j}\in\mathcal{C}_{0} as: • d_{0}(\mathbf{c}_{e}^{i},\mathbf{c}_{e}^{j})\coloneq\|{\bm{\epsilon}}_{{\bm{\theta}}_{0}}(\mathbf{z}_{T}|\mathbf{c}_{e}^{i})-{\bm{\epsilon}}_{{\bm{\theta}}_{0}}(\mathbf{z}_{T}|\mathbf{c}_{e}^{j})\|_{2}^{2} • d_{1}(\mathbf{c}_{e}^{i},\mathbf{c}_{e}^{j})\coloneq\|{\bm{\epsilon}}_{{\bm{\theta}}}(\mathbf{z}_{T}|\mathbf{c}_{e}^{i})-{\bm{\epsilon}}_{{\bm{\theta}}_{0}}(\mathbf{z}_{T}|\mathbf{c}_{e}^{j})\|_{2}^{2} • d_{2}(\mathbf{c}_{e}^{i},\mathbf{c}_{e}^{j})\coloneq\|{\bm{\epsilon}}_{{\bm{\theta}}}(\mathbf{z}_{T}|\mathbf{c}_{e}^{i})-{\bm{\epsilon}}_{{\bm{\theta}}}(\mathbf{z}_{T}|\mathbf{c}_{e}^{j,*})\|_{2}^{2} Table 1: Concept erasure robustness by UnlearnDiffAtk ( 56 ) . Method Pre-ASR ( \downarrow ) ASR ( \downarrow ) SD v1.4 100% 100% ESD 20.42% 76.05% AdvUnlearn 15.45% 64.79% ESD w/ AET 6.38% 26.06% AdvUnlearn w/ AET 1.41% 5.63% Impact of Concept Erasure. Fig. 2(a) reveals that noise for nudity and naked ( \mathcal{C}_{1} ) exhibit proximity, while sexual , erotic , and impure ( \mathcal{C}_{2} ) show significant divergence. After erasing nudity via ESD, Fig. 2(b) demonstrates enlarged d_{1} for \mathcal{C}_{1} but limited changes for \mathcal{C}_{2} , indicating ineffective removal of \mathcal{C}_{2} . This observation aligns with pre-attack success rate (Pre-ASR) results in Tab. 1 : ESD-fine-tuned DMs still generate nudity images in response to benign prompts from 56 , indicating that using an instance from \mathcal{C}_{1} or \mathcal{C}_{2} as \mathbf{c}_{e} cannot remove all information of \mathcal{C}_{0} from the DM. Impact of APAs. Despite ESD fine-tuning, Fig. 2(c) reveals consistently small d_{2} for \mathcal{C}_{0} . This low d_{2} enables adversarial prompts to achieve high ASR, as reported in Tab. 1 . Even AdvUnlearn, which integrates adversarial prompts into Eq. ( 2 ), fails to defend against UnlearnDiffAtk. From the phenomenon, we guess this vulnerability stems from inadequate mapping targets \tilde{\mathbf{c}} . As shown in Fig. 9 a of Appx. D.1 , if \mathbf{c}_{e} (e.g., member of \mathcal{C}_{1} ) is far from the concept center of \mathcal{C}_{0} , maximizing the distance between {\bm{\epsilon}}_{{\bm{\theta}}_{0}}(\mathbf{z}_{t}|\mathbf{c}_{e}) and {\bm{\epsilon}}_{{\bm{\theta}}}(\mathbf{z}_{t}|\mathbf{c}_{e}) , i.e., minimizing the gap between {\bm{\epsilon}}_{{\bm{\theta}}}(\mathbf{z}_{t}|\mathbf{c}_{e}) and {\bm{\epsilon}}_{{\bm{\theta}}_{0}}(\mathbf{z}_{t}|\tilde{\mathbf{c}}) , cannot remove the information of \mathcal{C}_{2} from the DM. Since c^{*} is not close to \mathcal{C}_{1} but also \mathcal{C}_{2} and similar to the concept center of \mathcal{C}_{0} , then the residual information of the undesirable concept will be leaked again. Under such circumstances, only when the gap between {\bm{\theta}} and {\bm{\theta}}_{0} is extremely large, where d_{1} is sufficient, the knowledge of \mathcal{C}_{0} can be erased. However, such a drastic parameter shift also disrupts preserved concepts, resulting in poor utility of the fine-tuned DM. That is the reason why existing methods exhibit a robustness–retention tradeoff—stronger erasure compromises generation fidelity . Impact of Adversarial Erasure Target. Given the small d_{2} values for all instances in \mathcal{C}_{0} , we hypothesize that the adversarial prompt used for concept erasure resides near the semantic center of \mathcal{C}_{0} . That means, as shown in Fig. 9 b of Appx. D.1 , maximizing the distance between \mathbf{c}_{e} and its concept center can enlarge d_{1} for all \mathcal{C}_{0} , enhance the concept erasure effectiveness. In light of this, we propose an adversarial erasure target (AET) updated by K=10 steps, defined as: \mathbf{c}^{{\prime}(k+1)}=\mathbf{c}^{{\prime}(k)}-\beta\cdot\mathrm{sign}\Big(\nabla\big(\|{\bm{\epsilon}}_{{\bm{\theta}}_{0}}(\mathbf{z}_{t}|\mathbf{c}^{{\prime}(k)})-{\bm{\epsilon}}_{{\bm{\theta}}}(\mathbf{z}_{t}|\mathbf{c}_{e})\|_{2}^{2}+\|{\bm{\epsilon}}_{{\bm{\theta}}_{0}}(\mathbf{z}_{t}|\mathbf{c}^{{\prime}(k)})-{\bm{\epsilon}}_{{\bm{\theta}}_{0}}(\mathbf{z}_{t}|\mathbf{c}_{e})\|_{2}^{2}\big)\Big), (5) where \beta is the step size, and \mathbf{c}^{{\prime}(0)} is randomly initialized with learnable parameters of length m^{\prime}=1 . We then apply the optimized \mathbf{c}^{{\prime}} into ESD, yielding the objective as: {\bm{\theta}}_{\text{AET}}=\argmin_{{\bm{\theta}}}\mathbb{E}_{(\mathbf{x},\mathbf{c}),t}[\|{\bm{\epsilon}}_{{\bm{\theta}}}(\mathbf{z}_{t}|\mathbf{c}_{e})-{\bm{\epsilon}}_{{\bm{\theta}}_{0}}(\mathbf{z}_{t}|\tilde{\mathbf{c}})\|_{2}^{2}], (6) where {\bm{\epsilon}}_{{\bm{\theta}}_{0}}(\mathbf{z}_{t}|\tilde{\mathbf{c}})={\bm{\epsilon}}_{{\bm{\theta}}_{0}}(\mathbf{z}_{t})-\eta({\bm{\epsilon}}_{{\bm{\theta}}_{0}}(\mathbf{z}_{t}|\mathbf{c}^{\prime})-{\bm{\epsilon}}_{{\bm{\theta}}_{0}}(\mathbf{z}_{t})) . With the target, as shown in Fig. 2(d) , fine-tuning with AET consistently enlarges d_{2} across all instances in \mathcal{C}_{0} . Moreover, compared with AdvUnlearn in Tab. 1 , ESD-AET achieves a 9.07% and 59.16% reduction in pre-ASR and ASR, respectively , indicating the information of nudity is effectively removed from the DM. Building on the effectiveness of AET, we propose AEGIS to enhance the concept erasure robustness. 4 Method: Adversarial Erasure with Gradient-Informed Synergy This section presents the details of the proposed AEGIS, illustrated in Fig. 3 . AEGIS comprises two core components: AET generation (§ 4.1 ) constructs AETs to enhance concept erasure robustness against APAs; GRP (§ 4.2 ) fine-tunes {\bm{\theta}} that minimizes the gap between predicted noises induced by erased concepts and their corresponding AETs. This procedure enhances concept erasure robustness with better retention. The optimization objective of AEGIS is formalized as: \underset{{\bm{\theta}}}{\min}\ \bigg[\underset{\tilde{\mathbf{c}}}{\max}~\underbrace{\mathbb{E}_{t}\big[\|{\bm{\epsilon}}_{{\bm{\theta}}}(\mathbf{z}_{t}|\mathbf{c}^{*})-{\bm{\epsilon}}_{{\bm{\theta}}_{0}}(\mathbf{z}_{t}|\tilde{\mathbf{c}})\|_{2}^{2}\big]}_{\text{Erasing Loss $\mathcal{L}_{e}$}}+~\lambda\cdot\underbrace{\frac{1}{2}\|{\bm{\theta}}-{\bm{\theta}}_{0}\|_{2}^{2}}_{\text{Retention Loss $\mathcal{L}_{r}$}}\bigg] (7) where \mathbf{c}^{*} is the adversarial prompt for \mathbf{c}_{e} , and {\bm{\epsilon}}_{{\bm{\theta}}_{0}}(\mathbf{z}_{t}|\tilde{\mathbf{c}})={\bm{\epsilon}}_{{\bm{\theta}}_{0}}(\mathbf{z}_{t})-\eta\big({\bm{\epsilon}}_{{\bm{\theta}}_{0}}(\mathbf{z}_{t}|\mathbf{c}^{\prime})-{\bm{\epsilon}}_{{\bm{\theta}}_{0}}(\mathbf{z}_{t})\big) . Figure 3: Overview of AEGIS. The proposed framework introduces two core components: (I) adversarial erasure target (AET) generation (§ 4.1 ), and (II) gradient regularization projection (GRP) fine-tuning (§ 4.2 ). 4.1 AET for Guiding the Robust Erasure Optimization Direction This part introduces the details of generating AET. As described in § 3 , empirical observations suggest that the adversarial prompt, corresponding to an instance of the erased concept, is close to the concept center. Thereby, we introduce AET in Eq. ( 5 ). Considering the dynamic changes of the erased concept, the AET is optimized to be close to the original and updated concept centers, maximizing the distance {\bm{\epsilon}}_{{\bm{\theta}}}(\mathbf{z}_{t}|\mathbf{c}_{e}) and {\bm{\epsilon}}_{{\bm{\theta}}_{0}}(\mathbf{z}_{t}|\mathbf{c}_{e}) continuously. Moreover, the iterative mechanism of AET generation in Eq. ( 5 ) reduces the efficiency of AEGIS, while the gap between \mathbf{c}^{\prime} and \mathbf{c}_{e} is not small enough with a single iteration. Thereby, following the fast adversarial training strategy ( 14 ; 15 ; 13 ) , for the \tau -th training epoch, AET is generated efficiently as \mathbf{c}^{{\prime}(\tau+1)}=\mathbf{c}^{{\prime}(\tau)}-\beta\cdot\mathrm{sign}\Big(\nabla\big(\|{\bm{\epsilon}}_{{\bm{\theta}}_{0}}(\mathbf{z}_{t}|\mathbf{c}^{{\prime}(\tau)})-{\bm{\epsilon}}_{{\bm{\theta}}}(\mathbf{z}_{t}|\mathbf{c}_{e})\|_{2}^{2}+\|{\bm{\epsilon}}_{{\bm{\theta}}_{0}}(\mathbf{z}_{t}|\mathbf{c}^{{\prime}(\tau)})-{\bm{\epsilon}}_{{\bm{\theta}}_{0}}(\mathbf{z}_{t}|\mathbf{c}_{e})\|_{2}^{2}\big)\Big). In line with 55 , similar to \mathbf{c}^{\prime} , \mathbf{c}^{*} for the erased concept \mathbf{c}_{e} is updated via \mathbf{c}^{{*}(\tau+1)}=\mathbf{c}^{{*}(\tau)}-\beta\cdot\mathrm{sign}\big(\nabla\|{\bm{\epsilon}}_{{\bm{\theta}}}(\mathbf{z}_{t}|\mathbf{c}^{{*}(\tau)})-{\bm{\epsilon}}_{{\bm{\theta}}_{0}}(\mathbf{z}_{t}|\mathbf{c}_{e})\|_{2}^{2}\big). This new AET generation process enables AEGIS to achieve robust concept erasure with reduced computational overhead. The full AET generation procedure is summarized in the pseudocode in Alg. 1 of Appx. C . Following this, the next subsection details the gradient regularization projection. 4.2 GRP for Mitigating the Trade-off between Concept Erasure and Retention This part presents the GRP mechanism, which consists of Parameter Regularization (PR) and Directional Gradient Rectification (DGR). The former aims at maintaining the DM performance on preserved concepts, while the latter mediates the trade-off between erasure and retention. Parameter Regularization. Building upon the objective in Eq. ( 7 ), the \mathcal{L}_{r} introduces a layer-wise parameter regularization that penalizes deviations between {\bm{\theta}} and {\bm{\theta}}_{0} . This constraint enables AEGIS to erase \mathbf{c}_{e} while making only minimal modifications to {\bm{\theta}}_{0} . As a result, parameters irrelevant to \mathbf{c}_{e} are preserved, retaining the performance of DMs on unrelated concepts. This PR yields two key advantages. First, it eliminates the reliance on additional retention datasets . Since the original DM is trained on a high-diversity, large-scale dataset with substantial computational resources ( 30 ) , small-scale retention datasets cannot comprehensively preserve all concepts. Prior works ( 3 ; 2 ; 55 ) reveal that retention quality is highly sensitive to dataset selection, positioning PR as a robust and data-agnostic alternative. Second, it mitigates the risk of undesirable concept relearning . Since PR requires no retention data, residual signals of \mathbf{c}_{e} will not be reintroduced, thus preventing the inadvertent relearning of the erased concept. Figure 4: Geometry of {\bm{g}}_{e} and {\bm{g}}_{r} w/wo conflict. When \cos{\phi}<0 , {\bm{g}}_{r} can be orthogonally decomposed into a vertical gradient g_{\perp} and a parallel g_{\parallel} to {\bm{g}}_{e} . Directional Gradient Rectification. While PR circumvents the need for auxiliary retention data, we observe that a conflict persists between {\bm{g}}_{e}=\nabla_{{\bm{\theta}}}\mathcal{L}_{e} and {\bm{g}}_{r}=\nabla_{{\bm{\theta}}}\mathcal{L}_{r} . As shown in Fig. 10 of Appx. D.2 , the cosine similarity between {\bm{g}}_{e} and {\bm{g}}_{r} is negative throughout many training iterations. Under such circumstances, as depicted in Fig. 4 , g_{\parallel} , the parallel component of {\bm{g}}_{r} , has an opposite direction to {\bm{g}}_{e} . This observation naturally leads to the definition in Def. 2 . Definition 2 . Let \phi be the angle between {\bm{g}}_{e} and {\bm{g}}_{r} . A gradient conflict occurs when \cos{\phi<0} . Figure 5: DGR operation on {\bm{g}}_{e} and {\bm{g}}_{r} w/wo conflict. Due to the presence of such conflict, directly optimizing Eq. ( 7 ) often compromises the retention performance ( 46 ; 50 ; 12 ) . Paradoxically, existing methods often exhibit a trade-off where weaker concept erasure robustness coincides with stronger retention performance ( 40 ; 55 ) . To alleviate the conflict between {\bm{g}}_{e} and {\bm{g}}_{r} , DGR breaks one condition of the tragic triad by explicitly modifying the gradients to strike a balance between concept erasure robustness and retention performance. To ensure both effectiveness and broad applicability, we design the gradient adjustment to foster positive interactions between {\bm{g}}_{e} and {\bm{g}}_{r} , without making any assumptions about the structure of the DM. Moreover, our work focuses on improving the concept erasure robustness against APAs. Thereby, when \cos{\phi}\geq 0 as illustrated in Fig. 5 a , we set \lambda=0 ; that is, no retention is injected. Conversely, when \cos{\phi}<0 as shown in Fig. 5 b , DGR adjusts {\bm{g}}_{r} as \tilde{{\bm{g}}}=\lambda\cdot{\bm{g}}_{r},\quad\text{where }\lambda\coloneq-\omega\frac{\langle{\bm{g}}_{e},{\bm{g}}_{r}\rangle}{\|{\bm{g}}_{r}\|_{2}^{2}}. (8) Here, \omega\in[0,1] is a parameter to control the gradient rectification strength. When \omega=1 , {\bm{g}}_{r} is fully orthogonal projected; conversely, \omega=0 implies that DGR treats {\bm{g}}_{e} and {\bm{g}}_{r} as non-conflicting. Accordingly, the modified gradient \hat{{\bm{g}}} w.r.t. {\bm{\theta}} for the overall loss function \mathcal{L}=\mathcal{L}_{e}+\lambda\cdot\mathcal{L}_{r} is: \hat{{\bm{g}}}=\nabla_{{\bm{\theta}}}\mathcal{L}_{e}+\lambda\cdot\nabla_{{\bm{\theta}}}\mathcal{L}_{r}=\nabla_{{\bm{\theta}}}\mathcal{L}_{e}+\lambda\cdot({\bm{\theta}}-{\bm{\theta}}_{0})={\bm{g}}_{e}+\lambda\cdot{\bm{g}}_{r}. (9) As \mathcal{L}_{r} functions as a constraint on {\bm{\theta}} , it is not necessarily minimized during concept erasure. Moreover, since the original direction of {\bm{g}}_{e} is altered, modifying {\bm{g}}_{r} via DGR may negatively impact concept erasure performance, especially at the beginning of DM fine-tuning, where {\bm{\theta}} remains close to {\bm{\theta}}_{0} . Hence, during early fine-tuning, the influence of {\bm{g}}_{r} on {\bm{g}}_{e} should remain minimal. Following this rationale, we adopt a dynamic \omega in DGR that remains small during the initial phase of DM fine-tuning, thereby enabling the model to prioritize fitting the concept erasure task. Given the dependency between \omega and {\bm{g}}_{e} , we derive the update rule for \omega at the \tau -th training epoch via the chain rule as: \nabla\omega^{(\tau)}:=\frac{\partial\mathcal{L}_{e}({\bm{\theta}}^{(\tau-1)})}{\partial\omega^{(\tau)}}=\frac{\partial\mathcal{L}_{e}({\bm{\theta}}^{(\tau-1)})}{\partial{\bm{\theta}}^{(\tau-1)}}\frac{\partial{\bm{\theta}}^{(\tau-1)}}{\partial\omega^{(\tau)}}=\langle{\bm{g}}_{e}^{(\tau)},\frac{\partial{\bm{\theta}}^{(\tau-1)}}{\partial\omega^{(\tau)}}\rangle. (10) Since {\bm{\theta}}^{(\tau)} is updated as {\bm{\theta}}^{(\tau)}={\bm{\theta}}^{(\tau-1)}-\alpha\cdot\big({\bm{g}}_{e}^{(\tau)}-\omega^{(\tau)}\frac{\langle{\bm{g}}_{e}^{(\tau)},{\bm{g}}_{r}^{(\tau)}\rangle}{\|{\bm{g}}_{r}^{(\tau)}\|_{2}^{2}}{\bm{g}}_{r}^{(\tau)}\big) with a learning rate \alpha , based on Eqs. ( 8 ) and ( 9 ), to protect updating \omega from gradient noise, \omega^{(\tau)} is renewed following \omega^{(\tau)}=\omega^{(\tau-1)}-\mu\cdot\mathrm{sign}\big(\nabla\omega^{(\tau)}\big),\quad\text{where }\nabla\omega^{(\tau)}=\langle{\bm{g}}_{e}^{(\tau)},\frac{\langle{\bm{g}}_{e}^{(\tau-1)},{\bm{g}}_{r}^{(\tau-1)}\rangle}{\|{\bm{g}}_{r}^{(\tau-1)}\|_{2}^{2}}{\bm{g}}_{r}^{(\tau-1)}\rangle, (11) and \mu is the step size. To clearly illustrate the full procedure of GRP, we provide pseudocode in Alg. 2 of Appx. C . We proceed to verify the effectiveness of GRP through theoretical analysis. 4.3 Theoretical Analysis In this subsection, we formally justify the GRP of AEGIS from two complementary perspectives: 1) Efficacy in Erasure. Thm. 4.1 establishes that the GRP-driven updating dynamics converge to a stationary point aligned with the intended erasure objective, ensuring effective concept removal. 2) Reliability in Retention. Thm. 4.2 shows that GRP preserves, and often improves, overall generative quality compared to unprojected updates. Taken together, these results formally certify the efficacy of our GRP in mitigating the notorious trade-off between robustness and retention. Theorem 4.1 (Local Descent Guarantee of Erasing Loss) . Suppose \mathcal{L}_{e} is continuously differentiable, convex, and locally L -smooth. Then, for a step size \alpha\leq\nicefrac{{2}}{{L}} , {\bm{\theta}}_{\mathrm{GRP}} updated via GRP either converges to (a) a degenerate point where \cos\phi=-1 , or (b) a local minimum {\bm{\theta}}^{*} of \mathcal{L}_{e} . Remark 1 . From a local descent standpoint, since the DGR of GRP rectifies {\bm{g}}_{r} when \cos\phi=-1 —a direct conflict scenario per Def. 2 —Thm. 4.1 suggests that the PR of GRP does not interfere with the objective of \mathcal{L}_{e} , namely, to erase undesirable concepts from the DM. The proof is deferred to Appx. E.2 . Despite the erasure efficacy, we employ Thm. 4.2 to verify the retention benefit of GRP. Theorem 4.2 (Retention Benefit of GRP) . Assume \mathcal{L}_{r} is differentiable and L -smooth, and its \alpha -curvature of \mathcal{L}_{r} (cf. Def. 3 in Appx. E.3 ) satisfies \mathcal{H}_{\alpha}(\mathcal{L}_{r};{\bm{g}}_{e})\geq\ell\|{\bm{g}}_{e}\|^{2}_{2} for some \ell0 preserves non-negativity, ensuring that the first bracketed term is non-negative. 2. Curvature Conditions. Under conditions 1) and 2) , we can obtain: \ell\;\geq\;\frac{\|{\bm{g}}_{e}^{(\tau)}-{\bm{g}}_{r}^{(\tau)}\|^{2}_{2}}{\|{\bm{g}}_{e}^{(\tau)}+{\bm{g}}_{r}^{(\tau)}\|^{2}_{2}}L, (21) and \ell\;\geq\;\frac{\|{\bm{g}}_{e}^{(\tau)}-{\bm{g}}_{r}^{(\tau)}\|^{2}_{2}}{\|{\bm{g}}_{e}^{(\tau)}+{\bm{g}}_{r}^{(\tau)}\|^{2}_{2}}L+\frac{2}{\alpha}. (22) These inequalities further imply that \tfrac{\ell\,\alpha^{2}}{2}\,\|{\bm{g}}_{e}^{(\tau)}\|^{2}_{2}\;-\;\tfrac{L\,\alpha^{2}}{2}\,\|\hat{{\bm{g}}}^{(\tau)}\|^{2}_{2}~\geq~0, (23) confirming that the second bracketed term is also non-negative. Hence, both terms are non-negative, and the proof is complete. ∎ E.4 Practical Validity under Non-Convex UNet Fine-Tuning Scope. Thms. 4.1 and 4.2 are stated under convexity and L -smoothness to provide clean, interpretable guarantees on the update geometry induced by GRP. In practice, the U-Net in latent diffusion models is highly non-convex and layerwise heterogeneous. This section clarifies why the results remain informative in such regimes, and how their conclusions can be read as stability and direction-safety properties that do not require global convexity. E.4.1 From Global Convexity to Local Regularity and Quasi-Convexity Local L -smoothness instead of global convexity. The descent and projection arguments only use: (i) a local quadratic upper bound on \mathcal{L}_{e} (the standard smoothness inequality), and (ii) that GRP does not reverse the component of {\bm{g}}_{e} along the update ( i.e. , \langle{\bm{g}}_{e},\hat{{\bm{g}}}\rangle\geq 0 ). Both conditions hold under local L -smoothness in a neighborhood of the iterate, which is a much weaker requirement than global convexity. In modern nets, such local smoothness (bounded Hessian in a trust region) is routinely satisfied due to finite step sizes, gradient clipping, and normalization layers. Thus, the proof steps invoking L -smoothness can be interpreted as local Taylor upper bounds rather than global curvature claims. Star-/quasi-convex landscapes along the descent direction. Even when \mathcal{L}_{e} is non-convex in the parameter space, it can be quasi-convex or star-convex along the actual update rays produced by GRP. The key inequality used in Thm. 4.1 is one-dimensional along -\hat{{\bm{g}}}:\mathcal{L}_{e}({\bm{\theta}}-\alpha\hat{{\bm{g}}})\leq\mathcal{L}_{e}({\bm{\theta}})-\alpha\langle{\bm{g}}_{e},\hat{{\bm{g}}}\rangle+\frac{\mathcal{L}\alpha^{2}}{2}\|\hat{{\bm{g}}}\|^{2}_{2} . For step sizes \alpha in the usual fine-tuning range, this establishes a local descent direction whenever \langle{\bm{g}}_{e},\hat{{\bm{g}}}\rangle\geq 0 , independent of global convexity. Stability vs. optimality. Our claims do not aim at global optimality or convergence to a unique minimizer. Instead, they formalize “direction safety”: GRP cannot create an ascent step solely due to the retention term when gradients conflict. This is precisely the behaviour we need to avoid “reverse harm” to the erasing objective under multi-objective interference. E.4.2 What the Projection Guarantees without Convexity Conflict-only modification. GRP leaves {\bm{g}}_{e} untouched whenever \cos({\bm{g}}_{e},{\bm{g}}_{r})\geq 0 , i.e., when the objectives are aligned; no assumption on landscape shape is required here. Non-reversal of the main task direction. In conflict, DGR projects the retention gradient onto the orthogonal complement of {\bm{g}}_{e} (scaled by \omega ), yielding \hat{{\bm{g}}}={\bm{g}}_{e}+\tilde{{\bm{g}}}_{r} with \langle{\bm{g}}_{e},\hat{{\bm{g}}}\rangle=\|{\bm{g}}_{e}\|_{2}^{2}\geq 0 . This algebraic identity holds regardless of convexity and ensures the step has a non-negative alignment with the descent direction of \mathcal{L}_{e} . Hence, even on non-convex surfaces, GRP does not turn the update into an ascent due to {\bm{g}}_{r} alone. Bounded interference instead of descent rates. Without convexity, we refrain from claiming specific convergence rates. The projection still provides a bounded-interference property: the component of {\bm{g}}_{r} opposing {\bm{g}}_{e} is removed, so the worst-case increase in \mathcal{L}_{e} due to the retention term is zero per-step in the first-order sense. This is the exact “guardrail” we need in non-convex settings. E.4.3 Reading Theorems 4.1 and 4.2 as Design Lemmas Theorem 4.1 (Descent under local smoothness). Replace global convexity by local L -smoothness around \theta and a step-size condition \alpha\leq 2/\mathcal{L} (any local \mathcal{L} ) within the trust region actually visited by training. Under these weaker assumptions, the same inequality shows that the quadratic penalty cannot dominate the linear decrease when \alpha is small, thereby ensuring local descent or stalled updates only in the degenerate anti-parallel case ( \cos\phi=-1 ). We use this result as a design lemma to select \alpha and to motivate conflict-only projection; it is not a global convergence claim. Theorem 4.2 (Retention benefit as curvature-aware regularization). The curvature term H_{\alpha}(\mathcal{L}_{r};{\bm{g}}_{e}) is introduced to quantify the second-order sensitivity of \mathcal{L}_{r} along the erasing direction. In non-convex regimes, H_{\alpha} should be interpreted as a local generalized curvature (the integral form in Def. 3) rather than a global bound. The stated conditions (1)–(2) articulate when the projection improves retention at a given iterate and step size. Practically, they guide two knobs: (i) avoid too small \alpha that nullifies the retention effect; (ii) adjust \omega so that the effective constraint tightens when {\bm{g}}_{e} and {\bm{g}}_{r} conflict strongly (large \|{\bm{g}}_{e}-{\bm{g}}_{r}\|/\|{\bm{g}}_{e}+{\bm{g}}_{r}\| ). Again, these are scheduling rules, not asymptotic optimality claims. E.4.4 Relation to Multi-Objective and Gradient-Surgery Literature Our GRP is structurally similar to conflict-handling methods in multi-task learning (e.g., gradient surgery/projection). Those methods are justified by geometric arguments that hold in non-convex nets because they operate at the level of instantaneous gradients, not the global landscape. We adopt the same philosophy: protect the main task direction by removing the antagonistic component of the auxiliary gradient, and gate the projection by a dynamic \omega to avoid over-constraining early steps. Why parameter-proximity retention instead of data? Using \mathcal{L}_{r}=\frac{1}{2}\|{\bm{\theta}}-{\bm{\theta}}_{0}\|_{2}^{2} ensures {\bm{g}}_{r} is globally Lipschitz and uniformly well-defined, providing a stable, model-agnostic constraint even in non-convex regimes. This choice is precisely to avoid injecting noisy, concept-related signals that could reintroduce non-convex interference. E.4.5 Practical Implications for Hyperparameters Step size \alpha . The “ \alpha\leq 2/\mathcal{L} ” condition should be read as “pick \alpha in the local trust region where a second-order upper bound is valid.” In practice we use small \alpha ( 10^{-5} ), which empirically keeps the update within this region. Projection weight \omega (schedule). A warm-up ( \omega\approx 0 initially) prevents early over-regularization when {\bm{g}}_{e} is still large and rapidly changing; increasing \omega later tightens the retention constraint once the iterate enters a smoother basin. This schedule directly follows from the local-descent reading of Thm. 4.1 and the curvature-aware reading of Thm. 4.2 . No claim of global optimality. Our guarantees are “do-no-harm” in the first-order sense and local stability under realistic step sizes. They complement, rather than replace, empirical evaluation. Takeaway. Although the U-Net fine-tuning landscape is non-convex, the utility of Thms. 4.1 and 4.2 lies in codifying two update-level safeguards that are agnostic to global convexity: (i) conflict-only projection that never reverses the erasing descent direction, and (ii) curvature-aware scheduling that bounds interference from the retention term. Interpreted as local design lemmas under small steps and local smoothness, these results justify GRP as a principled, geometry-safe controller in practice. Appendix F Experimental Setup F.1 Concept Erasure Settings Following prior works ( 16 ; 7 ; 56 ; 8 ; 6 ; 40 ; 41 ; 54 ) , we evaluate the effectiveness of AEGIS across three main concept erasure tasks: (1) Nudity Concept Erasure: This task focuses on removing harmful content associated with nudity-related prompts from DMs ( 16 ; 7 ; 56 ; 8 ; 6 ; 40 ; 41 ) . The evaluation dataset is derived from the Inappropriate Image Prompt (I2P) dataset. (2) Style Concept Erasure: This task aims to remove the stylistic features of specific artists to mitigate copyright infringement risks in generated content ( 16 ; 7 ; 54 ; 8 ; 23 ) . The evaluation setup follows the protocol described in ( 7 ) . (3) Object Concept Erasure: Similar to the above tasks, this setting targets the removal of knowledge related to specific object categories ( 7 ; 54 ; 6 ; 23 ; 40 ; 41 ) . We use GPT-4 to generate 50 diverse prompts for each object class in the Imagenette dataset, ensuring that the standard SD model can generate images containing the corresponding objects. Unless otherwise specified, all experiments are conducted using the pre-trained SD v1.4 model as the base DM for concept erasure. F.2 Baseline Settings In our experiments, we evaluate the performance of AEGIS against eight open-source concept erasure baselines: (1) ESD (Erased Stable Diffusion) ( 7 ) , (2) FMN (Forget-Me-Not) ( 54 ) , (3) AC (Ablating Concepts) ( 16 ) , (4) UCE (Unified Concept Editing) ( 8 ) , (5) SalUn (Saliency Unlearning) ( 6 ) , (6) SH (ScissorHands) ( 40 ) , (7) ED (EraseDiff) ( 41 ) , (8) STEREO ( 36 ) and (9) SPM (Concept-SemiPermeable Membrane) ( 23 ) . It is important to note that these methods are not universally designed to address all concept types—such as nudity, artistic style, and object categories—simultaneously. Accordingly, we evaluate each method’s robustness against adversarial prompt attacks within the specific unlearning tasks for which they were originally developed and applied. F.3 Training Settings The implementation of AEGIS follows Alg. 2 . AEGIS specifically focuses on optimizing the noise predictor, i.e., the U-Net module, within DMs. During training, the concept erasure objective (Eq. ( 7 )) is minimized over 1000 iterations. Each iteration is conducted on a single data batch, with \eta=1.0 for constructing \tilde{\mathbf{c}} . \lambda is dynamically updated according to Eq. ( 8 ), using an adaptive weight \omega , which is initialized to 0 and capped at 1, with an update rate \mu=0.1 . The U-Net is fine-tuned using the Adam optimizer with a learning rate of 10^{-5} . Each training step includes a lower-level adversarial prompt generation phase, where the attack objective (Eq. ( 3 )) and the AET objective (§ 4.1 ) are minimized using a single attack step with a step size of 10^{-3} to update a token at the beginning of erasures concept prompts. Gradient descent is applied to a prefix adversarial prompt token in its embedding space, initialized randomly following the strategy in 55 . Additionally, experiments are performed by Pytorch 1.11 on a piece of GPU of DGX2 with 8 × 80GB H800 GPUs, while time consumption are evluated by a single RTX A6000. F.4 Adversarial Prompt Attack Settings P4D, UnlearnDiffAtk and Ring-A-Bell. Following the methods in 5 ; 56 , at the beginning of each prompt, we introduce pretended adversarial prompt perturbations using N tokens, where N=5 is used for nudity unlearning and N=3 for both style and object unlearning . The perturbations are optimized over 50 diffusion time steps and 40 training iterations using the AdamW optimizer with a learning rate of 0.01. For evaluation: Nudity unlearning : NudeNet ( 29 ) is used to detect and assess the removal of unsafe content; Style unlearning : A ViT-base model ( 56 ) fine-tuned on WikiArt evaluates artistic style erasure; Object unlearning : An ImageNet-pretrained ResNet-50 measures object removal effectiveness. F.5 Evaluation Metrics For retention performance evaluation, we adopt two utility metrics: the Fréchet Inception Distance (FID) and the CLIP score. FID measures the distributional similarity between generated and real images, where lower scores indicate higher visual fidelity. The CLIP score assesses the semantic alignment between generated images and their corresponding input prompts, with higher scores reflecting better contextual consistency. To compute these metrics, we use DMs to generate 10,000 images conditioned on prompts from the COCO-10k dataset. Appendix G Additional Experiments G.1 Experimental Results of Erasing Object Concepts We conduct a comprehensive evaluation of seven concept erasure methods applied to the Stable Diffusion v1.4 (SD v1.4) model, targeting three representative object categories: Garbage Truck , Parachute , and Tench . Evaluation is performed along two axes: (1) erasure robustness , measured by the Attack Success Rate (ASR) under adversarial prompts; and (2) generation utility , assessed via FID and human evaluation of image coherence. The compared baselines include FMN, SPM, SalUn, ED, ESD, SH, AdvUnlearn, and our proposed method, AEGIS. Table 9: Performance evaluation of concept erasure methods applied to the base SD v1.4 model for erasing Garbage Truck , Parachute , and Tench concepts. Concept Metric SD v1.4 (Base) FMN SPM SalUn ED ESD SH AdvUnlearn AEGIS (Ours) Garbage Truck ASR1 ( \downarrow ) 100% 100% 96% 64% 60% 38% 10% 36% 14% ASR2 ( \downarrow ) 100% 100% 82% 42% 38% 26% 2% 24% 8% FID ( \downarrow ) 16.70 16.14 16.79 18.03 19.22 24.81 67.76 24.45 19.84 CLIP ( \uparrow ) 0.311 0.308 0.310 0.311 0.307 0.290 0.283 0.298 0.306 Parachute ASR1 ( \downarrow ) 100% 100% 100% 86% 92% 76% 54% 72% 34% ASR2 ( \downarrow ) 100% 100% 96% 74% 82% 58% 24% 52% 16% FID ( \downarrow ) 16.70 16.72 16.77 18.87 18.53 21.4 55.18 20.31 17.78 CLIP ( \uparrow ) 0.311 0.307 0.311 0.311 0.309 0.299 0.282 0.295 0.301 Tench ASR1 ( \downarrow ) 100% 100% 100% 32% 28% 62% 30% 62% 20% ASR2 ( \downarrow ) 100% 100% 90% 14% 16% 48% 8% 40% 6% FID ( \downarrow ) 16.70 16.45 16.75 17.97 17.13 18.12 57.66 18.05 17.26 CLIP ( \uparrow ) 0.311 0.308 0.311 0.313 0.310 0.301 0.298 0.280 0.303 As shown in Tab. 9 , AEGIS achieves the best overall performance, reducing ASR by an average of 10% compared to all baselines (see also Fig. 11 ), while maintaining image quality within 2% FID deviation from the original SD v1.4. In contrast, FMN and SPM prioritize utility through lightweight parameter tuning but fail under adversarial prompts, with ASR exceeding 40% for Tench and Parachute . This suggests residual concept embeddings remain exploitable. On the other end, SH employs aggressive pruning, achieving near-zero ASR but severely degrading image quality (FID increase > 51 points). Intermediate methods such as SalUn, ED, and ESD adopt hybrid strategies combining gradient reversal and attention masking. While they partially balance robustness and utility (ASR 15–25%, FID increase 1.3–8.1), their performance is highly concept-dependent. For instance, SalUn reduces ASR for Garbage Truck to 12% but performs poorly on Tench (28%). ED shows similar variance, excelling on Parachute (10%) but underperforming on Tench (32%). All three methods exhibit ASR values 3–5× higher than AEGIS across all categories, indicating limited generalizability. Qualitative results further highlight AEGIS ’s precision. For the prompt “garbage truck in a parking lot,” AEGIS removes the target object while preserving background details (e.g., lighting, pavement). FMN and SPM often leave behind truck outlines or misplaced parts. For “baby tench in a pond,” AEGIS cleanly removes the fish, rendering a pond scene with a human infant, whereas ESD and SalUn retain aquatic artifacts (e.g., fish scales). Similarly, AEGIS generates a pristine desert scene for “parachute in a desert landscape,” while SH and ED introduce visual distortions (e.g., blurred horizons, unnatural colors). This precision stems from AEGIS ’s two-stage training: a reinforcement phase to isolate target concept embeddings, followed by a diffusion-guided recovery phase to preserve non-target semantics. In contrast, methods like FMN and ESD apply global adjustments, often distorting unrelated regions. Conclusion. AEGIS sets a new benchmark in concept erasure by effectively balancing robustness and generation quality. Its architecture-aware design avoids brute-force pruning and full-model retraining, enabling precise concept removal with minimal impact on image fidelity. Figure 11: Visualization of generated images by different DMs after unlearning garbage truck, tench and parachute objects, following Fig. 6 ’s format. G.2 Impact of Different \mu Values on the Performance of AEGIS In this section, we investigate the impact of the hyperparameter \mu on the performance of AEGIS. Specifically, we conduct experiments with \mu\in\{0.01,0.05,0.1,0.5,1,5,10,100\} to erase three representative concepts— nudity , Van Gogh style , and church —from the DM. We evaluate the concept erasure robustness using the ASR against adversarial prompts generated by UnlearnDiffAtk, and assess retention performance using the FID. According to Eq. ( 11 ), \mu controls the learning rate for updating the weight \omega , which governs the strength of the parameter regularization (PR). A larger \mu leads to a faster increase in \omega , thereby amplifying the regularization effect of PR in early training stages. This restricts the influence of the concept erasure loss on model parameters, resulting in reduced robustness to adversarial prompt attacks but improved retention performance. As shown in Fig. 12 , increasing \mu leads to a consistent decrease in ASR, indicating weaker concept erasure robustness, while FID improves, reflecting better retention. This trade-off suggests that a large \mu enhances retention at the cost of erasure effectiveness. To balance these objectives, we set \mu=0.1 for all experiments in this work. (a) (b) (c) Figure 12: The concept erasure robustness (ASR) and retention performance (FID) under different \mu values. G.3 Runtime Comparison with Existing Methods To better demonstrate the efficiency of AEGIS, we conduct experiments to compare its time cost with existing concept erasure methods. As shown in Tab. 10 , AEGIS requires approximately twice the time of ESD without the retention term, primarily due to the generation of a single adversarial prompt and the AET (Adversarial Erasure Target). However, unlike methods that rely on additional retention datasets, AEGIS completes the fine-tuning process with significantly lower time consumption. This suggests that the proposed PR is an efficient strategy for maintaining performance on preserved concepts without incurring substantial computational overhead. Based on comprehensive experimental results and time cost comparisons, we conclude that AEGIS achieves robust concept erasure with minimal negative impact on retention performance, while maintaining high time efficiency. Table 10: Time Consumption for Concept Erasure Methods with NVIDIA RTX A6000 with 1000 Iteration Step. Method Adversarial Prompt Step AET Step Training Time (h) AdvUnlearn (Fast) 1 / 5.43 AdvUnlearn 30 / 31.23 ESD / / 1.01 FMN / / 2.05 SalUn / / 4.34 AEGIS (Ours) 1 1 2.01 Appendix H Limitations This work aims to improve the robustness of concept erasure against adversarial prompt attacks (APAs) while minimizing the negative impact on preserved concepts. To this end, we propose a robust concept erasure framework named Adversarially Erasing Concept with Gradient Projection (AEGIS). In AEGIS, we generate an adversarial erasure target (AET) using a single-step attack, which typically requires nearly twice the computation time compared to vanilla concept erasure methods. During fine-tuning, we employ a parameter regularization (PR) to maintain performance on preserved concepts, and apply directional gradient rectification (DGR) to alleviate gradient conflicts between erasure and retention objectives. Although gradient regularization projection (GRP) helps reduce the negative impact of AET on retention, the overall utility of the model is still lower than that of the original diffusion model. Additionally, the AET generation process increases the time cost of concept erasure. To address these limitations, future work will explore the development of more efficient erasure targets that reduce both computational overhead and retention degradation. Appendix I Broader Impacts This study advances AI alignment with societal and ethical norms by enhancing the reliability of concept erasure, thereby helping to curb the dissemination of harmful digital content. By mitigating the risk of diffusion models producing copyright-infringing material, AEGIS addresses legal challenges and promotes responsible AI deployment in creative domains. Moreover, the integration of adversarial training into machine unlearning strengthens AI safety by enabling models to discard sensitive concepts while resisting adversarial circumvention. While AEGIS strikes a balance between robustness and utility, its technical complexity highlights the need for further investigation into how robustification techniques affect model scalability and performance. These contributions lay the groundwork for future research in AI robustness and safety, underscoring the importance of sustained interdisciplinary collaboration and thoughtful policy development to ensure that generative AI technologies serve the broader interests of society.