M-ErasureBench: A Comprehensive Multimodal Evaluation Benchmark for Concept Erasure in Diffusion Models

Ju-Hsuan Weng, Jia-Wei Liao, Cheng-Fu Chou, Jun-Cheng Chen · 2025

Read M-ErasureBench: A Comprehensive Multimodal Evaluation Benchmark for Concept Erasure in Diffusion Models by Ju-Hsuan Weng, Jia-Wei Liao, Cheng-Fu Chou, Jun-Cheng Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-22877.txt.

Introduction

Ju-Hsuan Weng, Jia-Wei Liao, Cheng-Fu Chou, and Jun-Cheng Chen benchmark how well image generators can be made to forget a concept. Text-to-image models sometimes produce harmful or copyrighted content, so erasure methods try to remove ideas. Nearly all testing, however, only attacks the model through text prompts. Real uses such as image editing feed information through other channels that can smuggle the erased concept back in.

Their benchmark, M-ErasureBench, tests erasure through text prompts, learned embeddings, and inverted image latents, under white-box and black-box access. Current methods hold up against prompts but mostly fail against the other channels. Erased concepts reappear at very high rates in the white-box case. The authors answer with IRECE, an inference-time add-on that finds the concept through cross-attention and disturbs the matching latents during generation, cutting reappearance while keeping image quality.

Catalog record

CREATORJu-Hsuan Weng, Jia-Wei Liao, Cheng-Fu Chou, Jun-Cheng Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-22877

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

M-ErasureBench: A Comprehensive Multimodal Evaluation Benchmark for Concept Erasure in Diffusion Models Ju-Hsuan Weng, Jia-Wei Liao, Cheng-Fu Chou, Jun-Cheng Chen Abstract Text-to-image diffusion models may generate harmful or copyrighted content, motivating research on concept erasure. However, existing approaches primarily focus on erasing concepts from text prompts, overlooking other input modalities that are increasingly critical in real-world applications such as image editing and personalized generation. These modalities can become attack surfaces, where erased concepts re-emerge despite defenses. To bridge this gap, we introduce M-ErasureBench, a novel multimodal evaluation framework that systematically benchmarks concept erasure methods across three input modalities: text prompts, learned embeddings, and inverted latents. For the latter two, we evaluate both white-box and black-box access, yielding five evaluation scenarios. Our analysis shows that existing methods achieve strong erasure performance against text prompts but largely fail under learned embeddings and inverted latents, with Concept Reproduction Rate (CRR) exceeding 90% in the white-box setting. To address these vulnerabilities, we propose IRECE (Inference-time Robustness Enhancement for Concept Erasure), a plug-and-play module that localizes target concepts via cross-attention and perturbs the associated latents during denoising. Experiments demonstrate that IRECE consistently restores robustness, reducing CRR by up to 40% under the most challenging white-box latent inversion scenario, while preserving visual quality. To the best of our knowledge, M-ErasureBench provides the first comprehensive benchmark of concept erasure beyond text prompts. Together with IRECE, our benchmark offers practical safeguards for building more reliable protective generative models. 1 Introduction The contemporary diffusion models [ 37 , 16 , 33 , 8 , 32 , 30 , 36 ] have demonstrated remarkable progress in high-quality and versatile content generation, supporting tasks such as image synthesis [ 7 , 17 , 33 , 49 ] , image editing [ 28 , 15 , 4 , 18 ] , personalized generation [ 10 , 35 ] , and style transfer [ 6 , 51 ] . However, training on large-scale uncurated datasets makes them prone to reproducing copyrighted [ 3 , 14 ] or inappropriate content [ 21 ] . Retraining on filtered datasets offers a direct solution, but it is costly and often degrades generative quality [ 8 ] . Recent studies therefore explore concept erasure [ 9 , 48 , 11 , 12 , 23 , 20 , 27 , 2 , 24 , 40 , 46 , 45 ] , which aims to prevent text-to-image diffusion models from generating harmful or copyrighted content by suppressing specific concepts through fine-tuning cross-attention layers of diffusion models without retraining from scratch. Despite their effectiveness under text prompts, existing methods reveal critical weaknesses. In practice, users often rely on learned embeddings from personalization techniques [ 10 , 35 ] or noisy latent from inversion methods [ 7 , 29 ] , which fall outside the assumptions of text-based concept erasure. Our analysis shows that while suppression is reliable for basic prompts, concepts frequently re-emerge with learned embeddings or inverted latents, reaching Concept Reproduction Rate (CRR) over 90% in white-box setting with unconditional prompt. Geroge et al. [ 13 ] also found that the erased concepts can be revived through fine-tuning the model with a few samples. This indicates that current methods primarily disrupt text–image alignment rather than fully removing concepts. Moreover, although adversarial prompt attacks [ 5 , 31 ] expose vulnerabilities, defenses based on adversarial training [ 20 ] remain limited to textual inputs, underscoring the need to explore robustness beyond the text space. This raises our central research question: How robust are concept erasure methods across different input modalities, and can their vulnerabilities be mitigated without retraining? Motivated by these limitations, we introduce M-ErasureBench, a novel multimodal evaluation benchmark that systematically evaluates the robustness of the concept erasure methods across three representative input settings for text-to-image diffusion models: text prompts, learned embeddings, and inverted latents, under both white-box and black-box settings, respectively. With the extensive evaluations from M-ErasureBench, the results demonstrate that the state-of-the-art concept erasure methods are still vulnerable to various inference-time attack methods. Building upon these insights, we further propose Inference-time Robustness Enhancement for Concept Erasure (IRECE), a plug-and-play module that localizes target concepts via cross-attention and selectively perturbs associated latents during denoising without retraining. Our contributions are summarized as follows: • We introduce M-ErasureBench, the first comprehensive multimodal evaluation benchmark for concept erasure of text-to-image diffusion models, covering three input modalities: text prompts, learned embeddings, and inverted latents, under both white- and black-box settings. • Our study reveals that although existing methods are effective for text prompts, they largely fail under learned embeddings and latent inversion, with CRR exceeding 90% in the white-box setting with the unconditional prompt when adversarial training is absent. This indicates that current approaches disrupt alignment rather than fully erasing concepts. • We propose IRECE, a plug-and-play module that requires no retraining. By localizing target concepts via cross-attention and perturbing associated latents, IRECE reduces CRR by up to 40% under the white-box setting with the unconditional prompt, while maintaining visual quality. This provides a practical enhancement for safer generative models. Figure 1 : Illustration of ErasureBench and IRECE. We evaluate concept-erasure methods on text prompts, learned embeddings, and inverted latents. While erased models suppress concepts under simple text prompts, they fail in the other two settings, where the target concept (“airplane”) re-emerges. Our proposed IRECE module, applied at inference time, restores robustness and removes the target concept without retraining. 2 Related Works 2.1 Concept Erasure in Diffusion Models Text-to-image diffusion models [ 33 , 32 , 30 , 36 ] have achieved impressive performance in high-quality image generation, but their reliance on large-scale web data introduces risks of reproducing copyrighted or harmful content [ 3 , 14 , 21 ] . Retraining on curated datasets can mitigate these risks, yet it is costly and often compromises generation quality [ 8 ] , motivating research on concept erasure. Most existing approaches [ 9 , 48 , 11 , 12 , 23 , 20 , 27 , 2 ] fine-tune the UNet [ 34 ] , particularly its cross-attention layers [ 43 ] , and can be broadly categorized into anchor-based and anchor-free methods. Anchor-based approaches remove concepts by aligning their embeddings with substitutes (e.g., mapping “weapon” to “tool”), while anchor-free approaches steer generation away from the target concept without replacement. Representative methods include ESD [ 11 ] , which applies negative guidance to suppress target concepts; UCE [ 12 ] , which edits cross-attention projections in closed form to remove specific alignments; and Receler [ 20 ] , which enhances robustness via adversarial training and parameter-efficient fine-tuning [ 19 ] , providing tunable control over erasure strength. While effective for textual prompts, these text-driven methods struggle to generalize to non-textual inputs such as learned embeddings or inverted latents. EraseBench [ 1 ] provides a valuable benchmark for assessing post-erasure side effects within text prompts, but it does not evaluate robustness across different input modalities. This motivates our proposed framework, which systematically examines the performance of concept erasure methods under diverse modality conditions. 2.2 Adversarial Attacks on Concept Erasure Recent studies [ 41 , 50 , 42 , 31 , 42 , 47 ] reveal that concept erasure methods remain vulnerable to adversarial attacks,

Read the complete text