EMMA : Concept Erasure Benchmark with Comprehensive Semantic Metrics and Diverse Categories
Lu Wei Yuta Nakashima Noa Garcia · 2025
Read EMMA : Concept Erasure Benchmark with Comprehensive Semantic Metrics and Diverse Categories by Lu Wei Yuta Nakashima Noa Garcia (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-17320.txt.
Introduction
Lu Wei, Yuta Nakashima, and Noa Garcia built EMMA, a tougher exam for tools that claim to erase a concept from an image generator. Erasure is meant to strip out things like a celebrity's face or a living artist's style without retraining the whole model. Past exams used few concepts and plain, direct prompts, which made weak tools look strong.
EMMA grades five erasure methods over 13 measures in five domains, from objects to copyrighted styles. It probes sneaky prompts that hint at the banned concept, lookalike concepts that should survive, and bias by gender and ethnicity. Every method stumbled on hinted prompts and on lookalikes. Some even made bias worse than the original model, a warning for anyone shipping erasure as a safety fix.
Catalog record
| CREATOR | Lu Wei Yuta Nakashima Noa Garcia |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-17320
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EMMA : Concept Erasure Benchmark with Comprehensive Semantic Metrics and Diverse Categories Lu Wei Yuta Nakashima Noa Garcia Abstract The widespread adoption of text-to-image (T2I) generation has raised concerns about privacy, bias, and copyright violations. Concept erasure techniques offer a promising solution by selectively removing undesired concepts from pre-trained models without requiring full retraining. However, these methods are often evaluated on a limited set of concepts, relying on overly simplistic and direct prompts. To test the boundaries of concept erasure techniques, and assess whether they truly remove targeted concepts from model representations, we introduce EMMA , a benchmark that evaluates five key dimensions of concept erasure over 12 12 metrics. EMMA goes beyond standard metrics like image quality and time efficiency, testing robustness under challenging conditions, including indirect descriptions, visually similar non-target concepts, and potential gender and ethnicity bias, providing a socially aware analysis of method behavior. Using EMMA , we analyze five concept erasure methods across five domains (objects, celebrities, art styles, NSFW, and copyright). Our results show that existing methods struggle with implicit prompts ( i.e . generating the erased concept when it is indirectly referenced) and visually similar non-target concepts ( i.e . failing to generate non-targeted concepts resembling the erased one), while some amplify gender and ethnicity bias compared to the original model. Erasing… Dog MACE UCE ESD CA FMN An image of a dog. A loyal and energetic companion with a wagging tail and a playful demeanor, often eager for a game or a walk. Erasing… Vincent van Gogh MACE UCE ESD CA FMN A photo in the style of Vincent van Gogh. Passionate Arles and Saint-Rémy scenes depicting starry nights, sunflowers, and cypress trees with impasto brushwork, vibrating complementary colors, and psychological intensity. Erasing… Converse MACE UCE ESD CA FMN An image with the Converse logo. Black canvas high-tops feature the iconic star circle emblem. Figure 1 : Concept erasure methods break under pressure. When challenged with descriptive prompts, erased concepts like dog , Vincent van Gogh , and Converse resurface in the generated images. The EMMA benchmark systematically explores this and other limitations of current concept erasure techniques through a comprehensive evaluation framework. 1 Introduction With the growing adoption of text-to-image (T2I) generative models, including GAN ( 6 ; 20 ; 52 ) , autoregressive ( 33 ; 8 ; 48 ) , and diffusion-based approaches ( 40 ; 38 ; 14 ) , increasing attention has been drawn to their associated societal risks ( 17 ) , such as privacy violations ( 46 ) , social biases ( 23 ; 51 ) , and copyright infringement ( 18 ) . While these issues are often rooted, although not exclusively, in poor training data collection practices, retraining models from scratch whenever a problematic sample is identified is prohibitively costly and inefficient. Instead, machine unlearning has emerged as an alternative, enabling target removal of undesired data from a trained model without full retraining. In the context of T2I generation, unlearning is typically used to remove specific visual concepts, such as concrete objects ( 19 ; 9 ; 10 ) , celebrities ( 22 ; 54 ) , named art styles ( 11 ) , or toxic or inappropriate content ( 50 ; 49 ) . The techniques proposed to unlearn such visual concepts are collectively known as concept erasure and are commonly evaluated across four dimensions: erasing ability ( 41 ; 27 ) , which measures whether the target concept is effectively forgotten; retaining ability ( 55 ; 27 ) , which measures whether the model preserves its performance on non-target concepts; efficiency ( 27 ) , which computes the time required for unlearning; and image quality ( 41 ; 55 ; 27 ) , which measures the generated image-text fidelity performance. However, as summarized in Table 1 , evaluation protocols are fragmented among different methods and benchmarks, lacking a comprehensive framework for consistent assessment across the different criteria ( 41 ; 27 ; 55 ; 45 ; 56 ; 26 ) . Table 1 : Concept erasure (CE) evaluation tools in methods and benchmarks papers. Domains are split into object (O), celebrity (C), art style (A), NSFW (N), and copyright (CR). N/A indicates that the number of concepts is not reported in the original paper. Domains Evaluation dimensions Number of concepts Erasing ability (EA) Retaining ability (RA) Efficiency Bias Image Name Year O C A N CR Explicit Implicit Random Similar Unlearning Inference Gender Ethnicity quality Methods CA ( 19 ) 2023 <5 - - - 4 ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ESD ( 9 ) 2023 10 - 5 1 - ✓ ✗ ✓ ✗ ✗ ✗ ✗ ✗ ✓ FMN ( 54 ) 2024 <10 <5 <5 1 - ✓ ✗ ✓ ✗ ✗ ✗ ✗ ✗ ✗ UCE ( 10 ) 2024 10 - 1000 1 - ✓ ✗ ✓ ✗ ✗ ✗ ✓ ✓ ✓ MACE ( 22 ) 2024 10 100 100 1 - ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✓ SPM ( 25 ) 2024 6 - 2 1 - ✓ ✗ ✓ ✗ ✓ ✓ ✗ ✗ ✓ TRCE ( 5 ) 2025 - - - 7 - ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ SAGE ( 57 ) 2025 - - 2 7 - ✓ ✓ ✓ ✗ ✓ ✗ ✗ ✗ ✓ Benchmarks SLD ( 41 ) 2023 - - - 7 - ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ CI ( 30 ) 2024 10 2 6 1 - ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ Ring-A-Bell ( 45 ) 2024 - - - 2 - ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ UnlearnDiffAtk ( 56 ) 2024 4 - 1 3 - ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ UnlearnCanvas ( 55 ) 2024 60 - 20 - - ✓ ✗ ✓ ✗ ✓ ✗ ✗ ✗ ✓ CPDM ( 26 ) 2024 - N/A 100 - 200 ✓ ✗ ✓ ✗ ✗ ✗ ✗ ✗ ✓ MFC ( 39 ) 2025 4 - 1 1 - ✗ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✓ HUB ( 27 ) 2025 - 10 10 3 10 ✓ ✗ ✓ ✓ ✓ ✗ ✗ ✗ ✓ EMMA (ours) 2025 79 50 40 7 30 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ We provide a comprehensive analysis of concept erasure in T2I generation by introducing EMMA , a benchmark for concept E rasure with co M prehensive se MA ntic metrics and diverse categories. We show that current evaluation practices offer only a simplistic view of concept erasure methods’ performance, limited in both the concepts they evaluate and the depth of their metrics. For example, the assessment of erasing ability is often limited to prompts containing the explicit name of the erased concept ( 22 ; 9 ; 10 ; 54 ; 19 ; 25 ) , such as “a photo of a dog”. We prove this approach insufficient, as models can still generate the target concept when faced with more challenging, descriptive prompts like “a loyal and energetic companion with a wagging tail and a