Neighbor-Aware Localized Concept Erasure in Text-to-Image Diffusion Models

Zhuan Shi Alireza Dehghanpour Farashah Rik de Vries Golnoosh Farnadi · 2026

Read Neighbor-Aware Localized Concept Erasure in Text-to-Image Diffusion Models by Zhuan Shi Alireza Dehghanpour Farashah Rik de Vries Golnoosh Farnadi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-25994.txt.

Introduction

Shi and co-authors present NLCE, a method that erases a concept from image generators without harming its neighbors. Removing one concept can weaken related ones nearby in meaning. NLCE works in three stages and needs no retraining. First, a spectral tweak dims the target concept in embedding space while steadying its neighbors. Next, an attention gate finds spots where traces of the concept remain. A final gated step erases only those spots. On flower and dog breed data, close categories survive far better, the authors report.

Catalog record

CREATORZhuan Shi Alireza Dehghanpour Farashah Rik de Vries Golnoosh Farnadi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64169
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.25994

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Neighbor-Aware Localized Concept Erasure in Text-to-Image Diffusion Models Zhuan Shi Alireza Dehghanpour Farashah Rik de Vries Golnoosh Farnadi Abstract Concept erasure in text-to-image diffusion models seeks to remove undesired concepts while preserving overall generative capability. Localized erasure methods aim to restrict edits to the spatial region occupied by the target concept. However, we observe that suppressing a concept can unintentionally weaken semantically related neighbor concepts, reducing fidelity in fine-grained domains. We propose Neighbor-Aware Localized Concept Erasure ( NLCE ) , a training-free framework designed to better preserve neighboring concepts while removing target concepts. It operates in three stages: (1) a spectrally-weighted embedding modulation that attenuates target concept directions while stabilizing neighbor concept representations, (2) an attention-guided spatial gate that identifies regions exhibiting residual concept activation, and (3) a spatially-gated hard erasure that eliminates remaining traces only where necessary. This neighbor-aware pipeline enables localized concept removal while maintaining the surrounding concept neighborhood structure. Experiments on fine-grained datasets (Oxford Flowers, Stanford Dogs) show that our method effectively removes target concepts while better preserving closely related categories. Additional results on celebrity identity, explicit content and artistic style demonstrate robustness and generalization to broader erasure scenarios. Code is available at https://github.com/alirezafarashah/NLCE.git † † footnotetext: Corresponding author: [email protected]. 1 Introduction Recently, text-to-image (T2I) diffusion models have been widely adopted in creative and industrial domains for generating high-quality visuals from a wide range of prompts Song et al. 2020 ; Nichol et al. 2021 ; Rombach et al. 2022 ; Ramesh et al. 2022 ; Saharia et al. 2022 ; Yang et al. 2023 ; Liu et al. 2025 . However, their training on large-scale, uncurated datasets Schuhmann et al. 2022 ; Carlini et al. 2019 poses risks of generating harmful content Mirsky and Lee 2021 ; Schramowski et al. 2023 , reproducing copyrighted works or closely imitating distinctive artistic expressions Jiang et al. 2023 ; Setty 2023 ; Shi et al. 2024a . To enable safe and compliant deployment, concept erasure, which refers to the removal of specific visual concepts from the model’s generative capacity, has become a critical requirement. Figure 1 : Erasure Effectiveness and Neighbor Retention: GLoCE vs. NLCE (Ours) To address this need, a growing body of work has explored concept erasure mechanisms for T2I diffusion models. Early work pursues training-based erasure , adjusting model weights or prompt embeddings through fine-tuning and gradient-based editing Lu et al. 2024 ; Lyu et al. 2024 ; Gandikota et al. 2023 ; Zhang et al. 2024 ; Li et al. 2024a ; Liu et al. 2024 ; Liang et al. 2024 . While effective, these methods require compute, curated data, and internal model access. To improve scalability, recent efforts explore training-free erasure , manipulating embeddings or attention without requiring extensive retraining Gandikota et al. 2024 ; Schramowski et al. 2023 ; Gong et al. 2024 ; Wang et al. 2025 ; Biswas et al. 2025 ; Orgad et al. 2023 ; Yoon et al. 2024 . However, most operate globally, which can cause unintended semantic drift. To maintain fidelity and spatial precision, Localized Concept Erasure has recently been proposed in GLoCE Lee et al. 2025b , which seeks to remove the target concept only from the region of the image where it visibly appears, leaving the remainder of the scene untouched. In detail, GLoCE applies a gated low-rank adapter on image embeddings, where the gate activates only for target concepts to selectively suppress them. However, we identify a critical limitation in this formulation: the neighbor gap -a phenomenon where semantically adjacent concepts are unintentionally suppressed alongside the target. To illustrate this, we conduct an experiment in which a specific dog breed is erased. As shown in Fig. 1 , while both methods succeed in suppressing the target, GLoCE also degrades the generation quality of other dog breeds, indicating a lack of semantic precision in preserving neighboring concepts. To bridge this gap, we propose Neighbor-Aware Localized Concept Erasure ( NLCE ), a training-free framework that progressively removes target concepts while preserving semantically related neighbors and global image quality. NLCE operates in three coordinated stages: (1) Representation-space modulation suppresses target subspace directions through spectrally-weighted editing and restores neighboring semantics via CLIP-guided low-rank reinjection; (2) Attention-guided spatial gating extracts attention maps to localize residual activations, selectively reducing attention to live tokens in identified regions; and (3) Gated feature clean-up applies a spatially hard erasure within these gated areas, ensuring complete removal of residual traces while preserving unaffected features. Experiments on fine-grained datasets (Oxford Flowers, Stanford Dogs) demonstrate that NLCE suppresses target concepts while retaining closely related categories. Additional evaluations on celebrity identity, explicit content and artistic style confirm the robustness and generality of the approach. Our contributions are as follows: • We identify and formalize Concept Neighborhood gap in current localized, training-free concept erasure methods. • We propose NLCE , a principled, training-free pipeline that explicitly preserves neighbor concepts while achieving complete and localized erasure of the target. • Experiments show that NLCE achieves high erasure precision, preserves semantic neighbors, and maintains overall generation quality across different datasets and settings. 2 Related Work Concept Erasure in Diffusion Models. Concept erasure aims to remove specific semantics from text-to-image models while retaining generative capacity. Training-based methods achieve erasure through learned parameter updates or prompt finetuning, including weight regularization, preference optimization, and adversarial training Zhang et al. 2024 ; Li et al. 2024a ; Liu et al. 2024 ; Liang et al. 2024 ; Li et al. 2024b ; Shi et al. 2024b ; Park et al. 2024 ; Das et al. 2024 ; Lu et al. 2024 ; Heng and Soh 2023 . Although often effective, they require data and compute resources and may introduce distribution shifts or inadvertent forgetting Chang et al. 2024 . Training-free approaches update weights without requiring retraining, or intervene during inference without updating model weights. Closed-form model editing techniques modify the cross-attention weights to realign concept mappings Gandikota et al. 2024 ; Gong et al. 2024 . Spectral suppression modulates concept-aligned directions Biswas et al. 2025 and attention-guided editing suppresses cross-attention to target tokens Orgad et al. 2023 . These methods are lightweight and scalable, yet global edits may propagate across the scene and suppress nearby, valid concepts. Localized Concept Erasure. To preserve spatial fidelity, localized erasure restricts editing to regions where the target concept appears, typically guided by cross-attention or latent image activations Lee et al. 2025b . By avoiding global suppression, these methods better maintain scene structure and minimize collateral distortion. However, focusing on where to erase does not fully resolve what semantics should be preserved: erasing one fine-grained concept may weaken visually or semantically adjacent categories, exposing a concept neighborhood gap in localized, training-free settings. Neighbor-Aware Erasure. Recent work notes that concept representations are entangled and removing one may inadvertently attenuate neighbors Thakral et al. 2025 . Finetuning-based disentanglement can mitigate this but sacrifices efficiency. Training-free methods leverage embedding geometry and attention routing Gandikota et al. 2024 ; Biswas et al. 2025 but do not explicitly stabilize neighboring semantics. In contrast, our approach adopts localized inference-time editing while explicitly preserving semantic neighbors. We combine spectrally-weighted representation modulation, attention-guided spatial localization, and gated residual removal to selectively suppress targets while maintaining the fidelity

Read the complete text