Safe and Reliable Diffusion Models via Subspace Projection

Huiqiang Chen, Tianqing Zhu*, Linlin Wang, Xin Yu, Longxiang Gao, Wanlei Zhou · 2025

Read Safe and Reliable Diffusion Models via Subspace Projection by Huiqiang Chen, Tianqing Zhu*, Linlin Wang, Xin Yu, Longxiang Gao, Wanlei Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16835.txt.

Introduction

Huiqiang Chen and co-authors present SAFER, a 2025 method for scrubbing unwanted concepts out of text-to-image diffusion models for good. Today's filters often fail quietly: block Van Gogh by name and his style still returns when the prompt says Starry Night. Concepts hide in the geometry of the model's text space.

SAFER first finds the low-dimensional subspace tied to the target concept. It then projects every prompt embedding onto the remaining space, erasing the concept before generation starts. Learned embeddings from a reference image sharpen the estimate when words alone are vague, and a subspace expansion step widens the net. Tests showed thorough removal where earlier methods leaked. The approach is cheap enough to apply concept by concept, a practical safety dial for image generators.

Catalog record

CREATORHuiqiang Chen, Tianqing Zhu*, Linlin Wang, Xin Yu, Longxiang Gao, Wanlei Zhou
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-16835

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Safe and Reliable Diffusion Models via Subspace Projection Huiqiang Chen, Tianqing Zhu*, Linlin Wang, Xin Yu, Longxiang Gao, Wanlei Zhou Abstract Large-scale text-to-image (T2I) diffusion models have revolutionized image generation, enabling the synthesis of highly detailed visuals from textual descriptions. However, these models may inadvertently generate inappropriate content, such as copyrighted works or offensive images. While existing methods attempt to eliminate specific unwanted concepts, they often fail to ensure complete removal—allowing the concept to reappear in subtle forms. For instance, a model may successfully avoid generating images in Van Gogh’s style when explicitly prompted with “Van Gogh”, yet still reproduce his signature artwork when given the prompt “Starry Night”. In this paper, we propose SAFER, a novel and efficient approach for thoroughly removing target concepts from diffusion models. At a high level, SAFER is inspired by the observed low-dimensional structure of the text embedding space. The method first identifies a concept-specific subspace 𝒮 c \mathcal{S}_{c} associated with the target concept c c . It then projects the prompt embeddings onto the complementary subspace of 𝒮 c \mathcal{S}_{c} , effectively erasing the concept from the generated images. Since concepts can be abstract and difficult to fully capture using natural language alone, we employ textual inversion to learn an optimized embedding of the target concept from a reference image. This enables more precise subspace estimation and enhances removal performance. Furthermore, we introduce a subspace expansion strategy to ensure comprehensive and robust concept erasure. Extensive experiments demonstrate that SAFER consistently and effectively erases unwanted concepts from diffusion models while preserving generation quality. I Introduction Diffusion-based text-to-image (T2I) generative models have achieved remarkable success, enabling the creation of high-quality images from simple text inputs. These models can generate a diverse range of objects, styles, and scenes with remarkable realism and variety. The success of diffusion models is largely attributed to the extensive and massive-scale data used for training. However, these datasets, which are often scraped from the web without curation, may include images that are offensive, unsafe, or protected by intellectual property. As a result, diffusion models can inadvertently learn and reproduce inappropriate concepts , such as copyrighted artistic style, biased, or potentially harmful content [ 1 , 2 ] . The open-source release of Stable Diffusion (SD) [ 3 ] has made advanced image generation technology widely accessible. This highlights the urgent need to prevent these models from generating sensitive or harmful content. This raises a critical question: How can we prevent diffusion models from generating inappropriate images? In other words, how can we erase specific concepts from diffusion models? Fig. 1 : Results on removing Van Gogh’s style . Existing erasing methods remain incomplete—when replacing “ Van Gogh ” with “ Vincent Gogh ” or prompting “ Starry Night ”, the generated images still retain Van Gogh’s style. In contrast, our approach effectively removes the target style by projecting the prompt’s text embedding onto the complementary subspace of the concept embedding Given that diffusion models may memorize inappropriate concepts from training data, machine unlearning [ 4 , 5 ] seems like a plausible solution. Unfortunately, as pointed out by [ 6 ] , unlearning is not well-suited for this task. Because: 1) It is difficult to associate a concept with a specific set of samples. Unlearning algorithms aim to remove the influence of certain training samples as if they have never been used during training. However, the target concept is often abstract, such as a certain artistic style or nudity, which is not tied to an identifiable group of training images; More importantly, 2) removing a subset of training samples does not necessarily eliminate the learned concept from the model. For instance, even though Stable Diffusion v2.1 [ 3 ] was trained on a curated dataset with explicit content filtered out, the model still generates inappropriate images, demonstrating that filtering training data alone is insufficient. If unlearning is not the right approach, what alternatives exist? Recent efforts have focused on implementing safety mechanisms to prevent the generation of a certain concept. These approaches can be broadly divided into two classes. The first class preserves the original model weights and implements safety measures, such as censoring model outputs through safety checkers that detect inappropriate visual content [ 7 ] or applying classifier-free guidance to steer generation away from problematic concepts [ 8 ] . The second category modifies model weights through fine-tuning to erase specific target concepts. For example, [ 9 ] and [ 10 ] train multiple LoRAs [ 11 ] for individual concepts, while [ 6 ] fine-tunes the cross-attention module to achieve concept erasure. While effective to some extent, these methods have drawbacks, including additional computational overhead. More critically, they often result in incomplete erasure, as they tend to block specific keywords rather than fully remove the target concept. As a result, when prompted with related words or synonyms, the erased concepts can still reappear in the generated images. This limitation underscores the need for a more robust approach that comprehensively removes concepts rather than just blocking keywords, preventing their reappearance through indirect prompts. To address these challenges, we introduce SAFER, a novel approach to erasing concepts from diffusion models by leveraging concept subspace in the text embedding space. Our method utilizes textual inversion to capture target concepts from reference images, allowing for precise identification of the corresponding concept subspace. Furthermore, we expand this subspace to encompass a broader and more nuanced representation of the concept, ensuring complete and robust removal. The contribution of this paper is summarized as • We introduce a novel concept subspace identification method within the text embedding space, enabling precise targeting of complex concepts. • We propose a subspace expansion strategy to improve the completeness of concept removal, ensuring robust erasure. • We conduct extensive experiments demonstrating the superiority of our method in removing artistic styles, objects, and explicit content, as well as its capability to erase multiple concepts simultaneously. II Related Works Recent research [ 6 , 7 , 8 , 12 ] aims to mitigate the risks of harmful image generation and copyright issues associated with diffusion models. Most of the work primarily adopts two categories of methods: filtering-based approaches [ 7 , 8 , 1 , 13 ] and fine-tuning [ 6 , 9 , 10 , 12 , 14 , 15 ] . Filtering. The first category employs filtering methods to remove undesirable concepts. These approaches are broadly divided into in-processing and post-processing filtering. In-processing filtering . In-processing filtering focuses on mitigating harmful content during the generation process. For instance, SLD [ 8 ] leverages classifier-free guidance to suppress inappropriate elements, utilizing the model’s internal knowledge to identify and reduce harmful content during generation. ESPRESSO [ 13 ] intervenes at the latent space level, filtering harmful content before it is fully generated. This integration makes it harder for adversarial users to bypass safeguards. However, the approach’s dependency on the model’s inherent biases and training data can limit its ability to handle harmful concepts not well-represented in the training set. Filtering methods are fundamentally reactive. Their effectiveness depends on predefined or learned concepts of harm, making them vulnerable to evasion tactics such as minor prompt adjustments. Moreover, they do not alter the

Read the complete text