PromptSafe : Gated Prompt Tuning for Safe Text-to-Image Generation

Zonglei Jing, Xiao Yang, Xiaoqian Li, Siyuan Liang, Aishan Liu, Mingchuan Zhang, Xianglong Liu · 2025

Read PromptSafe : Gated Prompt Tuning for Safe Text-to-Image Generation by Zonglei Jing, Xiao Yang, Xiaoqian Li, Siyuan Liang, Aishan Liu, Mingchuan Zhang, Xianglong Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01272.txt.

Introduction

Zonglei Jing and co-authors present a 2025 paper on keeping text-to-image models from drawing unsafe pictures. Models can be prompted into violent or explicit output. Older defenses glue the same guard tokens onto every prompt, which costs computing, dulls harmless images, and fits real prompts poorly.

Their system, PromptSafe, first rewrites unsafe prompts into safe, meaning-preserving versions with a language model, building a cheap text-only training set. A universal soft prompt then steers image generation away from unsafe embeddings and toward safe ones. At run time, a gate estimates how toxic a prompt is and dials the defense to match. Benign prompts pass nearly untouched, while risky ones meet a strong shield.

Catalog record

CREATORZonglei Jing, Xiao Yang, Xiaoqian Li, Siyuan Liang, Aishan Liu, Mingchuan Zhang, Xianglong Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-01272

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PromptSafe : Gated Prompt Tuning for Safe Text-to-Image Generation Zonglei Jing, Xiao Yang, Xiaoqian Li, Siyuan Liang, Aishan Liu, Mingchuan Zhang, Xianglong Liu Abstract Text-to-image (T2I) diffusion models have demonstrated remarkable generative capabilities but remain vulnerable to producing not-safe-for-work (NSFW) content, such as violent or explicit imagery. While recent moderation efforts have introduced soft prompt-guided tuning by appending defensive tokens to the input, these approaches often rely on large-scale curated image-text datasets and apply static, one-size-fits-all defenses at inference time. However, this results not only in high computational cost and degraded benign image quality, but also in limited adaptability to the diverse and nuanced safety requirements of real-world prompts. To address these challenges, we propose PromptSafe , a gated prompt tuning framework that combines a lightweight, text-only supervised soft embedding with an inference-time gated control network. Instead of training on expensive image-text datasets, we first rewrite unsafe prompts into semantically aligned but safe alternatives using a large language model, constructing an efficient text-only training corpus. Based on this, we optimize a universal soft prompt that repels unsafe and attracts safe embeddings during the diffusion denoising process. To avoid over-suppressing benign prompts, we introduce a gated control mechanism that adaptively adjusts the defensive strength based on estimated prompt toxicity on-the-fly, thereby aligning defense intensity with prompt risk and ensuring strong protection for harmful inputs while preserving benign generation quality. Extensive experiments across multiple benchmarks and T2I diffusion models show that PromptSafe achieves a state-of-the-art low unsafe generation rate (2.36%), while preserving high benign fidelity. Furthermore, PromptSafe demonstrates strong generalization to unseen harmful categories, robust transferability across diffusion model architectures, and resilience under adaptive adversarial attacks, highlighting its practical value for safe and scalable deployment. 1 Introduction Text-to-image (T2I) diffusion models, such as Stable Diffusion CompVis 2022 ; StabilityAI 2023 , have achieved remarkable success in generating high-fidelity images from natural language prompts. However, the widespread deployment of these models has raised growing concerns over their potential misuse, particularly in generating not-safe-for-work (NSFW) content, such as harmful or unethical imagery Liu et al. 2025 ; Liang et al. 2025b ; Xu et al. 2025 ; Guo et al. 2024 ; Liang et al. 2023 ; Liang et al. 2025a . This risk underscores the urgent need for scalable, effective safety mechanisms that can reliably moderate content generation without compromising quality. Recent work has explored soft prompt-guided tuning as a lightweight solution for enhancing safety in T2I models, which typically prepend a soft token to user inputs that guides learnable embeddings to steer generation away from unsafe outputs. A representative example is PromptGuard Yuan et al. 2025 , which leverages textual inversion Gal et al. 2022 to learn detoxification embeddings from curated image-text pairs. While effective, this approach introduces two key limitations: ❶ it requires large-scale curated image-text pairs, imposing substantial computational costs and reducing scalability to newly emerging harmful concepts; ❷ it applies uniform, fixed-strength defenses to all prompts (malicious or benign), often degrading the quality of safe content generation due to over-suppression. Figure 1 : Illustration of PromptSafe defense. A toxicity-aware gated network dynamically adjusts the strength of soft prompt embeddings, enabling effective suppression of NSFW while preserving the benign generation quality. To address these, we propose PromptSafe , a lightweight and modular framework for prompt-aware defense in T2I generation. Our approach consists of two core components as follows. ❶ Text-only Supervised Prompt Tuning . Rather than relying on costly image-text datasets, we use a large language model to generate semantically aligned safe rewrites of unsafe prompts. These textual pairs form the basis for learning detoxification embeddings, which are optimized within the diffusion process. Specifically, we train soft prompts inserted into the token embedding sequence to steer unsafe prompts toward generating outputs that are semantically aligned with their safe counterparts. ❷ Gated Toxicity-Aware Inference . To minimize unnecessary suppression of benign prompts, we introduce a gated network that predicts a continuous toxicity score for each input. This score modulates the strength of the defensive prompt embedding at inference time, enabling dynamic, input-sensitive detoxification that is strong when necessary and minimal otherwise. Extensive experimental results on I2P Schramowski et al. 2023 and MS COCO 2017 datasets Lin et al. 2015 and SDv1.4 T2I models CompVis 2022 demonstrate that PromptSafe consistently outperforms existing methods ( e.g . , PromptGuard Yuan et al. 2025 , SafeGen Li et al. 2024 , SLD Schramowski et al. 2023 , and UCE Gandikota et al. 2024 ) across both safety and quality metrics, where our approach achieves the lowest unsafe content generation ratio at 2.36% while maintaining a higher CLIP score of 26.30 on benign prompts. We further verify the generalization and robustness of PromptSafe on multiple fronts, including unseen harmful categories ( e.g . , self-harm, hate, and illegal activity), diverse benign domains, cross-architecture transfer, and adaptive adversarial attacks, highlighting its practical value for safe and scalable deployment. Our main contributions are: • We introduce a text-driven soft prompt-guided defense framework that eliminates the need for curated image-text pairs or visual supervision, improving scalability and deployment efficiency. • We propose a gated network that predicts prompt toxicity scores and enables dynamic modulation of defense strength, allowing prompt-specific safety control during inference. • We empirically validate that PromptSafe achieves superior NSFW suppression, while preserving benign generation quality and demonstrating strong generalization and cross-model adaptability. 2 Preliminaries and Backgrounds Text-to-Image (T2I) Generation. Text-to-image models aim to synthesize high-fidelity images conditioned on natural language prompts. Modern T2I systems, such as Stable Diffusion, typically follow a latent diffusion pipeline: given a text prompt P , a pretrained text encoder \mathcal{E} first maps it to a semantic embedding \mathbf{E}_{p} . A U-Net denoiser \mathcal{U} then iteratively refines a noise sample z_{T}\sim\mathcal{N}(0,\mathbf{I}_{d}) into a coherent latent representation, guided by \mathbf{E}_{p} via cross-attention. Here, \mathbf{I}_{d} denotes the d -dimensional identity matrix, where d=768 as default in SDv1.4. Finally, a VAE decoder \mathcal{D} transforms the denoised latent into the output image I . The overall generation process \mathcal{G} can be formally expressed as: I=\mathcal{G}(P)=\mathcal{D}\left(\mathcal{U}(z_{T}\mid\mathbf{E}_{p}\right),\;\text{where}\;\mathbf{E}_{p}=\mathcal{E}(P). (1) NSFW Risks for T2I models. Despite their impressive generative capabilities, T2I models are vulnerable to attacks that trigger the synthesis of not-safe-for-work (NSFW) content, such as sexual, violent, or disturbing imagery. Jailbreak attacks Ying et al. 2025 ; Jing et al. 2024 are a typical method that often manipulate malicious prompts P^{m} through obfuscation, metaphors, or indirect phrasing to bypass safety filters while still activating harmful concepts within the latent space Gao et al. 2024 ; Deng and Chen 2024 ; Yang et al. 2024 ; Jing et al. 2025 ; Ying et al. 2025 , resulting in NSFW outputs: I^{\text{NSFW}}=\mathcal{D}\left(\mathcal{U}(z_{T}\mid\mathcal{E}(P^{m})\right). (2) 3 Related Work To mitigate NSFW content, existing defenses for T2I models typically fall into two categories: content moderation and model tuning. Content moderation includes input filtering OpenAI 2024 ; Hanu and Unitary team 2020 and output detection Qu et al. 2023 ; Fu et al. 2025 ; Schramowski et al. 2022 . Although effective, they often rely on auxiliary models or require additional post-processing steps within the diffusion pipeline. In contrast, model-level defenses embed safety constraints directly into the generation process. For example, SDv2.1 StabilityAI 2023 is retrained on censored datasets to suppress NSFW content. However,

Read the complete text