Safeguarding Text-to-Image Generation via Inference-Time Prompt-Noise Optimization
Jiangweizhi Peng, Zhiwei Tang, Gaowen Liu, Charles Fleming, Mingyi Hong · 2024
Read Safeguarding Text-to-Image Generation via Inference-Time Prompt-Noise Optimization by Jiangweizhi Peng, Zhiwei Tang, Gaowen Liu, Charles Fleming, Mingyi Hong (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03876.txt.
Introduction
Peng and co-authors tackle a safety problem in text-to-image models, which can produce harmful pictures despite built-in filters, in a 2024 paper. Existing guards are easily bypassed by crafted prompts. Their method, called prompt-noise optimization, needs no retraining of the model.
Instead, at generation time the method adjusts both the prompt's internal representation and the random noise that seeds the image, steering the output away from unsafe content. Tests show the approach blocks toxic generations at the level of the best current methods and resists attack prompts designed to break it. It runs at a comparable speed and keeps images faithful to safe prompts.
Catalog record
| CREATOR | Jiangweizhi Peng, Zhiwei Tang, Gaowen Liu, Charles Fleming, Mingyi Hong |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-03876
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Safeguarding Text-to-Image Generation via Inference-Time Prompt-Noise Optimization Jiangweizhi Peng, Zhiwei Tang, Gaowen Liu, Charles Fleming, Mingyi Hong Abstract Text-to-Image (T2I) diffusion models are widely recognized for their ability to generate high-quality and diverse images based on text prompts. Despite recent advances, these models are still prone to generating unsafe images containing sensitive or inappropriate content, which can be harmful to users. Current efforts to prevent inappropriate image generation for diffusion models are easy to bypass and vulnerable to adversarial attacks. How to ensure that T2I models align with specific safety goals remains a significant challenge. In this work, we propose a novel, training-free approach, called Prompt-Noise Optimization (PNO) , to mitigate unsafe image generation. Our method introduces a novel optimization framework that leverages both the continuous prompt embedding and the injected noise trajectory in the sampling process to generate safe images. Extensive numerical results demonstrate that our framework achieves state-of-the-art performance in suppressing toxic image generations and demonstrates robustness to adversarial attacks, without needing to tune the model parameters. Furthermore, compared with existing methods, PNO uses comparable generation time while offering the best tradeoff between the conflicting goals of safe generation and prompt alignment. CAUTION: This paper contains AI-generated images that may be considered offensive or inappropriate. 1 Introduction Text-to-image (T2I) generation has made significant progress in recent years due to advancements in diffusion models [ 12 , 16 , 36 , 7 ] . Leveraging classifier-free guidance [ 13 ] , these models can generate high-quality, diverse images from text prompts [ 28 , 30 ] , enabling applications across design, art, and content creation [ 8 , 32 ] . The exceptional capabilities of T2I diffusion models stem from extensive pre-training on large-scale datasets. However, while this vast amount of data enhances generative performance, the quality and content of these datasets are not guaranteed. This raises concerns about the safety and appropriateness of the generated images, as they may inherit biases and inappropriate contents from the training data, posing potential risks to users [ 25 , 33 ] . To address the growing concerns surrounding the generation of unsafe content in T2I diffusion models, various safety mechanisms have been proposed. These methods include filtering training datasets and retraining models from scratch [ 24 ] , modifying prompts with a large language model (LLM) to generate safe images [ 46 ] , directly fine-tuning diffusion models with safety objectives [ 10 , 19 , 23 , 48 , 9 ] , and intervening during the inference phase to constrain the generation process [ 30 , 3 , 33 , 38 ] . While each of these approaches has shown promise, they also have significant limitations, such as high computation and data requirements, limited generalization, image quality degradation, and above all, lack of substantial prevention of unsafe content generation [ 29 ] . More critically, these methods often fail to provide robustness against adversarial attacks, leaving the models vulnerable to intentional exploitation [ 47 , 41 , 22 , 49 ] . Figure 1 : The workflow of Prompt-Noise Optimization (PNO). (Left) demonstrates the use case of PNO, where the user provides a potentially toxic prompt to the model, and the model generates an image that is evaluated by a toxicity score, which is used to update the noise trajectory and prompt embedding. (Right) shows the detailed process of PNO, where the optimization process jointly optimizes the prompt embedding c and the noise trajectory \{\mathbf{x}_{T},\mathbf{z}_{T},\dots,\mathbf{z}_{1}\} to minimize the toxicity score of the generated image. Aligning T2I models to safety goals presents significant challenges. First, diffusion models trained on large, unfiltered datasets often inherit biases and inappropriate content from the training data. For example, in Stable Diffusion 1.5 [ 30 ] , trained on LAION-5B [ 35 ] , terms like “Japanese” or “Asian” can trigger sexually inappropriate outputs [ 33 ] . These unpredictable associations hinder effective text-level safety mechanisms [ 19 ] . Second, T2I systems are highly vulnerable to adversarial attacks; even black-box adversarial prompts can bypass safeguards and generate unsafe content [ 47 , 41 ] . Finally, there is a fundamental conflict between the model’s goal to faithfully follow text prompts [ 13 ] and the need to avoid unsafe outputs. Balancing these priorities is crucial for robust T2I safety. To address these challenges, we propose Prompt-Noise Optimization (PNO), a novel, training-free approach to mitigate unsafe image generation in T2I diffusion models. PNO introduces an optimization-based framework that adjusts both the noise trajectory and the continuous prompt embedding within the sampling process to produce safe images during inference time. By jointly optimizing these components, PNO aligns image outputs with specific safety goals—such as avoiding sensitive or inappropriate content—while preserving prompt-image adherence, a critical aspect often overlooked in previous works. PNO operates by iteratively generating images and evaluating them with a safety evaluator, then adjusting the noise trajectory and prompt embedding to minimize a toxicity score. See Fig. 1 for an illustration of the overall algorithm flow. Our extensive empirical evaluations on multiple benchmark datasets demonstrate that PNO is (1) highly effective and efficient in reducing unsafe content generation, (2) robust against adversarial attacks, ensuring reliability across diverse prompts and (3) capable of maintaining optimal prompt-image alignment; see Fig. 2 for an illustration of safety and alignment tradeoff achievable by PNO and other existing methods. Notably, PNO achieves dominant tradeoff curves, surpassing all evaluated baselines. Despite its iterative nature, PNO incurs only marginal extra inference costs while eliminating the need for additional training data or model fine-tuning processes that are far more resource-intensive. To the best of our knowledge, this is the first approach to leverage inference-time optimization for enhancing the safety of T2I diffusion models, and the first work to explore the synergy between prompt embedding and diffusion noise trajectory, which are two fundamental components of diffusion model inference. We briefly summarize our main contributions below. • We introduce Prompt-Noise Optimization (PNO), an efficient, training-free approach to mitigating unsafe image generation in T2I diffusion models, which can be flexibly tailored for different applications or users with specific safety concerns and priorities. • We validate the efficiency, effectiveness, and robustness of PNO through extensive experiments on various datasets, demonstrating state-of-the-art safety performance and strong resilience against adversarial attacks. • We empirically show that PNO can achieve an optimal tradeoff between prompt adherence and image safety, showcasing a Pareto frontier superior to existing methods; see Fig. 2 . • We provide both theoretical and practical insights into our approach, highlighting key advantages of optimizing prompts in the continuous embedding space and the benefits of jointly optimizing both the noise trajectory and prompt embeddings. Figure 2 : CLIP Score \uparrow and toxicity \downarrow tradeoff. PNO (with different learning rates specified in the parenthesis) offers superior tradeoff between image safety and prompt alignment, when compared with state-of-the-art T2I safety mechanisms. 2 Related Work Existing safety mechanisms for T2I diffusion models. Current safety mechanisms for T2I diffusion models generally fall into four categories: (1) data filtering and retraining, (2) fine-tuning with safety objectives, (3) guidance-based adjustment, and (4) model editing. Data filtering and retraining, as seen in SDXL with LAION’s NSFW detector, aim to exclude unsafe content from training data, though