Dynamic VLM-Guided Negative Prompting for Diffusion Models

Hoyeon Chang, Seungjin Kim, Yoonseok ChoiKAIST retapurayo, sjkim, uooh77 @kaist.ac.kr · 2025

Read Dynamic VLM-Guided Negative Prompting for Diffusion Models by Hoyeon Chang, Seungjin Kim, Yoonseok ChoiKAIST retapurayo, sjkim, uooh77 @kaist.ac.kr (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-26052.txt.

Introduction

Hoyeon Chang, Seungjin Kim, and Yoonseok Choi propose a way to steer image diffusion models away from unwanted content that adapts as the image forms. Diffusion models build an image by removing noise step by step, and negative prompting names what the model should avoid. Standard practice uses one fixed negative prompt for the whole run, which may not fit what actually appears midway.

Chang and the team predict the intermediate image at chosen denoising steps and ask a vision language model to write a fitting negative prompt for that stage. They test the approach on several benchmark data sets and study the trade off between negative guidance strength and how well the final image matches the text request.

Catalog record

CREATORHoyeon Chang, Seungjin Kim, Yoonseok ChoiKAIST retapurayo, sjkim, uooh77 @kaist.ac.kr
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE13 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-26052

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Dynamic VLM-Guided Negative Prompting for Diffusion Models Hoyeon Chang, Seungjin Kim, Yoonseok ChoiKAIST retapurayo, sjkim, uooh77 @kaist.ac.kr Abstract We propose a novel approach for dynamic negative prompting in diffusion models that leverages Vision-Language Models (VLMs) to adaptively generate negative prompts during the denoising process. Unlike traditional Negative Prompting methods that use fixed negative prompts, our method generates intermediate image predictions at specific denoising steps and queries a VLM to produce contextually appropriate negative prompts. We evaluate our approach on various benchmark datasets and demonstrate the trade-offs between negative guidance strength and text-image alignment. \workshoptitle The First Workshop on Generative and Protective AI for Content Creation 1 Introduction A prevalent method for content filtering in T2I is negative prompting, which guides the model away from specified concepts through Classifier-Free Guidance (CFG) ( Ho 2022 ) . However, this technique suffers from key limitations. First, it can disrupt the image generation process even when the unwanted content is not present, leading to over-correction and semantic drift ( Ban et al. 2024 ; Chang et al. 2024 ; Koulischer et al. 2024 ) . Second, negative prompts are typically predefined, yet it is difficult to anticipate all potential unwanted elements that might arise from a given positive prompt ( Yoon et al. 2024 ) . This can lead to either unnecessary or inefficient filtering. In this work, we propose a novel solution: Vision-Language guided Dynamic Negative Prompting (VL-DNP). By leveraging the advanced image and language understanding of modern open-source Vision-Language Models (VLMs), our method acts as a dynamic negative prompt generator. VL-DNP detects the emergence of unwanted content during the denoising process and generates targeted negative prompts in real-time to erase it. Importantly, our framework can be easily integrated into any pretrained CFG-based diffusion model without requiring joint training or model modifications. Figure 1 : VL-DNP inference pipeline. The positive text prompt is fed to a pretrained diffusion model. At a small set of timesteps t_{i}\!\in\!\mathcal{T} we predict the clean image \hat{x}_{0} , query a lightweight vision–language model (VLM) and obtain a dynamic negative prompt . The prompt is fed back as classifier-free guidance, steering the remaining denoising steps away from any unsafe content detected in the intermediate image. 2 Related Work Negative prompting and Negative Guidance Negative prompting was proposed to filter out unwanted content. In this work, we write positive conditions we like to align as {\color[rgb]{0,0,1}{c}+} , and negative conditions we want to filter as {\color[rgb]{1,0,0}{c}-} . {s}_{\theta}(x_{t},t,{\color[rgb]{0,0,1}{c}+},{\color[rgb]{1,0,0}{c}-})\;=\;\nabla_{{x}_{t}}\log p({x}_{t}|{\color[rgb]{0,0,1}{c}+})\;+\;\omega\left(\nabla_{{x}_{t}}\log p({x}_{t}|{\color[rgb]{0,0,1}{c}+})\;-\;\nabla_{{x}_{t}}\log p({x}_{t}|{\color[rgb]{1,0,0}{c}-})\right). (1) In Eq. ( 1 ), conventional negative prompting is considered a particular example that blends positive and negative guidance. The corresponding score function augmented with both positive and negative guidance is defined as follows: \displaystyle{s}_{\theta,\text{mixed}}(x_{t},t,{\color[rgb]{0,0,1}{c}+},{\color[rgb]{1,0,0}{c}-})=~\nabla_{x_{t}}\log p(x_{t})+ \displaystyle\omega_{\text{pos}}\Big(\nabla_{x_{t}}\log p(x_{t}|{\color[rgb]{0,0,1}{c}+})-\nabla_{x_{t}}\log p(x_{t})\Big) (2) \displaystyle-\omega_{\text{neg}}\Big(\nabla_{x_{t}}\log p(x_{t}|{\color[rgb]{1,0,0}{c}-})-\nabla_{x_{t}}\log p(x_{t})\Big). Using Eq. ( 2 ) for sampling implies sampling from \tilde{p}({x}_{0})\propto p({x}_{0})\frac{p({\color[rgb]{0,0,1}{c}+}|{x}_{0})^{\omega_{\text{pos}}}}{p({\color[rgb]{1,0,0}{c}-}|{x}_{0})^{\omega_{\text{neg}}}} . The importance of negative guidance at each intermediate generation step might differ. Building upon this intuition, recent work ( Koulischer et al. 2024 ) attempts to determine the weight of the negative guidance scale dynamically based on the estimated p_{t}({\color[rgb]{1,0,0}{c}-}|{x}_{t}) . However, their approach still relies on a pre-defined negative prompt specified prior to inference. Vision-Language Models in Generation VLMs have demonstrated remarkable capabilities in understanding visual content and generating descriptive text ( Radford et al. 2021 ; Li et al. 2022 ; Li et al. 2023 ; Liu et al. 2023 ) . In this work, we harness the image recognition and language generation capabilities of VLMs to provide dynamically adapting negative prompts during diffusion model inference. 3 Methodology The framework operates through the following steps: (1) Perform standard CFG denoising steps using the positive prompt. (2) At predefined timesteps, predict the denoised image \hat{x}_{0} from current latent x_{t} . (3) Query a VLM to generate negative prompts based on the predicted image \hat{x}_{0} . (4) Apply negative guidance using the VLM-generated prompt for subsequent denoising steps. Mathematical Formulation Based on the negative prompting formulation in eq. 2 , we introduce temporal adaptivity to the negative conditioning. Let \Theta denote a Vision-Language Model, and \mathcal{T}={t_{1},t_{2},\ldots,t_{k}}\subseteq[0,T] be a predefined set of timesteps where VLM queries occur. At each query timestep t_{i}\in\mathcal{T} , we predict the denoised image denoted by \hat{x}_{0}^{(i)} using the current estimate as follows: \hat{x}_{0}^{(i)}=\frac{x_{t_{i}}+({1-\bar{\alpha}_{t_{i}}})\cdot s_{\theta,\text{cfg}}(x_{t_{i}},t_{i},{\color[rgb]{0,0,1}{c}+})}{\sqrt{\bar{\alpha}_{t_{i}}}}, (3) where \bar{\alpha}_{t_{i}}=\prod_{s=1}^{t_{i}}(1-\beta_{s}) , and \beta_{s} is the forward process variance at timestep s ( Ho et al. 2020 ) . Classifier-free guidance (CFG) is defined as: {s}_{\theta,\text{cfg}}(x_{t},t,{\color[rgb]{0,0,1}{c}+})\;=\;\nabla_{{x}_{t}}\log p({x}_{t})\;+\;\omega\left(\nabla_{{x}_{t}}\log p({x}_{t}|{\color[rgb]{0,0,1}{c}+})\;-\;\nabla_{{x}_{t}}\log p({x}_{t})\right). The VLM then generates an adaptive negative prompt: {\color[rgb]{1,0,0}{c}-}_{t_{i}}=\Theta(\hat{x}_{0}^{(i)},\mathcal{D}), (4) where \mathcal{D} represents optional few-shot demonstration examples provided to the VLM. Although \hat{x}_{0}^{(i)} is generally blurred during the initial denoising stage, we find that the VLM can detect objects even at early stages. VLM Integration The VLM receives two inputs: (1) the intermediate image prediction \hat{x}_{0}^{(i)} , (2) demonstration examples that guide the model toward generating appropriate negative content descriptors. Our prompting strategy instructs the VLM to identify potentially inappropriate or unwanted visual elements in the predicted image and generate concise negative prompts to suppress such content. Dynamic Negative Guidance For timesteps t_{i}<t<\text{min}(t_{i+1},T) , we apply the augmented score function: \displaystyle\tilde{s}_{\theta}(x_{t},t,{\color[rgb]{0,0,1}{c}+},{\color[rgb]{1,0,0}{c}-}_{t_{i}})= \displaystyle\nabla_{x_{t}}\log p(x_{t}|{\color[rgb]{0,0,1}{c}+})+\omega_{\text{pos}}\Big(\nabla_{x_{t}}\log p(x_{t}|{\color[rgb]{0,0,1}{c}+})-\nabla_{x_{t}}\log p(x_{t})\Big) (5) \displaystyle-\omega_{\text{neg}}\Big(\nabla_{x_{t}}\log p(x_{t}|{\color[rgb]{1,0,0}{c}-}_{t_{i}})-\nabla_{x_{t}}\log p(x_{t})\Big) The key advantage of this formulation is that {\color[rgb]{1,0,0}{c}-}_{t_{i}} adapts to the evolving image content, allowing for more precise and contextual content filtering compared to static negative prompting approaches. 4 Experimental Setup Datasets To evaluate the general performance of image generation, we use 100 prompts randomly sampled from COCO-30K ( Lin et al. 2014 ) . We compute a CLIP and FID score for the COCO-100 prompt set. To evaluate the filtering of unsafe images, we use Ring-a-Bell-16 ( Tsai et al. 2023 ) , P4D ( Chin et al. 2024 ) , and Unlean-diff ( Zhang et al. 2024 ) datasets, which consist of adversarial prompts designed to test content filtering in T2I tasks. Implementation Details We use Stable Diffusion v1.4 ( Rombach et al. 2022 ) as the base diffusion model and Qwen2.5-VL-7B-Instruct ( Bai et al. 2025 ) as the Vision-Language Model for dynamic negative prompt generation. The denoising process employs DPM-Solver++ ( Lu et al. 2022 ) with 50 inference steps, where VLM queries are performed at timesteps {45,44,43,41,38,34,29,23,16,8} out of the total 50 steps. We evaluate our approach using CLIP-based metrics for text-image alignment, complemented by NudeNet ( notAI tech 2019 ) Attack Success Rate and Toxic Rate measurements to assess content filtering effectiveness. Baseline Comparisons We compare our approach against baseline configurations: fixed negative prompting with various negative guidance scales, token embedding projection method SAFREE ( Yoon et al. 2024 ) and our dynamic VLM-guided approach across different negative guidance scale settings. 5 Results Table 1 : Safety ( ASR , TR ) vs. alignment/quality ( CLIP , FID \downarrow ). “VL–DNP” = our dynamic VL-guided negative prompting, “Neg” = conventional static negative prompting. Lower is better for ASR, TR, and FID; higher is better for CLIP. InfT denotes the average time required to generate a single image. Ring-a-Bell-16 P4D Unlearn-Diff COCO-100 InfT Method ASR \downarrow TR \downarrow ASR TR ASR TR CLIP \uparrow FID \downarrow \text{sec}\downarrow SD v1.4 (no neg) 0.958 0.961 0.960 0.935 0.697 0.734 0.312 – 8.0 VL-DNP (Ours) \omega_{\text{neg}}=7.5 0.495 0.521 0.497 0.547 0.310

Read the complete text