Wukong Framework for Not Safe For Work Detection in Text-to-Image systems
Mingrui Liu, Sixiao Zhang, Cheng Long · 2025
Read Wukong Framework for Not Safe For Work Detection in Text-to-Image systems by Mingrui Liu, Sixiao Zhang, Cheng Long (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-00591.txt.
Introduction
Mingrui Liu and co-authors catch unsafe images while they are still forming inside the generator. Text-to-image systems can be coaxed into violent or explicit pictures. Text filters miss model-specific tricks and fall to adversarial prompts; image filters wait for the finished picture and pay full compute. The team's framework, Wukong, watches the diffusion process itself, reading early denoising steps where the picture's layout is already set.
It reuses the model's own cross-attention weights, the parts that tie words to regions, inside a small transformer judge. Detection fires early, cheaply, and accurately: on their new labeled dataset and two public benchmarks, Wukong beat text safeguards clearly and matched image filters at a fraction of the cost. Speed, they argue, is a safety feature.
Catalog record
| CREATOR | Mingrui Liu, Sixiao Zhang, Cheng Long |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-00591
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Wukong Framework for Not Safe For Work Detection in Text-to-Image systems Mingrui Liu, Sixiao Zhang, Cheng Long Abstract . Text-to-Image (T2I) generation is a popular AI-generated content (AIGC) technology enabling diverse and creative image synthesis. However, some outputs may contain Not Safe For Work (NSFW) content (e.g., violence), violating community guidelines. Detecting NSFW content efficiently and accurately, known as external safeguarding, is essential. Existing external safeguards fall into two types: text filters, which analyze user prompts but overlook T2I model-specific variations and are prone to adversarial attacks; and image filters, which analyze final generated images but are computationally costly and introduce latency. Diffusion models, the foundation of modern T2I systems like Stable Diffusion, generate images through iterative denoising using a U-Net architecture with ResNet and Transformer blocks. We observe that: (1) early denoising steps define the semantic layout of the image, and (2) cross-attention layers in U-Net are crucial for aligning text and image regions. Based on these insights, we propose Wukong, a transformer-based NSFW detection framework that leverages intermediate outputs from early denoising steps and reuses U-Net’s pre-trained cross-attention parameters. Wukong operates within the diffusion process, enabling early detection without waiting for full image generation. We also introduce a new dataset containing prompts, seeds, and image-specific NSFW labels, and evaluate Wukong on this and two public benchmarks. Results show that Wukong significantly outperforms text-based safeguards and achieves comparable accuracy of image filters, while offering much greater efficiency. Warning : This paper contains potentially offensive text and images. 1. Introduction Text-to-Image (T2I) generation is a widely used AI-generated content (AIGC) system, enabling diverse and creative image production. However, some generated images may violate community guidelines by including content that is Not Safe For Work (NSFW), such as depictions of explicit nudity (e.g., “a naked woman”). Effectively and efficiently detecting NSFW content in T2I scenarios is essential to ensure these models adhere to ethical standards and community rules. Figure 1. An illustrative example of modifying the textual condition during the early denoising steps in the Stable Diffusion process. The initial latent noise is conditioned with an unsafe prompt ( prompt_1 ) in the first step. From the second step onward, the first row remains conditioned on prompt_1 , while the second row switches to a safe prompt ( prompt_2 ). Existing T2I systems have implemented various safety-oriented strategies to prevent the generation of offensive content in images. These defensive methods can be broadly categorized into two types: internal safeguards and external safeguards ( Zhang et al. 2024b ) . Internal safeguards typically involve model-editing techniques aimed at removing unsafe concepts from T2I models (e.g., making the model "forget" the concept of nudity) ( Kumari et al. 2023 ; Ni et al. 2023 ; Heng and Soh 2024 ; Gandikota et al. 2023 ; Orgad et al. 2023 ; Kim et al. 2023 ; Wu et al. 2024 ; Kim et al. 2024 ; Huang et al. 2024 ; Zhang et al. 2024a ) . However, internal safeguards have three main drawbacks: (1) They may degrade image quality in normal (safe) use cases. (2) They remain vulnerable to adversarial attacks (e.g., prompts that appear safe but ultimately generate unsafe images) ( Tsai et al. 2023 ) . (3) They introduce ethical concerns by misinterpreting user input. Additionally, some commercial T2I service providers (e.g., DALL·E 3 ( Betker et al. 2023 ) ) offer tiered generation services, where regular users can generate only safe images, while subscribed users have access to NSFW image generation. In contrast, internal safeguards cannot provide this functionality, as they remove all unsafe concepts. Due to these limitations, we focus solely on external safeguards, where the users’ generation requests will be rejected if unsafe contents are detected. External safeguards can be categorized into text and image filters ( Zhang et al. 2024b ) . Text filters are simple and efficient but fail to account for T2I model-specific variations, as safety outcomes can vary across models or even within the same model under different random seeds. For example, some textually offensive prompts may generate safe images in certain models, while seemingly harmless prompts can produce unsafe images under specific seeds ( Li et al. 2024 ; Du et al. 2023 ; Gao et al. 2023 ; Zhuang et al. 2023 ) . Additionally, adversarial techniques ( Yang et al. 2024a ; Yang et al. 2024c ) have been developed to bypass text filters, leading to suboptimal performance. Image filters adopt a post-hoc approach, where images are first generated by T2I models and then evaluated by an image classification model to assess their safety ( Schramowski et al. 2022 ; Christoph Schuhmann 2023 ; lakshaychhabra 2018 ) . While effective, this method incurs high computational costs and significant response delays, as it requires waiting for the full image generation before analysis. Therefore, there remains a need for an external safeguard that can effectively and efficiently determine whether a user’s input will generate unsafe images . The dominant approach to Text-to-Image generation is the Diffusion series of models ( Ho et al. 2020 ) . One typical model, Stable Diffusion ( Rombach et al. 2022 ) begins by initializing a Gaussian noise tensor, followed by iterative denoising using a U-Net architecture that incorporates textual information as guidance, predicting and removing noise at each step. We visualize the denoising results at each step and empirically observe that the initial denoising steps play a crucial role in shaping the semantic content of the generated images . As shown in Figure 1 , the initial latent noise tensor is conditioned on the unsafe prompt, " A girl is covered with blood in the bathroom ", in the first step. However, even when the prompt is modified to a completely unrelated and safe content, " A dog is on the grass ", from the second step onward, the generated image still retains unsafe elements (e.g., blood), as seen in the second row of Figure 1 . This suggests that, despite appearing as white noise to the human eye, the early denoising steps have already established the fundamental structure and layout of the image . Additionally, we also observe that cross-attention layers in U-Net play a key role in mapping textual concepts to image regions . Based on these insights, we propose Wukong, a transformer-based NSFW detector that utilizes intermediate latent representations from early denoising steps as input, i.e., it does not need to wait for the whole image diffusion process to be completed so as to improve the efficiency. It also leverages pre-trained cross-attention layer parameters in U-Net for feature extraction. To summarize, our contributions in this paper are as follows: • We propose Wukong framework for NSFW contents detection. To the best of our knowledge, Wukong is the first external safeguard to leverage intermediate U-Net outputs for NSFW detection, enabling T2I model-specific filtering. • We construct a new dataset for NSFW detection, where each sample includes a textual prompt, a generator seed (ensuring reproducibility of T2I outputs, with each prompt assigned multiple seeds to generate diverse images), and NSFW category-specific labels, obtained using high-quality vision-language models. • We evaluate Wukong on both our newly proposed dataset and two public benchmarks, demonstrating that it outperforms text-based