T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generation

Lijun Li, Zhelun Shi, Xuhao Hu, Bowen Dong, Yiran Qin, Xihui Liu, Lu Sheng, Jing Shao · 2025

Read T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generation by Lijun Li, Zhelun Shi, Xuhao Hu, Bowen Dong, Yiran Qin, Xihui Liu, Lu Sheng, Jing Shao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-12612.txt.

Introduction

Lijun Li and seven colleagues wrote a 2025 paper on a safety test for AI image generators. Text-to-image models can draw sharp pictures from a prompt. They can also produce harmful, biased, or private content, and checks for these risks lag behind the tools.

Their benchmark, T2ISafety, grades models on three fronts. The title names them: fairness, toxicity, and privacy. A fine-grained tree of 12 tasks breaks each front into testable parts. Models face prompts built to expose weak spots in each area. A shared yardstick helps buyers and builders compare models on safety, not just on looks. The authors frame it as a step toward safer image AI.

Catalog record

CREATORLijun Li, Zhelun Shi, Xuhao Hu, Bowen Dong, Yiran Qin, Xihui Liu, Lu Sheng, Jing Shao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-12612

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generation Lijun Li, Zhelun Shi, Xuhao Hu, Bowen Dong, Yiran Qin, Xihui Liu, Lu Sheng, Jing Shao Abstract Text-to-image (T2I) models have rapidly advanced, enabling the generation of high-quality images from text prompts across various domains. However, these models present notable safety concerns, including the risk of generating harmful, biased, or private content. Current research on assessing T2I safety remains in its early stages. While some efforts have been made to evaluate models on specific safety dimensions, many critical risks remain unexplored. To address this gap, we introduce T2ISafety, a safety benchmark that evaluates T2I models across three key domains: toxicity, fairness, and bias. We build a detailed hierarchy of 12 tasks and 44 categories based on these three domains, and meticulously collect 70K corresponding prompts. Based on this taxonomy and prompt set, we build a large-scale T2I dataset with 68K manually annotated images and train an evaluator capable of detecting critical risks that previous work has failed to identify, including risks that even ultra-large proprietary models like GPTs cannot correctly detect. We evaluate 12 prominent diffusion models on T2ISafety and reveal several concerns including persistent issues with racial fairness, a tendency to generate toxic content, and significant variation in privacy protection across the models, even with defense methods like concept erasing. Data and evaluator are released under https://github.com/adwardlee/t2i_safety . Figure 1 : Overview of T2ISafety. (a) Comparison of T2ISafety with others. (b) Taxonomy of T2ISafety with three key safety domains. (c) T2ISafety evaluation pipeline. † † footnotetext: ⋆ Equal contribution † Corresponding author 1 Introduction Benchmark Domain Categories Prompts Quality check Evaluation HEIM ( Lee et al. 2024 ) Toxicity & Fairness 2 Human ✗ Pretrained-CLIP I2P ( Schramowski et al. 2023 ) Toxicity 1 Human ✗ Finetuned-CLIP HRS-Bench ( Bakr et al. 2023 ) Fairness 1 GPT ✗ Pretrained-MLLM FAIntbench ( Luo et al. 2024 ) Fairness 1 GPT ✗ Pretrained-CLIP DALL-EVAL ( Cho et al. 2023 ) Fairness 1 Human ✗ Pretrained-MLLM T2ISafety(Ours) Toxicity & Privacy & Fairness 3-12-44 Human ✓ Finetuned-MLLM Table 1: Comparison between T2I safe-related benchmarks and our T2ISafety. Multi-levels refers to the evaluation of multiple safe dimensions. ✗denotes the benchmark lacks this feature. Pretrained means only use public pretrained models to evaluate. The rapid rise of text-to-image (T2I) models ( Rombach et al. 2022 ; OpenAI 2022 ; Saharia et al. 2022 ) has been used to generate high-quality, realistic images from text descriptions across various domains and art styles. This accessibility has led to widespread use in creative applications ( Gal et al. 2022 ; Shi et al. 2024 ; Li et al. 2023b ) . However, the impressive capabilities of T2I models also raise significant concerns regarding their social impacts and potential risks of generating harmful, biased, or private content ( Bird et al. 2023 ; Yang et al. 2024 ) . Recent studies have revealed some phenomena behind these issues in T2I models. Studies have shown that malicious text prompts can lead models to generate inappropriate, offensive, or dangerous images that poses serious risks to users ( Hao et al. 2024 ; Qu et al. 2023 ) . Fairness issues arise when models amplify social biases and stereotypes such as gender and racial biases ( Seshadri et al. 2023 ; Bianchi et al. 2023 ; Bird et al. 2023 ) , leading to content that misrepresents or discriminates against certain groups. Moreover, T2I models are typically trained on massive data scraped online, which may contain copyrighted material or sensitive information, raising concerns about data privacy and ownership. To mitigate these issues, external defense methods like plug-and-play safety filters are employed to detect inappropriate textual inputs or visual outputs during image generation, but these filters can be easily bypassed ( Ba et al. 2023 ; Li et al. 2024b ) . This vulnerability highlights the need to enhance the inherent safety mechanisms within T2I models themselves. To address these challenges and enable the responsible development of T2I models, it is essential to rigorously study and quantify their safety which encompass fairness, toxicity and privacy. Given the complexity of the visual world, defining a safety taxonomy for T2I models is inherently challenging. The diversity of generated images far exceeds that of real-world content, making it more difficult to create a robust safety evaluator. Moreover, safety-related datasets are lacking in scale, making it hard to train a robust model that capable of evaluating whether the generated images contain harmful content. Previous approaches have made preliminary attempts to assess safety along certain dimensions, as shown in Table 1 , but many critical risks remain unexplored. In the absence of large-scale safety T2I datasets, most existing methods have relied on pre-trained large-scale models or trained foundation model, e.g. CLIP, on small-scale datasets for safety evaluations. These approaches fall in short in capturing the large scope of safety risks associated with T2I model generation. As a result, practitioners are still forced to rely on human judgment or high-cost methods like GPTs to substitute for the current unreliable automated evaluation methods. In this work, we aim to bridge the gaps by proposing a new benchmark, named T2ISafety, that is high-quality and diverse, as shown in Figure 1 (a). To begin, we define a three-level hierarchical taxonomy based on three key domains, covering a broad spectrum of safety dimensions including 12 tasks and 44 categories. Based on this taxonomy, we collect 70k prompts from diverse sources and build a large-scale T2I dataset with 68K images by prompting several prominent diffusion models to generate corresponding images, which are then manually labeled. To ensure automatic, reproducible and accurate evaluations, we also develop a image safety evaluator, ImageGuard , based on a Multimodal Large Language Model (MLLM). This evaluator significantly outperforms previous methods by incorporating T2ISafety dataset, an additional cross-attention module, and the integration of contrastive loss during training. As shown in Figure 1 (c), during the evaluation of T2I models, we apply safety rate in toxicity and privacy domains to provide a clear reflection of harmful content levels. Additionally, we propose using normalized Kullback-Leibler (KL) divergence to measure fairness, offering a more interpretable and asymmetric approach. Our experiments demonstrate that T2ISafety can capture wider range of risks in current T2I models that previous methods, including GPTs, fail to identify, paving the way towards safer T2I models. In summary, our contributions are three-folded. (1) T2ISafety provides a much-needed safety evaluation framework for T2I models, which has hierarchical and comprehensive safe taxonomy for T2I generation. (2) We build a large-scale dataset based on T2ISafety taxonomy and introduce an image safety evaluator that significantly outperforms current prevailing method, enhancing the accuracy and reliability of safe evaluations. (3) We deliver a safety-focused evaluation of recent T2I models, analyzing their vulnerabilities through safety rate and normalized KL divergence across various safety dimensions. 2 Related works 2.1 Safety datasets on T2I models Existing benchmarks for T2I models primarily emphasize image quality ( Hu et al. 2023 ) , text-image alignment ( Lin et al. 2024b ) , and specific capabilities like compositionality and counting ( Park et al. 2021 ) . Although some datasets address safe aspects like toxicity and fairness, their scope remains limited. For

Read the complete text