AutoPrompt: Automated Red-Teaming of Text-to-Image Models via LLM-Driven Adversarial Prompts
Yufan Liu, Wanqian Zhang, Huashan Chen, Lin Wang, Xiaojun Jia, Zheng Lin, Weiping Wang · 2025
Read AutoPrompt: Automated Red-Teaming of Text-to-Image Models via LLM-Driven Adversarial Prompts by Yufan Liu, Wanqian Zhang, Huashan Chen, Lin Wang, Xiaojun Jia, Zheng Lin, Weiping Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24034.txt.
Introduction
Yufan Liu and six co-authors red-team text-to-image models with adversarial prompts that read like ordinary English. Safety filters catch many attacks because optimizer-generated prompts are gibberish, and most attack methods need white-box access. Their framework, AutoPrompt, works black-box: a large language model writes human-readable adversarial suffixes for benign prompts.
Training alternates between optimizing suffixes and fine-tuning the language model on the winners. Two tricks defeat defenses. An auxiliary model's perplexity score keeps the suffixes fluent, slipping past perplexity filters. Penalties on banned tokens keep blacklisted words out of the text entirely. The resulting prompts transfer zero-shot to unseen prompts and exposed weaknesses in a commercial service, Leonardo.Ai. The authors frame the work as proactive auditing: readable, filter-resistant attacks reveal where image generators actually break, so builders can fix the holes before someone less friendly finds them.
Catalog record
| CREATOR | Yufan Liu, Wanqian Zhang, Huashan Chen, Lin Wang, Xiaojun Jia, Zheng Lin, Weiping Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-24034
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AutoPrompt: Automated Red-Teaming of Text-to-Image Models via LLM-Driven Adversarial Prompts Yufan Liu, Wanqian Zhang, Huashan Chen, Lin Wang, Xiaojun Jia, Zheng Lin, Weiping Wang Abstract Despite rapid advancements in text-to-image (T2I) models, their safety mechanisms are vulnerable to adversarial prompts, which maliciously generate unsafe images. Current red-teaming methods for proactively assessing such vulnerabilities usually require white-box access to T2I models, and rely on inefficient per-prompt optimization, as well as inevitably generate semantically meaningless prompts easily blocked by filters. In this paper, we propose APT (AutoPrompT), a black-box framework that leverages large language models (LLMs) to automatically generate human-readable adversarial suffixes for benign prompts. We first introduce an alternating optimization-finetuning pipeline between adversarial suffix optimization and fine-tuning the LLM utilizing the optimized suffix. Furthermore, we integrates a dual-evasion strategy in optimization phase, enabling the bypass of both perplexity-based filter and blacklist word filter: (1) we constrain the LLM generating human-readable prompts through an auxiliary LLM perplexity scoring, which starkly contrasts with prior token-level gibberish, and (2) we also introduce banned-token penalties to suppress the explicit generation of banned-tokens in blacklist. Extensive experiments demonstrate the excellent red-teaming performance of our human-readable, filter-resistant adversarial prompts, as well as superior zero-shot transferability which enables instant adaptation to unseen prompts and exposes critical vulnerabilities even in commercial APIs (e.g., Leonardo.Ai.). Warning: This paper contains model outputs that are offensive in nature. Figure 1 : We propose a LLM-driven automated red-teaming framework for T2I models. (a) We compare red-teaming performance on SLD-MAX, our method generates human-readable adversarial prompts that successfully induce safe T2I models to produce unsafe content. (b) Our approach achieves the highest red-teaming success rate on emerging safe T2I models while maintaining the lowest (best) perplexity (PPL) scores. Notably, our method generates adversarial prompts nearly three orders of magnitude faster than baseline approaches. * * footnotetext: Corresponding author 1 Introduction In recent years, text-to-image (T2I) diffusion models [ 26 , 29 , 30 , 2 , 5 ] have advanced unprecedented generative capability through large-scale multimodal learning, which creates images that visually aligned with textual descriptions. Despite great success, they inherit risks from uncontrolled data collection, leading to the Not-Safe-For-Work (NSFW) outputs caused by maliciously devised adversarial prompts. Consequently, numerous security policies for T2I models have been developed to mitigate the production of unsafe content, varying from training data filtering [ 28 ] to safety checker [ 27 ] , as well as the inference phase guidance [ 32 ] and fine-tuning the model to eliminate undesired concepts [ 7 , 8 , 21 , 13 , 42 , 40 ] . Although these interventions suppress undesirable outputs to some extent, their efficacy and robustness remain inadequately quantified due to the lack of standardized and automatic adversarial evaluation framework. Recently, some red-teaming tools for diffusion models have been proposed, which is essential to the development of safe and reliable T2I methods. However, these red-teaming methods typically require the gradient information of the target diffusion model, indicating less scalibility due to the time-consuming optimization process in discrete space [ 34 , 14 , 43 , 6 , 38 , 37 , 9 ] . Besides, the adversarial prompts generated by these methods are not human-readable, i.e., semantically meaningless, thus can be easily filtered by perplexity-based mitigation strategies [ 15 ] . Moreover, these adversarial prompts often explicitly contain words associated with undesirable outputs, which can also be easily banned by blacklist word filters [ 1 ] . These limitations impede the effectiveness of current red-teaming evaluation frameworks in practical applications. To address the above challenges, we introduce a novel automated red-teaming framework, called AutoPrompT (APT), which efficiently generates human-readable , unblocked adversarial prompts via only black-box access to T2I models. The main intuition is to leverage the natural language generation capabilities of LLMs to automatically generate adversarial suffixes for benign prompts. Specifically, we design an alternating ‘optimization-and-finetuning’ pipeline. During the optimization phase, we first utilize the frozen LLM to generate optimized adversarial suffixes for benign prompts via stochastic beam search. Our primary optimization objective, defined as the alignment constraint, is to minimize the gap between the outputs of adversarial suffixes on target T2I models and unsafe content. In the fine-tuning phase, these optimized suffixes are paired with their corresponding benign prompts to enable LLM supervised training. Besides, we implement a dual-evasion strategy to bypass content safeguards in the optimization phase: To evade detection by perplexity-based filter, we introduce another LLM to measure prompt perplexity as perplexity constraint, which is integrated to the alignment constraint to guide adversarial suffix generation. To elude blacklist word filter, we also devise banned-token penalties during adversarial suffix optimizing, thereby suppressing the generation of blacklisted keywords. Generally, our APT method overcomes key limitations of existing methods through following distinct advantages: Black-box access : Our approach does not necessitate accessing the internal parameters or gradients of target T2I models, which solely requires the loss computation based on output images. This is more realistic and applicable than previous white-box approaches. Human-readability : Our method leverages LLMs’ powerful text generation capability combined with perplexity constraint to synthesize coherently human-readable adversarial prompts. This contrasts sharply with prior discrete optimization strategies that yield semantically meaningless prompts. Unblocked : Our dual-evasion strategy ensures generated prompts circumvent both perplexity-based filter and blacklist word filter, while previous methods either produce unreadable prompts or generate blacklisted tokens blocked by these two prompt filters. Our work not only provides a practical and scalable red-teaming framework for vulnerability assessment of existing T2I safety protocols, but also incentivizes the community to develop robust, powerful safeguards for AIGC. Our main contributions are summarized as: • We propose a novel red-teaming framework for T2I models called AutoPrompt. By introducing LLMs’ natural language generation capabilities, we establish a practical black-box solution for T2I red-teaming that generates human-readable adversarial prompts. • We introduce LLMs to iteratively generate human-readable adversarial suffixes and alternate with suffix generator fine-tuning. The proposed dual-evasion strategy further enables bypasses of unsafe content protections like perplexity-based filter and blacklist word filter. • Extensive experiments show the effectiveness of our method against advanced defense mechanisms of T2I models. More crucially, our approach demonstrates zero-shot transferability between unseen prompts, enhancing the scalability of large-scale safety evaluations. 2 Related Work Red-teaming against T2I models. Red-teaming [ 47 , 33 , 45 , 11 , 25 , 23 , 44 , 39 , 16 ] is essentially adversarial attacks against target models to expose their vulnerabilities. For T2I models, red-teaming methods specifically aim to discover adversarial prompts that induce the generation of inappropriate image content. Numerous existing studies [ 46 , 34 , 6 , 43 , 38 ] on adversarial attacks against T2I models have demonstrated their vulnerabilities, even when equipped with various safety mechanisms. The Ring-A-Bell [ 34 ] first extracts unsafe concepts and constructs problematic prompts, then leverages a genetic algorithm to optimize discrete variables, aligning its soft prompts with the problematic prompts. P4D [ 6 ] optimizes continuous prompts to minimize the discrepancy between their corresponding diffusion model’s predicted noise and the predicted noise associated with unsafe prompts. UnlearnDiffAtk [ 43 ] employs the PGD algorithm to optimize discrete prompts, minimizing the discrepancy between the predicted noise when NSFW image input and random Gaussian