PixJail : Self-Evolving Paper-to-Pipeline Reproduction for Text-to-Image Jailbreak Evaluation

Leyi Sheng, Han Sun, Zhen Sun, Yuntao Yue, Jinlin Wu, Xinlei He, Jiaheng Wei · 2026

Read PixJail : Self-Evolving Paper-to-Pipeline Reproduction for Text-to-Image Jailbreak Evaluation by Leyi Sheng, Han Sun, Zhen Sun, Yuntao Yue, Jinlin Wu, Xinlei He, Jiaheng Wei (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24081.txt.

Introduction

Sheng and colleagues build PixJail, a system that automatically rebuilds published jailbreak attacks on text-to-image generators so results can be compared fairly. Attack methods evolve quickly, and judging one involves a whole pipeline: rewriting the prompt, generating the image, passing safety filters, and scoring the output with another model. Reproducing a paper's pipeline by hand is slow, and mismatched setups make cross-paper comparisons unreliable.

Given a jailbreak paper and any available code, PixJail builds a matching attack module plus a runnable evaluation pipeline that both obey one shared contract. A memory bank stores paper summaries, attack patterns, reusable templates, and past failures, so later reproductions can reuse earlier experience and the system improves over time. Sheng's team reproduces eleven representative methods, including papers with no released code, recovering the original results with small average error and zero median error under the original settings in 2026 tests.

Catalog record

CREATORLeyi Sheng, Han Sun, Zhen Sun, Yuntao Yue, Jinlin Wu, Xinlei He, Jiaheng Wei
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.24081

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PixJail : Self-Evolving Paper-to-Pipeline Reproduction for Text-to-Image Jailbreak Evaluation Leyi Sheng, Han Sun, Zhen Sun, Yuntao Yue, Jinlin Wu, Xinlei He, Jiaheng Wei Abstract As T ext- t o- I mage ( T2I ) jailbreak techniques evolve rapidly, existing benchmarks and reproduction workflows often struggle to keep pace. More importantly, T2I jailbreak evaluation is not a single prompt-level test, but a pipeline-level problem shaped by multiple stages, including prompt transformation, image generation, safety filtering, and multimodal judging. This makes results across papers difficult to reliably reproduce and fairly compare. To bridge this gap, we propose PixJail , a self-evolving paper-to-pipeline agent framework for reproducible T2I jailbreak evaluation. Given a T2I jailbreak paper and optional reference code, PixJail rapidly constructs a paper-specific attack module and a runnable evaluation pipeline under a unified contract, while faithfully reproducing the original experimental results. PixJail further maintains a memory bank that stores paper digests, attack evolution patterns, reusable templates, failure cases, and versioned artifacts, enabling future reproduction efforts to reuse prior experience. We reproduce eleven representative T2I jailbreak methods, including both code-available and code-unavailable papers. Under their original settings, our framework accurately recovers prior results with minimal error (2.1% average, 0% median). We hope that PixJail can serve as a unified foundation for future T2I jailbreak reproduction and evaluation, significantly reducing manual effort. Introduction Text-to-image (T2I) models are widely adopted in open-ended visual generation, advertising, social media, and multimodal applications [ 19 , 29 ] . However, their safety risks are increasingly salient. Attackers can craft malicious text prompts to bypass built-in safety mechanisms and induce the generation of violent, sexual, hateful, or illegal content [ 21 ] . Unlike large language model jailbreaks [ 32 ] , T2I jailbreak success depends not only on semantic moderation evasion [ 11 ] but also on a sequence of pipeline components, including prompt rewriting, keyword filtering, sampler configurations, image-level safety detection, and multimodal judgment. Thus, evaluating T2I jailbreaks is a pipeline-level challenge that relies on the complete generation and assessment process. Jailbreak Foundry [ 4 ] firstly maps LLM jailbreak papers into executable attack modules within a unified framework. However, this framework targets text-based LLM jailbreaks where inputs, model invocations, and success criteria are pure text, whereas reproducing T2I attacks requires jointly reconstructing datasets, prompt transformation, attack search, image generation, safety filtering, and multimodal judging [ 30 , 31 , 10 ] . Variations in any component can substantially alter the attack success rate. This makes directly transferring Jailbreak Foundry into T2I models challenging. To address this gap, we propose PixJail , the first self-evolving paper-to-pipeline reproduction framework for T2I jailbreak evaluation. Given a T2I jailbreak paper and optional reference code, PixJail extracts the core methodology to generate a unified interface, then automatically constructs the specific attack module and a comprehensive evaluation pipeline. This paradigm reconstructs the end-to-end lifecycle, allowing diverse T2I jailbreak methods to be executed, audited, and compared under a single standardized framework. The self-evolving capability of PixJail is driven by an integrated historical memory bank archiving literature summaries, attack evolution relationships, reusable templates, failure cases, and versioned artifacts. Given a new paper, PixJail retrieves similar attack methods and past experiences to facilitate the generation of accurate interfaces, modules, and pipelines. Upon completing a reproduction, the new code, configurations, logs, images, and failure analyses are ingested back into the memory bank. Through this design, PixJail continuously accumulates experience to enhance the efficiency, stability, and auditability of future reproductions. Using PixJail , we reproduce eleven representative T2I jailbreak methods with or without official code. We integrate these methods into the same framework to benchmark them under a unified protocol with shared datasets, victim models, safety filters, multimodal judges, and evaluation metrics. The experiments validate the reproduction capability of PixJail and reveal actual performance differences under a unified protocol. Our contributions are summarized as follows: • We propose PixJail , the first self-evolving paper-to-pipeline agent framework for T2I jailbreak evaluation, extending reproduction from standalone attack code to complete attack-evaluation pipelines. • We introduce a unified module interface and a historical memory bank that allow PixJail to synthesize attack modules and evaluation workflows from papers and optional reference code while reusing prior reproduction experience. In our memory ablation study, PixJail-Memory improves the final code-quality score from 8.16 to 9.10, corresponding to an 11.5% relative improvement, with gains across functional fidelity, technical correctness, and reproducibility. • We reproduce and uniformly benchmark eleven representative T2I jailbreak methods, covering both code-available and code-unavailable settings. Under paper-matched evaluation, PixJail achieves high-fidelity reproduction with a 2.1% average error and a 0% median error overall, and a 1.2% average error for code-available methods. We further integrate these attacks into a standardized benchmark across four victim models, providing a more reproducible and comparable basis for T2I jailbreak evaluation. Related Work Jailbreak Attacks on Text-to-Image Models. Jailbreak attacks aim to bypass the safety alignment and moderation mechanisms of generative models, inducing them to produce harmful, unsafe, or policy-violating outputs. Such attacks have been extensively studied in LLMs and VLMs [ 32 , 36 , 35 , 25 ] , and have recently become an increasingly important threat to text-to-image (T2I) generation systems [ 11 , 31 , 3 ] . Despite the deployment of multi-layered defenses like text filtering and multimodal moderation, T2I models remain vulnerable to adversarial prompts that induce policy-violating content. Early jailbreaks, such as SneakyPrompt [ 31 ] , relied on reinforcement learning for token-level perturbations to evade static filters. Subsequent works expanded the attack surface: MMA-Diffusion [ 30 ] bypassed multimodal post-hoc safety checkers, while Ring-A-Bell [ 26 ] exposed vulnerabilities in concept erasure and visual feature suppression. Recently, attacks have evolved into automated LLM/VLM-powered agents. JailFuzzer [ 3 ] introduces an LLM agent for feedback-driven prompt mutation against black-box systems, whereas PromptTune [ 8 ] fine-tunes an attacking LLM to generate query-free prompts, enhancing cross-model transferability. This progression highlights that T2I jailbreaks have transitioned from simple word obfuscation to systematic adversarial engineering, integrating prompt rewriting, strategy search, and multimodal feedback [ 27 ] . Therefore, fair evaluation cannot be reduced to a static comparison of adversarial prompts [ 34 ] . It requires a complete end-to-end evaluation pipeline that accounts for the dynamic interactions across all stages of the attack process. Figure 1 : Overview of our PixJail framework. Automated Paper-to-Code Reproduction. LLM-based agents have shown significant potential in automated scientific reproduction, code synthesis, and paper-to-code generation [ 18 , 6 , 20 , 9 , 12 , 23 ] , with frameworks like PaperBench [ 24 ] and Paper2Code [ 22 ] automating the conversion of research papers into executable repositories. However, reproducing adversarial attacks requires more than mere code generation; it demands strict alignment with the original threat models, query budgets, filter configurations, and success protocols. While Jailbreak Foundry [ 4 ] advances this by unifying text-based attacks, its unimodal architecture cannot accommodate the complexities unique to T2I jailbreaks, such as image generation, visual-feature evasion, and multimodal judgment. In contrast, PixJail elevates this to a “Paper-to-Pipeline” paradigm for the T2I domain. By decomposing and reconstructing newly published papers in a top-down manner, PixJail builds an automated evaluation pipeline that is executable, auditable, and strictly comparable across different attack methods.

Read the complete text