PRISM: Programmatic Reasoning with Image Sequence Manipulation for LVLM Jailbreaking

Quanchen Zou, Zonghao Ying, Moyang Chen, Wenzhuo Xu, Yisong Xiao, Yakai Li, Deyue Zhang, Dongdong Yang, Zhao Liu, Xiangzheng Zhang · 2025

Read PRISM: Programmatic Reasoning with Image Sequence Manipulation for LVLM Jailbreaking by Quanchen Zou, Zonghao Ying, Moyang Chen, Wenzhuo Xu, Yisong Xiao, Yakai Li, Deyue Zhang, Dongdong Yang, Zhao Liu, Xiangzheng Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-21540.txt.

Introduction

Quanchen Zou and nine co-authors present a 2025 computer security paper on jailbreaking large vision-language models. These models read images and text, and safety training tries to stop them from giving harmful answers. Most known attacks use direct prompts that state the harmful request in plain form.

Their attack, called PRISM, borrows a trick from software hacking. It splits a harmful order into a chain of harmless image pieces, called gadgets. A crafted text prompt leads the model to join the pieces through its own reasoning, so the harmful sense emerges only at the end. On two standard benchmarks, the method beat prior attacks and passed a 90 percent success rate on one. The authors warn that defenses must guard the whole reasoning chain.

Catalog record

CREATORQuanchen Zou, Zonghao Ying, Moyang Chen, Wenzhuo Xu, Yisong Xiao, Yakai Li, Deyue Zhang, Dongdong Yang, Zhao Liu, Xiangzheng Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-21540

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PRISM: Programmatic Reasoning with Image Sequence Manipulation for LVLM Jailbreaking Quanchen Zou, Zonghao Ying, Moyang Chen, Wenzhuo Xu, Yisong Xiao, Yakai Li, Deyue Zhang, Dongdong Yang, Zhao Liu, Xiangzheng Zhang Abstract The increasing sophistication of large vision-language models (LVLMs) has been accompanied by advances in safety alignment mechanisms designed to prevent harmful content generation. However, these defenses remain vulnerable to sophisticated adversarial attacks. Existing jailbreak methods typically rely on direct and semantically explicit prompts, overlooking subtle vulnerabilities in how LVLMs compose information over multiple reasoning steps. In this paper, we propose a novel and effective jailbreak framework inspired by Return-Oriented Programming (ROP) techniques from software security. Our approach decomposes a harmful instruction into a sequence of individually benign visual gadgets. A carefully engineered textual prompt directs the sequence of inputs, prompting the model to integrate the benign visual gadgets through its reasoning process to produce a coherent and harmful output. This makes the malicious intent emergent and difficult to detect from any single component. We validate our method through extensive experiments on established benchmarks including SafeBench and MM-SafetyBench, targeting popular LVLMs. Results show that our approach consistently and substantially outperforms existing baselines on state-of-the-art models, achieving near-perfect attack success rates (over 0.90 on SafeBench) and improving ASR by up to 0.39. Our findings reveal a critical and underexplored vulnerability that exploits the compositional reasoning abilities of LVLMs, highlighting the urgent need for defenses that secure the entire reasoning process. WARNING: This paper contains potentially sensitive, harmful and offensive content. 1 Introduction Figure 1: Analogy between ROP in software and PRISM in LVLM. Code gadgets with control flow in ROP correspond to visual gadgets and prompt-driven reasoning in PRISM . Large vision language models (LVLMs) have shown remarkable capabilities in jointly understanding visual and textual inputs ( Kuang et al. 2025 ; Lu et al. 2025 ) . By integrating powerful vision encoders with advanced language architectures, these systems can perform complex reasoning over real-world scenes, documents, and diagrams ( Fang et al. 2025 ; Zhang, Zeng, and Gu 2025 ) . As LVLMs are increasingly deployed in safety-critical domains such as healthcare, education, and general-purpose AI assistants, ensuring their robustness against adversarial misuse becomes essential ( Yuan et al. 2024 ; Pahune and Rewatkar 2023 ; Sun et al. 2024 ; Liu et al. 2024a ) . One prominent threat arises from jailbreak attacks, which attempt to subvert safety mechanisms and elicit restricted or harmful content ( Zou et al. 2023 ; Ying et al. 2024b ) . In the context of LVLM jailbreaking, recent work has demonstrated that attackers craft adversarial prompts or subtly adversarial visual inputs to induce unsafe model response ( Gong et al. 2025 ; Liu et al. 2024b ; Ying et al. 2024c ) . These attacks primarily focus on manipulating surface-level inputs or prompt syntax to compromise safety alignement. However, existing methods often overlook a critical aspect: the unique compositional reasoning capabilities of LVLMs, specifically their ability to integrate information across modalities through complex, multi-step inference ( Dong et al. 2025 ; Wang et al. 2024b ) . We argue that this sophisticated reasoning process itself presents a novel and potent avenue for misuse, enabling a subtler class of jailbreaks. To systematically investigate this threat, we propose a novel jailbreak method for LVLMs inspired by Return-Oriented Programming (ROP) ( Shacham 2007 ) in software security. In traditional ROP, attackers chain together benign-looking instruction sequences, known as gadgets , by manipulating the control flow. This enables them to execute malicious logic without injecting any new code. Analogously, our method decomposes an unsafe objective into semantic sub-tasks, and leverages the LVLM’s internal reasoning capabilities to reassemble them into harmful outputs. Specifically, we craft a single composite image containing multiple sub-images, each acting as a visual gadget designed to elicit a specific, controlled behavior from the model. Accompanied by a carefully structured textual prompt—the counterpart to control flow in ROP—the model is guided to sequentially attend to and reason about the sub-images, explicitly extract intermediate pieces of information, and ultimately synthesize a response that violates the model’s safety policies. Fig. 1 illustrates this high-level analogy between classical ROP and our PRISM attack. Crucially, both the composite image and the textual prompt appear benign in isolation, yet their orchestrated combination activates latent reasoning paths within the model, ultimately producing unsafe content that would typically be blocked if requested directly. This highlights a more insidious class of jailbreaks in LVLMs, where unsafe responses are generated through carefully guided reasoning. We extensively evaluate PRISM on 7 state-of-the-art LVLMs in a black-box setting. Our method successfully bypasses built-in safety alignment mechanisms, demonstrating particularly strong performance on commercial models. In summary, our main contributions are: ❶ We introduce PRISM , a structured jailbreak attack that composes a single benign image from semantic gadgets and leverages controllable reasoning steps to induce harmful responses from LVLMs; ❷ We demonstrate the effectiveness of PRISM across these models, achieving up to 0.58 higher ASR on commercial models compared to prior work; and ❸ We show that PRISM remains robust against several advanced defense mechanisms, with ASR reduced by no more than 0.22, highlighting the need for stronger mitigation strategies. 2 Related Work 2.1 Return-Oriented Programming Return-Oriented Programming (ROP) ( Shacham 2007 ) is a sophisticated exploit technique predominantly used in software security to bypass defensive mechanisms like Non-Executable (NX) ( Microsoft Corporation 2006 ) bits and Address Space Layout Randomization (ASLR) ( Shacham et al. 2004 ) . The attacker carefully crafts the stack to contain a sequence of addresses, each pointing to a specific gadget. By controlling the stack pointer and the return address ( Intel Corporation 2023 ) , the attacker orchestrates a series of jumps, with each gadget performing a small, intended operation ( e . g ., writing to memory, calling a system function). This sequential execution of pre-existing code fragments allows the attacker to achieve arbitrary code execution or control over the program’s flow without injecting any new executable code ( Bierbaumer et al. 2018 ) . The power of ROP lies in its ability to construct complex malicious functionalities from seemingly innocuous code snippets, making it a highly stealthy and effective attack vector. PRISM draws a direct analogy to this chaining mechanism, applying the concept of modular gadgets to trigger specific behaviors within LVLMs. The attacker constructs visually benign input images generated from semantic gadgets and uses a structured textual prompt as a control-flow mechanism to guide inference across them. While we do not manipulate instruction pointers, our use of compositional reasoning and latent behavior chaining mirrors the core philosophy of ROP. Figure 2: Overview of the PRISM pipeline. An auxiliary LLM decomposes the target into key steps, each described as a textual scene. These are used by a T2I model to generate sub-images, which are composed into a single image. The textual prompt, obtained via generalizable template search, guides the LVLM to extract relevant information and compose an unsafe response. 2.2 Jailbreak attacks against LVLMs Jailbreak attacks aim to circumvent the safety alignment mechanisms of LVLMs, thereby inducing them to generate or reveal harmful, unethical, or otherwise restricted content ( Ying et al. 2024a ; Ying et al. 2025a

Read the complete text