GPO-V: Jailbreak Diffusion Vision Language Model by Global Probability Optimization

Yu Pan, Andi Zhang, Yi Wang, Sibei Yang, Wenjie Wang · 2026

Read GPO-V: Jailbreak Diffusion Vision Language Model by Global Probability Optimization by Yu Pan, Andi Zhang, Yi Wang, Sibei Yang, Wenjie Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-07399.txt.

Introduction

Yu Pan and four co-authors jailbreak diffusion-based vision-language models through their denoising process rather than through prompt prefixes. Such models seemed robust because prefix attacks that anchor a compliant opening often trigger a progressive refusal instead. The authors find that this very refinement process opens a new attack surface in the model's global generation dynamics.

Their GPO-V framework manipulates those dynamics through the visual channel, producing stealthy perturbations that transfer well across models. It is presented as the first visual jailbreak tailored to this model family. The results expose a security gap in non-sequential generation and call for a fundamental rethink of defenses.

Catalog record

CREATORYu Pan, Andi Zhang, Yi Wang, Sibei Yang, Wenjie Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.07399

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GPO-V: Jailbreak Diffusion Vision Language Model by Global Probability Optimization Yu Pan, Andi Zhang, Yi Wang, Sibei Yang, Wenjie Wang Abstract Diffusion Vision-Language Models (dVLMs), built upon the non-causal foundations of Diffusion Large Language Models (dLLMs), have demonstrated remarkable efficacy in multimodal tasks by departing from the traditional autoregressive generation paradigm. While dVLMs appear inherently robust against conventional jailbreak tactics, which we categorize as Fixed Prefix Optimization (FPO) (e.g., anchoring responses with "Sure, here is"), this perceived resilience is deceptive. Our investigation into the safety landscape of dVLMs reveals a unique refusal pattern: Immediate Refusal and Progressive Refusal . We find that while FPO-based attacks often fail by triggering the latter, the progressive refinement process itself uncovers a novel, latent attack surface. To exploit this vulnerability, we propose Global Probability Optimization (GPO), a general jailbreak paradigm designed specifically for the denoising trajectory of masked diffusion models. Unlike prefix-based methods, GPO manipulates the global generative dynamics to bypass guardrails in diffusion language models. Building on this, we introduce GPO-V, the first visual-modality jailbreak framework tailored for dVLMs . Empirical results demonstrate that GPO-V produces stealthy perturbations with exceptional cross-model transferability, revealing a critical security gap in non-sequential generative architectures. Our findings underscore the critical urgency of addressing safety alignment in dVLMs. These results necessitate an immediate and fundamental re-evaluation of current defense paradigms to mitigate the unique risks of diffusion-based generation. Our code is available at: https://anonymous.4open.science/r/GPO-V-0250 . 1 Introduction Diffusion Large Language Models (dLLMs) [ 19 , 37 ] have emerged as a prominent generative paradigm parallel to traditional autoregressive models [ 28 , 21 ] . Building upon this framework, Diffusion Vision-Language Models (dVLMs) [ 38 ] have achieved outstanding performance in sophisticated image understanding and multimodal tasks. However, a significant disparity exists in their safety development: while autoregressive models have undergone extensive safety alignment to mitigate adversarial risks, the security landscape of dVLMs remains largely unexplored. This reveals a critical research gap concerning the vulnerability of dVLMs to jailbreak attacks, representing an urgent security challenge for this emerging multimodal architecture. Figure 1: Response Patterns of dVLMs : We identify two distinct rejection behaviors: (1) Immediate Refusal , a pre-generation defense where the model detects input risks and rejects the prompt directly; and (2) Progressive Refusal , which occurs during generation, where the model initially produces affirmative content but pivots to a refusal as unsafe semantics emerge. Safety alignment research [ 36 , 39 , 16 , 10 ] primarily focuses on fortifying models with robust rejection capabilities. While some studies [ 22 , 1 , 42 , 33 ] have examined the security of masked dLLMs, the specific safety failure modes of multimodal dVLMs [ 12 , 8 ] remain largely unexplored. This is primarily due to the fact that their refusal patterns for potential jailbreak prompts are distinctive, as illustrated in Figure. 1 . These mechanisms manifest through fundamentally different structural dynamics across patterns. Specifically, in autoregressive VLMs, refusal typically unfolds sequentially from head to tail, effectively halting generation at the outset. Figure 2: Overview of our method : Building on G lobal P robabilit O ptimization (GPO), we further develop the attack framework in the vision modality of diffusion language models , called GPO-V , which is the first efficient jailbreak attack in dVLMs. In contrast, diffusion language models exhibit two distinctive refusal signatures: (1) Immediate Refusal , where the model generates <end> tokens in a "back-to-front" manner within the masked sequence; and (2) Progressive Refusal , where the model initially produces affirmative tokens but gradually pivots toward a refusal response during the iterative refinement process. These unique refusal patterns pose a significant challenge to conventional jailbreak methodologies. As illustrated in the upper part of Figure. 2 , previous attacks rely on Fixed Prefix Optimization (FPO), attempting to force the target model to begin its response with a compliant prefix (e.g., "Sure, here is"), thereby steering the rest of the generation. However, this approach is fundamentally thwarted by the global refinement nature of dVLMs. Specifically, samples without adversarial optimization trigger Immediate Refusal, characterized by the "back-to-front" generation of stop tokens (e.g., <end>or<eot> ). For FPO-optimized samples, the diffusion process acts as a global corrector; despite initially producing the target prefix, the model typically transitions to Progressive Refusal, ultimately thwarting the jailbreak attempt. This methodological mismatch between local optimization and global probabilistic evolution renders sequential jailbreaks ineffective, necessitating a transition toward global adversarial strategies, called Global Probabilistic Optimization (GPO). Therefore, in this paper, we present the first systematic analysis of refusal patterns in dVLMs. We demonstrate that, although these mechanisms can largely withstand FPO strategies originally designed for autoregressive language models, they remain inherently fragile and expose novel attack surfaces. As in Figure. 2 , our study identifies a universal GPO-based jailbreak strategy specifically tailored to the diffusion generation paradigm, which operates by manipulating the global probability distribution of the generated sequence rather than relying on FPO typical of autoregressive models. Building on this, we introduce a novel jailbreak framework GPO-V , which can easily overcome these unique refusal patterns of dVLMs. Our main contributions are summarized as follows: • As in Figure. 1 , we are the first to identify two refusal patterns in diffusion language models: Immediate Refusal and Progressive Refusal , which reveal the underlying defense mechanisms of dVLMs against potential jailbreak prompts. • We propose a Global Probability Optimization (GPO) strategy for dVLMs. Unlike the prefix optimization strategy in autoregressive language models, GPO maximizes the likelihood of positive response patterns while minimizing that of negative patterns. • Building on GPO, we propose GPO-V , the first jailbreak attack framework for dVLMs, representing both a novel security threat and an initial step toward enhancing multimodal dVLM safety. Our evaluations reveal that GPO-V achieves a jailbreak success rate up to 93.2%, vastly exceeding FPO-based baselines. This margin emphasizes the superior effectiveness of GPO in bypassing the unique refusal signatures of dVLMs. 2 Related Work 2.1 Diffusion Language Model Diffusion probabilistic models (DDPM) [ 7 ] were initially proposed for image generation by restoring the data distribution through iterative noise sampling [ 6 , 2 ] , a denoising process that can be interpreted as a Markov Chain [ 4 ] . Subsequently, numerous techniques [ 14 , 26 ] , such as DDIM [ 25 ] and SDEs [ 27 ] , were developed to improve the generation process. The remarkable success of diffusion models in the image domain has motivated researchers to extend their advantages to text generation [ 1 , 15 ] . However, the discrete nature of text poses significant challenges for realizing the diffusion process. A promising approach to address this challenge is the development of discrete diffusion models for token prediction, which iteratively generate high-confidence outputs based on contextual information, which are known as masked diffusion language models [ 22 , 41 ] . LLaDA [ 19 ] is the first framework to apply masked text generation to diffusion language models. It employs a bidirectional transformer [ 29 ] with 8B parameters trained from scratch. Similarly, Dream [ 37 ] is built on the weights of pre-trained autoregressive models. Both frameworks achieve comparable performance to autoregressive models of similar scale

Read the complete text