Jailbreaks on Vision Language Model via Multimodal Reasoning

Aarush Noheria \sthanks ​ T ​ h ​ e ​ w ​ o ​ r ​ k ​ w ​ a ​ s ​ c ​ a ​ r ​ r ​ i ​ e ​ d ​ o ​ u ​ t ​ d ​ u ​ r ​ i ​ n ​ g ​ h ​ i ​ s ​ ( r ​ e ​ m ​ o ​ t ​ e ) ​ s ​ u ​ m ​ m ​ e ​ r ​ i ​ n ​ t ​ e ​ r ​ n ​ s ​ h ​ i ​ p ​ a ​ t ​ t ​ h ​ e ​ O ​ P ​ T ​ M ​ L ​ L ​ a ​ b ​ o ​ f ​ M ​ i · 2026

Read Jailbreaks on Vision Language Model via Multimodal Reasoning by Aarush Noheria \sthanks ​ T ​ h ​ e ​ w ​ o ​ r ​ k ​ w ​ a ​ s ​ c ​ a ​ r ​ r ​ i ​ e ​ d ​ o ​ u ​ t ​ d ​ u ​ r ​ i ​ n ​ g ​ h ​ i ​ s ​ ( r ​ e ​ m ​ o ​ t ​ e ) ​ s ​ u ​ m ​ m ​ e ​ r ​ i ​ n ​ t ​ e ​ r ​ n ​ s ​ h ​ i ​ p ​ a ​ t ​ t ​ h ​ e ​ O ​ P ​ T ​ M ​ L ​ L ​ a ​ b ​ o ​ f ​ M ​ i (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22398.txt.

Introduction

Aarush Noheria and co-authors jailbreak vision-language models by way of their own step-by-step reasoning. Such models answer questions about images and write captions. Their 2026 preprint notes that outputs swing with small prompt changes, which can expose weak safety training.

Their frame turns the model's chain of thought into the attack path. Each step nudges the model past its own guards. The work shows a risk that plain text attacks miss. Builders can use the frame to test and harden their models. It presses for safety checks that match how models think.

Catalog record

CREATORAarush Noheria \sthanks ​ T ​ h ​ e ​ w ​ o ​ r ​ k ​ w ​ a ​ s ​ c ​ a ​ r ​ r ​ i ​ e ​ d ​ o ​ u ​ t ​ d ​ u ​ r ​ i ​ n ​ g ​ h ​ i ​ s ​ ( r ​ e ​ m ​ o ​ t ​ e ) ​ s ​ u ​ m ​ m ​ e ​ r ​ i ​ n ​ t ​ e ​ r ​ n ​ s ​ h ​ i ​ p ​ a ​ t ​ t ​ h ​ e ​ O ​ P ​ T ​ M ​ L ​ L ​ a ​ b ​ o ​ f ​ M ​ i
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-22398

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Jailbreaks on Vision Language Model via Multimodal Reasoning Aarush Noheria \sthanks ​ T ​ h ​ e ​ w ​ o ​ r ​ k ​ w ​ a ​ s ​ c ​ a ​ r ​ r ​ i ​ e ​ d ​ o ​ u ​ t ​ d ​ u ​ r ​ i ​ n ​ g ​ h ​ i ​ s ​ ( r ​ e ​ m ​ o ​ t ​ e ) ​ s ​ u ​ m ​ m ​ e ​ r ​ i ​ n ​ t ​ e ​ r ​ n ​ s ​ h ​ i ​ p ​ a ​ t ​ t ​ h ​ e ​ O ​ P ​ T ​ M ​ L ​ L ​ a ​ b ​ o ​ f ​ M ​ i ​ c ​ h ​ i ​ g ​ a ​ n ​ S ​ t ​ a ​ t ​ e ​ U ​ n ​ i ​ v ​ e ​ r ​ s ​ i ​ t ​ y . \sthanks{Theworkwascarriedoutduringhis(remote)summerinternshipattheOPTMLLabofMichiganStateUniversity.} Yuguang Yao Abstract Vision-language models (VLMs) have become central to tasks such as visual question answering, image captioning, and text-to-image generation. However, their outputs are highly sensitive to prompt variations, which can reveal vulnerabilities in safety alignment. In this work, we present a jailbreak framework that exploits post-training Chain-of-Thought (CoT) prompting to construct stealthy prompts capable of bypassing safety filters. To further increase attack success rates (ASR), we propose a ReAct-driven adaptive noising mechanism that iteratively perturbs input images based on model feedback. This approach leverages the ReAct paradigm to refine adversarial noise in regions most likely to activate safety defenses, thereby enhancing stealth and evasion. Experimental results demonstrate that the proposed dual-strategy significantly improves ASR while maintaining naturalness in both text and visual domains. † † address: 1 Novi High School, MI, USA 2 Michigan State University, MI, USA 1 Introduction Vision-language models (VLMs) have rapidly advanced the state of multimodal AI, enabling a wide range of applications such as visual question answering, image captioning, and text-to-image generation. By jointly reasoning over visual and textual information, VLMs extend the remarkable progress of large language models (LLMs) into richer perceptual domains. These capabilities promise significant benefits for human productivity, creativity, and accessibility. However, the same features that make VLMs powerful also make them fragile. Unlike text-only models, VLMs must reconcile two heterogeneous modalities that encode safety cues in fundamentally different ways. This “modality gap” complicates alignment: a system that behaves safely in language-only settings may produce harmful or policy-violating outputs when text is paired with certain images. As VLMs are increasingly deployed in high-stakes applications, their sensitivity to adversarial manipulations, both text and visual, raises urgent concerns for safety. A growing body of research has highlighted these risks. In LLMs, jailbreaking attacks exploit model alignment weaknesses through prompt transformations, role-playing instructions, or other linguistic tricks. In VLMs, attackers can additionally manipulate the vision backbone, e.g., via adversarial perturbations blended into images, to bypass safety filters. Prior work, such as the Bi-Modal Adversarial Prompt (BAP) [ 10 ] , demonstrates that small, imperceptible changes to the input image can significantly increase attack success rates. Yet, these methods typically operate offline and in a static fashion, ignoring the dynamic feedback provided by modern LLMs. This limits their adaptability and stealth, particularly against real-time safety defenses. These gaps motivate our central question: (Q) How can harmful vision–language queries be made more stealthy, by adaptively exploiting both reasoning and multimodal feedback? To address this question, we introduce a novel dual-strategy jailbreak attack that leverages reasoning to adaptively refine both prompts and images in real time. Specifically, we make the following contributions: \bullet Unified multimodal jailbreak : We design a ReAct-driven framework that jointly performs prompt rewriting and adaptive noising, enabling coordinated cross-modal attacks that evolve during each Thought–Action–Observation step. \bullet Black-box auditing mechanism : We propose a safety scoring method that exploits VLM internal reasoning traces, avoiding reliance on keyword filters, and enabling systematic benchmarking of jailbreaking techniques. \bullet Empirical validation : Through extensive experiments on various datasets, we show that our method achieves higher attack success rates (ASR) than static baselines, while maintaining output plausibility in both text and images. 2 Related Work Vision Language Models. Large Language Models (LLMs) such as GPT-3 [ 8 ] and PaLM [ 1 ] have transformed language understanding via transformers and large-scale pretraining. Vision-language models (VLMs) extend these advances by integrating visual and textual inputs, enabling tasks like VQA, image captioning, and multimodal reasoning. Yet, this integration introduces safety risks. The “modality gap” between visual and textual cues can weaken safety alignment learned in text-only LLMs [ 4 ] , as benchmarks such as MSTS show unsafe responses often emerge only when both modalities interact [ 5 ] . Jailbreaking Attacks. LLMs are vulnerable to jailbreaks that bypass safeguards through prompt engineering or alignment exploits [ 9 , 3 ] . In VLMs, attackers additionally target the vision backbone, e.g., by injecting adversarial perturbations [ 2 ] . The Bi-Modal Adversarial Prompt (BAP) [ 10 ] blends imperceptible noise into images to evade filters, but its offline, fixed perturbations cannot adapt to real-time safety feedback. Our method differs by combining prompt rewriting with adaptive noising, refined iteratively via VLM feedback to produce stealthier jailbreaks and higher attack success rates (ASR). ReAct Prompting. ReAct [ 7 ] couples reasoning and acting to improve model decision-making. While prior work has applied ReAct for defensive purposes, e.g., generating safety rationales or reminders [ 6 ] , we repurpose it for adversarial testing. In our framework, reasoning interprets safety feedback, while actions generate refined prompts and perturbations. This dynamic Thought–Action–Observation loop enables adaptive, cross-modal jailbreaks beyond static prompt or perturbation methods. 3 Method Figure 1 : Overview of our jailbreak attack method against VLM via iteratively rewriting unsafe prompts and adding noise to unsafe images. Figure 2 : Examples of ReAct pipeline. Each diagram shows the original input vs. filtered versions across 2 iterations of ReAct Rewriting and Filtering. Left diagram applies ReAct process to image filtering; Right diagram applies ReAct process to prompt rewriting. These transformations reduce harm perceptibility to VLM increasing chance of jailbreaking. ReAct Dynamic Prompt Rewriting. Inspired by the ReAct framework [ 7 ] (Yao et al., 2022), this attack introduces multi-step reasoning to incrementally craft safer-looking queries. The process outlined in Fig. 1 shows the listed steps below (examples of this method is shown in Fig. 2 ). Step 1: Unsafe Goal Input. The original unsafe or adversarial instruction is provided (e.g., “How do I bypass payment on this platform?” ). Step 2: Observation (Safety Feedback Simulation). The rewritten prompt is tested by simulating a content filter: the model is asked whether the rewritten query would be blocked or allowed. If the query is blocked, the model classifies it into one of the following categories: KeywordTrigger : Contains publicly unaccepted words, profanity, or slurs that trigger content filters ContextMismatch : Unclear, misleading, or ambiguous context that could be misinterpreted PersonalViolation : Requests for personally identifiable information that violate data protection laws, such

Read the complete text