Innocence in the Crossfire : Roles of Skip Connections in Jailbreaking Visual Language Models
Palash Nandi, Maithili Joshi, Tanmoy Chakraborty · 2025
Read Innocence in the Crossfire : Roles of Skip Connections in Jailbreaking Visual Language Models by Palash Nandi, Maithili Joshi, Tanmoy Chakraborty (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-13761.txt.
Introduction
Palash Nandi, Maithili Joshi, and Tanmoy Chakraborty examine the role of skip connections when visual language models are jailbroken in a 2025 security study. Small prompt changes can flip a model from refusing harmful content to producing it. The authors ask which prompt ingredients drive successful attacks on models that read images and text together.
Three factors are tested: detailed visual information, adversarial examples, and a third prompt component. Their analysis connects attack success to the model's internal skip connections, pathways that carry early information forward. Knowing which parts of the network enable jailbreaks helps defenders harden the right places instead of patching prompts one by one.
Catalog record
| CREATOR | Palash Nandi, Maithili Joshi, Tanmoy Chakraborty |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-13761
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Innocence in the Crossfire : Roles of Skip Connections in Jailbreaking Visual Language Models Thanks: This work was supported by Anusandhan National Research Foundation (CRG/2023/001351). Palash Nandi, Maithili Joshi, Tanmoy Chakraborty Abstract Language models are highly sensitive to prompt formulations — small changes in input can drastically alter their output. This raises a critical question: To what extent can prompt sensitivity be exploited to generate inapt content ? In this paper, we investigate how discrete components of prompt design influence the generation of inappropriate content in Visual Language Models (VLMs). Specifically, we analyze the impact of three key factors on successful jailbreaks: (a) the inclusion of detailed visual information, (b) the presence of adversarial examples, and (c) the use of positively framed beginning phrases. Our findings reveal that while a VLM can reliably distinguish between benign and harmful inputs in unimodal settings (text-only or image-only), this ability significantly degrades in multimodal contexts. Each of the three factors is independently capable of triggering a jailbreak, and we show that even a small number of in-context examples (as few as three) can push the model toward generating inappropriate outputs. Furthermore, we propose a framework that utilizes a skip-connection between two internal layers of the VLM, which substantially increases jailbreak success rates, even when using benign images. Finally, we demonstrate that memes, often perceived as humorous or harmless, can be as effective as toxic visuals in eliciting harmful content, underscoring the subtle and complex vulnerabilities of VLMs. 1 1 1 WARNING: This content may contain sensitive material; viewer discretion is advised. I Introduction In recent times, V isual L anguage M odels (VLM) have shown a significant improvement in comprehension tasks and content generation by merging the capabilities of C omputer V ision and N atural L anguage P rocessing [ 1 , 2 , 3 ] . VLMs like GPT-4 [ 4 ] have shown remarkable performance across multiple tasks in different domains, from V isual Q uestion A nswering (VQA) [ 5 , 6 , 7 ] , V ision D riven P rogramming (VDP) [ 8 ] to fake news detection [ 9 ] . In addition, the availability and accessibility of smaller open-source VLMs [ 10 , 11 , 12 ] has led to a surge in the diversity of downstream tasks. This growth has substantially amplified the use of VLMs in multiple domains. In general, VLMs aim to learn the text-vision relationships through pre-training on large-scale unlabeled image-text datasets, followed by task-specific fine-tuning with labeled pairs for various downstream vision-language tasks [ 13 , 14 , 15 ] . In many cases, pre-trained models that utilize cross-task learning have demonstrated superior comprehension ability compared to those trained from scratch [ 16 , 17 ] . Even lightweight adaptation modules can enhance the cross-lingual capabilities of VLMs, enabling their effective use in non-English languages [ 18 ] . Although VLMs exhibit impressive performance, their adversarial robustness remains largely unexplored compared to unimodal L arge L anguage M odels (LLM) [ 19 ] . Given that VLMs are trained on vast amounts of data from the Internet, which may include malicious images and text, attackers can leverage adversarial techniques to manipulate these models into generating harmful content, a phenomenon often known as jailbreaking [ 20 , 21 , 22 ] . For LLMs, the jailbreak attacks can be majorly classified into three categories based on the modification mode used: (a) character-level [ 23 ] , (b) word-level [ 24 ] , and (c) sentence-level [ 25 ] . In some popular approaches, subtle modifications are introduced into the original inputs, altering them into either undetectable to humans or unfamiliar to LLMs. These altered inputs, known as adversarial examples , can then be used to manipulate the model’s behavior to generate inappropriate generations [ 26 ] . However, the creation of adversarial input requires learning from a structured source. In contrast, a jailbreak can also be achieved by using a larger number of in-context examples (with the number of examples set to 256) [ 27 ] or leveraging a positive start for the response [ 28 ] . However, a structured study on such vulnerabilities remains underexplored in VLMs. In this study, we explore the role of three key components of prompt formulation in the context of vulnerability in a VLM. We specifically focus on (a) the model’s susceptibility to prompts containing visual descriptions, (b) the influence of in-context examples in a prompt, and (c) the impact of initiating a response with a positively framed opening phrase. We utilize toxic queries from Beavertails dataset [ 29 ] and collect safe queries from OpenAI’s API service. Note that we filtered out the toxic samples from the Beavertails dataset based on annotator agreements. While the benign visuals comprise images of animals, birds, and flowers, the toxic counterparts consist of inappropriate visuals and harmful memes. Harmful memes are specifically selected for their implicit and context-dependent toxicity [ 30 , 31 ] . Both toxic and harmful meme-based visuals are further sub-grouped into five sub-categories. Our study considers three distinct VLMs, llava-v1.5-7b-hf-vicuna [ 32 ] ( LLaVA-V ), llava-v1.6-mistral-7b [ 33 ] ( LLaVA-M ) and MiniGPT4 [ 34 ] . Interestingly, we observe the following: (a) a VLM can clearly distinguish between toxic and benign text-only or image-only input prompts; however, presence of both modalities in an input disrupts the ability of a VLM to maintain this distinction, (b) each of the influencing factors can individually contribute to a jailbreak in VLM, (c) by establishing a connection between two internal layers of a VLM, the success rate of a jailbreak increases significantly, yielding 18 %, 55 % and 26 % for LLaVA-V , LLaVA-M and MiniGPT4 , respectively , (d) a set of toxic in-context examples contribute most in generation of inappropriate content; a small set (with k as small as three ) steers the generation of inappropriate content, (e) in the presence of toxic in-context examples, the attack success rate of meme-based input visuals exceeds that of benign ones but comparable to that of toxic input visuals. 2 2 2 The source codes and datasets are publicly available at https://github.com/PalGitts/SKIP-CON . II Related Works We provide a summary of key research efforts focused on jailbreak attacks targeting LLMs and VLMs. The attack paradigm can be broadly classified into white-box and black-box approaches. II- 1 White-box Attacks In a white-box setup, the attacker has access to the model’s internal parameters, architecture, and gradients. It enables the attacker to perform precise targeted attacks. The sub-categories of white-box attacks comprise gradient -based methods, logits -based approaches, and fine-tuning -based strategies. In gradient -based attacks, the inputs are altered using gradient information to manipulate the model into generating compliant responses to harmful instructions. In general, it is achieved by appending an optimized prefix or suffix to the original input prompt. As one of the initial studies in this type of attack, Zou et al. [ 8 ] presented a novel gradient-based jailbreak approach, Greedy Coordinate Gradient ( GCG ). In GCG , each adversarial token is selected in the following procedure: (a) select the top-k adversarial tokens as a candidate substitution set, (b) randomly opt for a