Robustness of Vision Language Models Against Split-Image Harmful Input Attacks
Md Rafi Ur Rashid, MD Sadik Hossain Shanto, Vishnu Asutosh Dasu, Shagufta Mehnaz · 2026
Read Robustness of Vision Language Models Against Split-Image Harmful Input Attacks by Md Rafi Ur Rashid, MD Sadik Hossain Shanto, Vishnu Asutosh Dasu, Shagufta Mehnaz (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-08136.txt.
Description
Vision-language models resist jailbreaks hidden in a single image, thanks to safety alignment, but what if the harmful meaning only emerges by combining image fragments? Md Rafi Ur Rashid and three co-authors identify exactly that gap: alignment training uses whole images, while pretraining generalizes readily to split inputs.
Their split-image visual jailbreak attacks distribute unsafe cues across fragments so no single piece triggers refusal, yet the combination does the harm. Unlike optimization-based attacks, these transfer well across models in black-box settings. The paper exposes a misalignment between capability generalization and safety generalization, urging safety teams to align on the fragmented inputs attackers actually use, not just clean single images.
Catalog record
| CREATOR | Md Rafi Ur Rashid, MD Sadik Hossain Shanto, Vishnu Asutosh Dasu, Shagufta Mehnaz |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 93 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-08136