Robustness of Vision Language Models Against Split-Image Harmful Input Attacks

Md Rafi Ur Rashid, MD Sadik Hossain Shanto, Vishnu Asutosh Dasu, Shagufta Mehnaz · 2026

Read Robustness of Vision Language Models Against Split-Image Harmful Input Attacks by Md Rafi Ur Rashid, MD Sadik Hossain Shanto, Vishnu Asutosh Dasu, Shagufta Mehnaz (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-08136.txt.

Description

Vision-language models resist jailbreaks hidden in a single image, thanks to safety alignment, but what if the harmful meaning only emerges by combining image fragments? Md Rafi Ur Rashid and three co-authors identify exactly that gap: alignment training uses whole images, while pretraining generalizes readily to split inputs.

Their split-image visual jailbreak attacks distribute unsafe cues across fragments so no single piece triggers refusal, yet the combination does the harm. Unlike optimization-based attacks, these transfer well across models in black-box settings. The paper exposes a misalignment between capability generalization and safety generalization, urging safety teams to align on the fragmented inputs attackers actually use, not just clean single images.

Catalog record

CREATORMd Rafi Ur Rashid, MD Sadik Hossain Shanto, Vishnu Asutosh Dasu, Shagufta Mehnaz
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE93 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-08136

Browse the full library · About the Tower of Babel