Why does weak-OOD help ? A Further Step Towards Understanding Jailbreaking VLMs

Yuxuan Zhou 1 † , Yuzhao Peng 1 † , Yang Bai 2 , Kuofeng Gao 1 , Yihao Zhang 3 , Yechao Zhang 4 , Xun Chen 2 , Tao Yu 5 , Tao Dai 6 ∗ , Shu-Tao Xia 1 1 Tsinghua University, 2 ByteDance, 3 Peking University, 4 Nanyang Technological University, 5 CASIA, 6 Shenzhen University † Equal contribution. ∗ Co · 2025

Read Why does weak-OOD help ? A Further Step Towards Understanding Jailbreaking VLMs by Yuxuan Zhou 1 † , Yuzhao Peng 1 † , Yang Bai 2 , Kuofeng Gao 1 , Yihao Zhang 3 , Yechao Zhang 4 , Xun Chen 2 , Tao Yu 5 , Tao Dai 6 ∗ , Shu-Tao Xia 1 1 Tsinghua University, 2 ByteDance, 3 Peking University, 4 Nanyang Technological University, 5 CASIA, 6 Shenzhen University † Equal contribution. ∗ Co (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-08367.txt.

Description

Yuxuan Zhou and co-authors explain why mildly out of distribution inputs jailbreak vision language models better than either clean or heavily altered ones. Safety aligned multimodal models can be tricked by perturbed images or embedded text, and out of distribution attacks are simple yet effective. The authors name the sweet spot weak OOD and trace it to a trade off between two forces: how well the model reads user intent, and how strongly its refusal system triggers.

Those forces respond differently as perturbation grows, consistent with broad pretraining robustness sitting beside narrower safety tuning. Building on optical character recognition, a core pretraining skill, they design JOCR, an OCR aware embedded text attack that achieves the best average success rate among tested baselines.

Catalog record

CREATORYuxuan Zhou 1 † , Yuzhao Peng 1 † , Yang Bai 2 , Kuofeng Gao 1 , Yihao Zhang 3 , Yechao Zhang 4 , Xun Chen 2 , Tao Yu 5 , Tao Dai 6 ∗ , Shu-Tao Xia 1 1 Tsinghua University, 2 ByteDance, 3 Peking University, 4 Nanyang Technological University, 5 CASIA, 6 Shenzhen University † Equal contribution. ∗ Co
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-08367

Browse the full library · About the Tower of Babel