Why does weak-OOD help ? A Further Step Towards Understanding Jailbreaking VLMs
Yuxuan Zhou 1 † , Yuzhao Peng 1 † , Yang Bai 2 , Kuofeng Gao 1 , Yihao Zhang 3 , Yechao Zhang 4 , Xun Chen 2 , Tao Yu 5 , Tao Dai 6 ∗ , Shu-Tao Xia 1 1 Tsinghua University, 2 ByteDance, 3 Peking University, 4 Nanyang Technological University, 5 CASIA, 6 Shenzhen University † Equal contribution. ∗ Co · 2025
Read Why does weak-OOD help ? A Further Step Towards Understanding Jailbreaking VLMs by Yuxuan Zhou 1 † , Yuzhao Peng 1 † , Yang Bai 2 , Kuofeng Gao 1 , Yihao Zhang 3 , Yechao Zhang 4 , Xun Chen 2 , Tao Yu 5 , Tao Dai 6 ∗ , Shu-Tao Xia 1 1 Tsinghua University, 2 ByteDance, 3 Peking University, 4 Nanyang Technological University, 5 CASIA, 6 Shenzhen University † Equal contribution. ∗ Co (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-08367.txt.
Description
Yuxuan Zhou and co-authors explain why mildly out of distribution inputs jailbreak vision language models better than either clean or heavily altered ones. Safety aligned multimodal models can be tricked by perturbed images or embedded text, and out of distribution attacks are simple yet effective. The authors name the sweet spot weak OOD and trace it to a trade off between two forces: how well the model reads user intent, and how strongly its refusal system triggers.
Those forces respond differently as perturbation grows, consistent with broad pretraining robustness sitting beside narrower safety tuning. Building on optical character recognition, a core pretraining skill, they design JOCR, an OCR aware embedded text attack that achieves the best average success rate among tested baselines.
Catalog record
| CREATOR | Yuxuan Zhou 1 † , Yuzhao Peng 1 † , Yang Bai 2 , Kuofeng Gao 1 , Yihao Zhang 3 , Yechao Zhang 4 , Xun Chen 2 , Tao Yu 5 , Tao Dai 6 ∗ , Shu-Tao Xia 1 1 Tsinghua University, 2 ByteDance, 3 Peking University, 4 Nanyang Technological University, 5 CASIA, 6 Shenzhen University † Equal contribution. ∗ Co |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-08367