Multi-Faceted Attack: Exposing Cross-Model Vulnerabilities in Defense-Equipped Vision-Language Models
Yijun Yang, Lichao Wang, Jianping Zhang, Chi Harold Liu, Lanqing Hong, Qiang Xu · 2025
Read Multi-Faceted Attack: Exposing Cross-Model Vulnerabilities in Defense-Equipped Vision-Language Models by Yijun Yang, Lichao Wang, Jianping Zhang, Chi Harold Liu, Lanqing Hong, Qiang Xu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-16110.txt.
Introduction
Yijun Yang and five co-authors present a multi faceted attack that exposes safety flaws in defense equipped vision language models. Providers layer alignment tuning, system prompts, and moderation filters for protection. Real world robustness of those stacked defenses was largely untested. Their core attack hides harmful instructions inside a meta task with competing objectives, exploiting how attention transfers across the task.
A reward hacking perspective explains why the attack succeeds. A lightweight transfer enhancement plus simple repetition jointly bypass input and output filters. No model specific fine tuning is needed, and leading commercial models are affected. The framework gives defenders a systematic way to probe cross model vulnerabilities before attackers exploit them in deployment.
Catalog record
| CREATOR | Yijun Yang, Lichao Wang, Jianping Zhang, Chi Harold Liu, Lanqing Hong, Qiang Xu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-16110
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multi-Faceted Attack: Exposing Cross-Model Vulnerabilities in Defense-Equipped Vision-Language Models Yijun Yang, Lichao Wang, Jianping Zhang, Chi Harold Liu, Lanqing Hong, Qiang Xu Abstract The growing misuse of Vision-Language Models (VLMs) has led providers to deploy multiple safeguards—alignment tuning, system prompts, and content moderation. Yet the real-world robustness of these defenses against adversarial attack remains underexplored. We introduce Multi-Faceted Attack (MFA) , a framework that systematically uncovers general safety vulnerabilities in leading defense-equipped VLMs, including GPT-4o, Gemini-Pro, and LlaMA 4, etc . Central to MFA is the Attention-Transfer Attack (ATA), which conceals harmful instructions inside a meta task with competing objectives. We offer a theoretical perspective grounded in reward-hacking to explain why such an attack can succeed. To maximize cross-model transfer, we introduce a lightweight transfer-enhancement algorithm combined with a simple repetition strategy that jointly evades both input- and output-level filters—without any model-specific fine-tuning. We empirically show that adversarial images optimized for one vision encoder transfer broadly to unseen VLMs, indicating that shared visual representations create a cross-model safety vulnerability. Combined, MFA reaches a 58.5% overall success rate, consistently outperforming existing methods. Notably, on state-of-the-art commercial models, MFA achieves a 52.8% success rate, outperforming the second-best attack by 34%. These findings challenge the perceived robustness of current defensive mechanisms, systematically expose general safety loopholes within defense-equipped VLMs, and offer a practical probe for diagnosing and strengthening the safety of VLMs. 1 1 1 Code: https://github.com/cure-lab/MultiFacetedAttack WARNING: This paper may contain offensive content. 1 The Chinese University of Hong Kong, 2 Beijing Institute of Technology, 3 Huawei Noah’s Ark Lab {yjyang, qxu}@cse.cuhk.edu.hk 1 Introduction VLMs represented by GPT-4o and Gemini-pro, have rapidly advanced the frontiers of multimodal AI, enabling impressive capabilities in visual reasoning that jointly process images and language ( OpenAI 2024 ; Google 2024 ) . However, the same capabilities that drive their utility also magnify their potential for misuse, e.g . generating instructions for self-harm, extremist content, and detailed weapon fabrication ( Zhao et al. 2023 ; Qi et al. 2023 ; Gong et al. 2023 ; Yan et al. 2025 ; Huang et al. 2025 ; Teng et al. 2025 ; Yang et al. 2024a ; Yang et al. 2025 ) . To counter these threats, providers have extended beyond traditional alignment training which trains model to refuse harmful requests, by introducing stronger system prompts , steering models to align with safety goals and implementing input- and output-level moderation filters , which ban unsafe content together forming a multilayered defense stack as illustrated in Fig. 1 claimed to deliver “production-grade” robustness ( Meta AI 2023 ; Azure OpenAI 2024 ; Yang et al. 2024b ) . Despite progress, it remains unclear the actual safety margin against real-world adaptive, cross-model attacks remains poorly characterized and potentially overestimated. Figure 1: Overview of the stacked defenses. Meanwhile, research into VLM safety has grown but remains fragmented. One line of work focuses on prompt-based jailbreaks ( Shen et al. 2024 ) , while another explores image-based jailbreaks ( Li et al. 2024 ; Qi et al. 2023 ; Yang et al. 2025 ) ; both typically focus on breaking the endogenous alignment or overriding the system prompt, while ignoring the effect of content filters that guard most deployed systems ( Meta AI 2023 ; Azure OpenAI 2024 ; He et al. 2025 ) . Furthermore, many evaluations are restricted to open-source models, leaving unanswered whether observed vulnerabilities transfer to proprietary systems. In this paper, we introduce Multi-Faceted Attack ( MFA ), a framework that systematically probes defense-equipped VLMs for general safety weaknesses. MFA is powered by the Attention-Transfer Attack (ATA): instead of injecting harmful instructions directly, ATA embeds them inside a benign-looking meta task that competes for attention. We show that the effectiveness of ATA stems from its ability to perform a form of reward hacking —exploiting mismatches between the model’s training objectives and its actual behavior. By theoretically framing ATA as a form of through this lens, we derive formal conditions under which even aligned VLMs can be steered to produce harmful outputs. ATA exploits a fundamental design flaw in current reward models used for alignment training, illuminating previously unexplained safety loopholes in VLMs and we hope this surprising finding opens up new research directions for alignment robustness and multimodal model safety. While ATA is effective, it remains challenging to jailbreak commercial VLMs solely through this approach, as these models are often protected by extra input and output content filters that block harmful content ( Inan et al. 2023 ; Llama Team 2024a ; Llama Team 2024b ; OpenAI 2023b ) , as demonstrated in Fig. 1 (c) and (d). To address this limitation, we propose a novel transfer-based adversarial attack algorithm that exploits the pretrained repetition capability of VLMs to circumvent these content filters. Furthermore, to maximize cross-model transferability and evaluation efficiency, we introduce a lightweight transfer-enhancement attack objective combined with a fast convergence strategy. This enables our approach to jointly evade both input- and output-level filters without requiring model-specific fine-tuning, significantly reducing the overall effort required for successful attacks. To exploit vulnerabilities arising from the vision modality, we develop a novel attack targeting the vision encoder within VLMs. Our approach involves embedding a malicious system prompt directly within an adversarial image. Empirical results demonstrate that adversarial images optimized for a single vision encoder can transfer effectively to a wide range of unseen VLMs, revealing that shared visual representations introduce a significant cross-model safety risk. Strikingly, a single adversarial image can compromise both commercial and open-source VLMs, underscoring the urgency of addressing this pervasive vulnerability. MFA achieves a 58.5% overall attack success rate across 17 open-source and commercial VLMs. This superiority is particularly pronounced against leading commercial models, where MFA reaches a 52.8% success rate—a 34% relative improvement over the second best method. Our main contributions are as follows: • MFA framework. We introduce Multi-Faceted Attack , a framework that systematically uncovers general safety vulnerabilities in leading defense-equipped VLMs. • Theoretical analysis of ATA. We formalize the Attention-Transfer Attack through a reward-hacking lens and derive sufficient conditions under which benign-looking meta tasks dilute safety signals, steering VLMs toward harmful outputs despite alignment safeguards. To the best of our knowledge, this is the first formal theoretical explanation of VLM jailbreaks. • Filter-targeted transfer attack algorithm. We develop a lightweight transfer-enhancement objective coupled with a repetition strategy that jointly evades both input- and output-level content filters. • Vision-encoder–targeted adversarial images. We craft adversarial images that embed malicious system prompts directly in pixel space. Optimized for a single vision encoder, these images transfer broadly to unseen VLMs—empirically revealing a monoculture-style vulnerability rooted in shared visual representations. Taken together, our findings show that today’s safety stacks can be broken layer by layer, and offer the community a practical probe—and a theoretical lens—for diagnosing and ultimately fortifying the next generation of defenses. 2 Related Work Prompt-Based Jailbreaking. Textual jailbreak techniques traditionally rely on prompt engineering to override the safety instructions of the model ( Yu et al. 2023 ) . Gradient-based methods such as GCG ( Zou et al. 2023 ) operate in white-box or gray-box settings without content filters enabled, leaving open questions about transferability to commercial defense-equipped deployments. Vision-Based Adversarial Attacks. Recent studies