Transferable Adversarial Attacks on Black-Box Vision-Language Models

Kai Hu, Weichen Yu, Li Zhang, Alexander Robey · 2025

Read Transferable Adversarial Attacks on Black-Box Vision-Language Models by Kai Hu, Weichen Yu, Li Zhang, Alexander Robey (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-01050.txt.

Introduction

Kai Hu and co-authors show that crafted image tweaks can fool closed vision-language models into attacker-chosen readings, in a paper from two thousand twenty-five. Such models read both pictures and text for users. Attacks were known to transfer between open models, but their reach into paid black-box systems was untested for this model class.

Tweaks built on open models swayed top commercial systems, including the best-known chat models. Harmful scenes could be read as safe, and restricted content could slip past notice. One universal tweak worked across many images and models. Tests spanned object naming, questions, and captions. Stronger defenses are urgent, the authors warn.

Catalog record

CREATORKai Hu, Weichen Yu, Li Zhang, Alexander Robey
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-01050

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Transferable Adversarial Attacks on Black-Box Vision-Language Models Kai Hu, Weichen Yu, Li Zhang, Alexander Robey Abstract Vision Large Language Models (VLLMs) are increasingly deployed to offer advanced capabilities on inputs comprising both text and images. While prior research has shown that adversarial attacks can transfer from open-source to proprietary black-box models in text-only and vision-only contexts, the extent and effectiveness of such vulnerabilities remain underexplored for VLLMs. We present a comprehensive analysis demonstrating that targeted adversarial examples are highly transferable to widely-used proprietary VLLMs such as GPT-4o, Claude, and Gemini. We show that attackers can craft perturbations to induce specific attacker-chosen interpretations of visual information, such as misinterpreting hazardous content as safe, overlooking sensitive or restricted material, or generating detailed incorrect responses aligned with the attacker’s intent. Furthermore, we discover that universal perturbations—modifications applicable to a wide set of images—can consistently induce these misinterpretations across multiple proprietary VLLMs. Our experimental results on object recognition, visual question answering, and image captioning show that this vulnerability is common across current state-of-the-art models, and underscore an urgent need for robust mitigations to ensure the safe and secure deployment of VLLMs. Kai Hu † † heading: 1 2025 1- † † shortheadings: Blackbox VLLM Attack / Blackbox VLLM Attack † † firstpage: 1 † † editor: Kai Hu keywords Adversarial attack, vision large language model, black-box attack 1 Introduction The quickly advancing capabilities of foundation models has driven exciting new progress across fields as diverse as robotics ( Ma et al., 2023a ; Brohan et al., 2023 ) , healthcare ( Singhal et al., 2023 ; D’Antonoli et al., 2024 ) , and software development ( Yang et al., 2024 ) . Central to this progress is the use of internet-scale data corpora during training, which enables highly performant models capable of processing text (e.g., the GPT, Claude, or Gemini families ( Achiam et al., 2023 ; Anthropic, 2024 ; Team et al., 2023b ) ) as well as visual inputs (e.g., ResNet and DenseNet architectures ( He et al., 2016 ; Huang et al., 2017 ) ). And while these datasets contain valuable information, training on this raw data also exposes models to vast amounts of objectionable content. Visual input example 1: misclassify an image of a fly as a Siamese cat. Input image: Source: NIPS 2017 Adversarial Learning Challenges ( Kurakin et al., 2018 ) . User Provide a concise description of the image using no more than three sentences. GPT-4o The image features a Siamese cat with a dark face and ears, set against a background of vibrant green hues. The composition appears layered, with another faint cat figure… Claude 3.5 Sonnet This appears to be an artistic portrait of a Siamese cat with striking blue eyes, rendered in a painterly style with predominant green tones in the background… Gemini 1.5 Pro The image depicts two cats against a vibrant green background with intricate patterns. The cat in the foreground has striking blue eyes and a pointed face, while the cat in the… Visual input example 2: attack a receipt image to misrecognize the text Input image: Source: Retail Receipt Dataset TrainingDataPro (2025) . User How many items were sold? GPT-4o The receipt shows that 4 items were sold. Claude 3.5 Sonnet According to the receipt shown in the image, 4 items were sold Table 1 : Visual examples of the adversarial attack results on proprietary LLMs. More can be found in Table 8 , 9 , 10 , 11 of the appendix. Given the tendency for foundation models to output the content they see during training ( Nasr et al., 2023 ) , both text- and image-based models are fine-tuned to refuse to respond to queries requesting objectionable responses Hacker et al. (2023) ; Ouyang et al. (2022) . This process—known as model alignment—involves using human feedback to “align” generated responses with human values. As recent work has shown that this type of alignment is often vulnerable to adversarial attacks ( Zou et al., 2023 ; Chao et al., 2024 ) , the security risks surrounding these models has grown significantly. Toward meeting the growing needs of real-world applications, vision-enabled large language models (VLLMs), which process both visual and textual inputs, have become remarkably proficient at a wide range of tasks like visual question-answering, reasoning, and zero-shot classification ( Liu et al., 2024a ; Ma et al., 2023b ) . Compared with single-modality models, the space of possible attacks on VLLMs is significantly larger: beyond the fact that attackers can potentially manipulate both inputs ( Dong et al., 2023 ; Qi et al., 2023 ) , the safe deployment of a VLLM for many tasks—e.g., autonomous vehicle stacks or military robotics, among many others—requires robust processing and interpretation of visual information ( Eykholt et al., 2018 ; Julian et al., 2020 ) . However, relatively little is known about the vulnerabilities introduced by multimodality, particularly in the realistic setting wherein the attacker has only black-box access to the multimodal model. To assess the risks of VLLMs deployed in safety-critical settings, in this paper, we develop a novel attack for VLLMs designed to find image perturbations by targeting adversarially chosen text embeddings. By using an ensemble of open-source models during the attack process, we enhance the transferability of these adversarial examples to proprietary blackbox VLLMs. We further adapt our attack objective to achieve universality by creating perturbations that generalize across different images and models. While our attack is based on the same principles as prior work on image-only and text-only models, we emphasize that the choice of attack objective for multimodal transfer accounts for the significant improvements in transfer success over recent methods. We conduct extensive experiments to evaluate the effectiveness of our attack across various tasks, including object recognition, image captioning, and visual question-answering. Through ablation studies, we identify the factors that most significantly contribute to multimodal transferability, such as the impact of model ensembling and the specifics of the attack objective. Our results demonstrate higher transfer rates than previously reported (an early work ( Dong et al., 2023 ) achieves 45% untargeted attack successful rate on GPT-4V while our method archives over 95% targeted attack successful rate on GPT-4o), underscoring the severity of the vulnerabilities introduced by multimodality. 2 Related Work Adversarial Attacks on VLLMs The vulnerability of machine learning models to adversarial examples is well-documented, with early studies focusing primarily on image-based classifiers ( Szegedy et al., 2014 ; Liu et al., 2016 ; Biggio et al., 2013 ; Cohen et al., 2019 ) . This research has since been extended to evaluate the robustness of language models against adversarial attacks ( Zou et al., 2023 ; Wei et al., 2024b ; Wei et al., 2024a ; Liu et al., 2024b ; Shin et al., 2020 ; Chao et al., 2024 ; Perez et al., 2022 ) . And despite progress toward designing effective defenses against these attacks ( Zou et al., 2024 ; Jain et al., 2023 ; Mazeika et al., 2024 ; Robey et al., 2023 ) , adaptive and multi-turn attacks are still known to bypass the alignment of these models ( Li et al., 2024b ; Russinovich et al., 2024 ; Andriushchenko et al.,

Read the complete text