Navigating the Trade-off: A Synthesis of Defensive Strategies for Zero-Shot Adversarial Robustness in Vision-Language Models

Zane Xu, Jason Sun · 2025

Read Navigating the Trade-off: A Synthesis of Defensive Strategies for Zero-Shot Adversarial Robustness in Vision-Language Models by Zane Xu, Jason Sun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-05237.txt.

Introduction

Zane Xu and Jason Sun review ways to defend vision-language models against attacks that use subtly altered images. Such models can name and describe things they were never trained on, a skill called zero-shot use. Small crafted changes to an image can break that skill, and defenses tend to cost some of it.

The review sorts defenses into two camps. One retrains the model, with steps that range from gentle tuning to reshaping its inner space. The other leaves the model alone and cleans inputs or inner states at test time. Eight key papers anchor the survey. The authors close with open problems, among them mixed defenses and training that bakes in resistance from the start.

Catalog record

CREATORZane Xu, Jason Sun
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-05237

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Navigating the Trade-off: A Synthesis of Defensive Strategies for Zero-Shot Adversarial Robustness in Vision-Language Models Zane Xu, Jason Sun Abstract This report synthesizes eight seminal papers on the zero-shot adversarial robustness of vision-language models (VLMs) like CLIP. A central challenge in this domain is the inherent trade-off between enhancing adversarial robustness and preserving the model’s zero-shot generalization capabilities. We analyze two primary defense paradigms that have emerged from recent research. The first, Adversarial Fine-Tuning (AFT), involves modifying model parameters. We trace its evolution from early methods focused on protecting vision-language alignment (TeCoA) and mitigating overfitting through pre-trained model guidance (PMG-AFT, TGA-ZSR), to more advanced strategies that proactively re-engineer the embedding space’s geometry to build intrinsic robustness (LAAT, TIMA). The second paradigm, Training-Free and Test-Time Defenses, avoids parameter modification altogether. We examine its progression from simple, heuristic-based input manipulations (AOM, TTC) to theoretically grounded purification methods in the model’s latent space (CLIPure). By analyzing these works, we distill the field’s core problems—including the robustness-generalization dilemma, geometric vulnerabilities, and computational costs—and identify key insights. We conclude by outlining promising future directions, such as the development of hybrid defense models and the pursuit of large-scale adversarial pre-training to create natively robust foundation models. Introduction The Promise and Peril of Foundation Models In recent years, the field of artificial intelligence has witnessed a paradigm shift led by large-scale foundation models. Among these, Vision-Language Models (VLMs) such as CLIP (Contrastive Language-Image Pre-training) ( Radford et al. 2021 ) have been particularly noteworthy ( Mao et al. 2023 ; Wang et al. 2024 ; Li et al. 2024 ) . By performing contrastive learning on hundreds of millions of image-text pairs, models like CLIP have acquired unprecedented zero-shot generalization capabilities. They can achieve high-accuracy image classification on unseen downstream tasks and datasets using only simple text descriptions (e.g., ”a photo of a dog”), significantly broadening the application boundaries of computer vision ( Mao et al. 2023 ; Tong et al. 2024 ) . However, this powerful generalization capability conceals a major security vulnerability: extreme susceptibility to adversarial perturbations. Numerous studies have shown that adding tiny, human-imperceptible noise to an input image can cause the classification performance of these advanced models to drop precipitously, or even fail completely ( Mao et al. 2023 ; Ma et al. 2024 ) . For instance, in the CLIP model, a minute vector with a perturbation magnitude of no more than 1/255 under the l_{\infty} norm is sufficient to cause a subversive error in its prediction ( Mao et al. 2023 ) . This vulnerability is not merely an academic issue; it poses a severe challenge to practical applications in safety-critical domains such as autonomous driving, medical diagnosis, and content moderation. Therefore, building robust defense mechanisms for these powerful foundation models has become an urgent research imperative. Defining the Core Challenge: The Robustness-Generalization Dilemma The concept of ”zero-shot adversarial robustness” has emerged to address this need. Its core objective is to enhance a model’s ability to withstand adversarial attacks while maximally preserving its invaluable zero-shot generalization capability. This goal is far more challenging than adversarial defense in traditional supervised learning scenarios because it introduces a fundamental dilemma: the inherent conflict between robustness and generalization. Research has found that directly applying traditional adversarial defense methods, particularly Adversarial Training (AT), to foundation models like CLIP often leads to catastrophic consequences. For example, in a typical scenario, researchers conducted standard adversarial fine-tuning on the CLIP model using the ImageNet dataset. The results showed that while the model’s adversarial robustness on the ImageNet validation set improved, its classification accuracy on 15 other unseen zero-shot datasets plummeted, with its generalization ability almost completely destroyed ( Mao et al. 2023 ; Wang et al. 2024 ) . This phenomenon is known as catastrophic overfitting, indicating that the robust features learned through standard AT are highly coupled with the specific training data distribution, thereby corrupting the transferable knowledge gained from large-scale pre-training. How to resolve this ”robustness-generalization” trade-off is the central challenge that the eight papers synthesized in this report collectively aim to address. A Taxonomy of Defense Paradigms To systematically organize and understand the various strategies for tackling this challenge, this report categorizes these eight pioneering works into two major defense paradigms. This classification is based on whether they require modification of the model’s parameters to enhance robustness and at which stage of the model’s lifecycle the defense mechanism operates. • Paradigm I: Adversarial Fine-Tuning (AFT) : The core idea of these methods is to ”inject” robustness into the model itself by updating its parameters (i.e., fine-tuning) on a dataset containing adversarial examples. This process typically involves meticulously designed loss functions and regularization terms in the hope of learning robust features while avoiding catastrophic forgetting of pre-trained knowledge. The methods TeCoA ( Mao et al. 2023 ) , PMG-AFT ( Wang et al. 2024 ) , LAAT ( Li et al. 2024 ) , TIMA ( Ma et al. 2024 ) , and TGA-ZSR ( Yu, Zhang, and Xu 2024 ) analyzed in this report belong to this paradigm. • Paradigm II: Training-Free and Test-Time Defenses : These methods take a different path, completely avoiding any modification of the model’s parameters. They intervene during the model’s inference (test) stage by processing the input data or its internal representations to ”purify” or ”correct” the effects of adversarial perturbations. The advantage of this approach lies in its flexibility and low setup cost, as it bypasses the resource-intensive process of adversarial training. The methods AOM ( Tong et al. 2024 ) , TTC ( Xing, Zhao, and Sebe 2025 ) , and CLIPure ( Zhang et al. 2025 ) analyzed herein are outstanding representatives of this paradigm. The following sections will provide an in-depth analysis of the methods within these two paradigms and synthesize their findings to illuminate the core problems and future directions of the field. Paradigm I: Adversarial Fine-Tuning for Zero-Shot Robustness Adversarial Fine-Tuning (AFT) is a mainstream approach for enhancing model robustness. However, in the zero-shot setting, the challenge lies in guiding the model to learn generalizable robustness rather than merely memorizing attack patterns from a specific dataset. This section delves into five advanced AFT methods, revealing a progressive deepening of thought in solving the robustness-generalization dilemma. The Foundation: Preserving Vision-Language Alignment with TeCoA TeCoA ( Mao et al. 2023 ) is one of the pioneering works in this field. It was the first to keenly observe that the reason standard adversarial training destroys CLIP’s zero-shot capability is that it severs the core principle established during pre-training: vision-language alignment. Traditional adversarial training uses one-hot labels and cross-entropy loss, which decouples the optimization objective of the visual encoder from the text encoder, leading to the collapse of the joint embedding space. Methodology To address this issue, TeCoA proposed a novel training objective: the Text-guided Contrastive Adversarial (TeCoA) training loss . Its core innovation is to shift the goal of adversarial training from ”correctly classifying adversarial examples” to ”maintaining correct vision-language alignment under adversarial perturbation.” Specifically, this loss function is a cross-modal contrastive loss that aims to minimize the feature distance between an adversarial image embedding and its corresponding correct text embedding , while maximizing its distance from incorrect

Read the complete text