DAC-LoRA: Dynamic Adversarial Curriculum for Efficient and Robust Few-Shot Adaptation

Ved Umrajkar · 2025

Read DAC-LoRA: Dynamic Adversarial Curriculum for Efficient and Robust Few-Shot Adaptation by Ved Umrajkar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-20792.txt.

Introduction

Ved Umrajkar tackles a safety weakness in cheap fine-tuning methods for vision-language models in a 2025 workshop paper. The paper was accepted at an ICCV 2025 workshop on safe and trustworthy multimodal AI systems. The proposed method is called DAC-LoRA, short for dynamic adversarial curriculum LoRA.

Methods like LoRA adapt big models efficiently by training only a small set of extra weights. Those adapted models still break under adversarial attacks, which matters for uses like driving and medical diagnosis. DAC-LoRA folds adversarial training into the fine-tuning process. Attacks grow harder in a planned sequence, like a curriculum. The author argues the curriculum idea is general and could strengthen few-shot adaptation wherever parameter-efficient tuning is used.

Catalog record

CREATORVed Umrajkar
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-20792

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DAC-LoRA: Dynamic Adversarial Curriculum for Efficient and Robust Few-Shot Adaptation Ved Umrajkar Abstract Vision-Language Models (VLMs) are foundational to critical applications like autonomous driving, medical diagnosis, and content moderation. While Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA enable their efficient adaptation to specialized tasks, these models remain vulnerable to adversarial attacks that can compromise safety-critical decisions. CLIP, the backbone for numerous downstream VLMs, is a high-value target whose vulnerabilities can cascade across the multimodal AI ecosystem.We propose Dynamic Adversarial Curriculum (DAC-LoRA) , a novel framework that integrates adversarial training into PEFT. The core principle of our method—an intelligent curriculum of progressively challenging attacks—is general and can potentially be applied to any iterative attack method. Guided by the First-Order Stationary Condition (FOSC) and a TRADES-inspired loss, DAC-LoRA achieves substantial improvements in adversarial robustness without significantly compromising clean accuracy. Our work presents an effective, lightweight, and broadly applicable method to demonstrate that the DAC-LoRA framework can be easily integrated into a standard PEFT pipeline to significantly enhance robustness. 1 Introduction Vision-Language Models (VLMs) have revolutionized computer vision by learning rich, transferable representations from vast amounts of image-text data [ 14 , 9 , 11 ] . Their ability to perform zero-shot classification on unseen tasks has made them a cornerstone of modern AI [ 19 ] . However, to achieve state-of-the-art performance on specialized, downstream tasks, these powerful models require adaptation. Full fine-tuning of such massive models is often computationally prohibitive. This has spurred the development of Parameter-Efficient Fine-Tuning (PEFT) techniques [ 2 , 10 , 17 , 21 , 15 , 7 ] , some of which adapt a model by training only a small fraction of its parameters [ 7 , 17 , 4 ] . Among these, Low-Rank Adaptation (LoRA) has emerged as a particularly effective and efficient method [ 7 ] . Recent work shows that applying LoRA to CLIP (CLIP-LoRA) is a powerful strategy for few-shot adaptation, achieving impressive accuracy on new tasks with minimal training and no task-specific hyperparameter tuning [ 19 ] . Despite their accuracy on clean data, a critical and often overlooked aspect of fine-tuned VLMs is their vulnerability to adversarial attacks [ 6 , 12 ] . These attacks use small, human-imperceptible perturbations to cause the model to elicit incorrect classifications with high confidence. Crucially, standard fine-tuning on clean examples does not inherently confer robustness [ 8 ] , leaving these powerful models fragile and unreliable in security-sensitive applications. To address this gap, our work integrates robust adversarial training directly into the parameter-efficient LoRA framework. Instead of fine-tuning only on clean data or with fixed-strength attacks, our core contribution is the use of a dynamic curriculum of adversarial examples . Inspired by findings that a gradually increasing attack difficulty is more effective [ 18 ] , we use the First-Order Stationary Condition (FOSC) to intelligently control the attack potency throughout training. This allows the model to first build a foundation of robustness against simpler attacks before being exposed to more challenging ones. The result is DAC-LoRA, a method that produces an accurate, efficient, and robustly adapted model. Figure 1 : Adversarial examples ( \mathbf{x}_{\text{adv}} ) are generated from clean images ( \mathbf{x}_{\text{clean}} ) using a PGD attack, with its potency dynamically controlled by the FOSC criterion. A TRADES-inspired loss, computed from the LoRA-adapted image and text encoders, is used to update only the low-rank matrices ( \mathbf{A},\mathbf{B} ), while the original CLIP weights remain frozen. 2 Preliminaries Our approach integrates parameter-efficient fine-tuning with a novel adversarial training strategy. This section reviews the four foundational concepts upon which our work is built: Low-Rank Adaptation (LoRA), standard adversarial attack generation, the TRADES framework for robust optimization, and the FOSC criterion for measuring attack potency. 2.1 Few-Shot Adaptation with LoRA To adapt large pre-trained models without incurring the prohibitive cost of full fine-tuning, we use the Parameter-Efficient Fine-Tuning (PEFT) technique known as Low-Rank Adaptation (LoRA) [ 7 ] . LoRA injects small, trainable low-rank matrices into the existing layers of a model. For a given weight matrix \mathbf{W} , its update is approximated by two smaller matrices, \mathbf{A} and \mathbf{B} , such that the forward pass becomes h=\mathbf{W}\mathbf{x}+\gamma\mathbf{B}\mathbf{A}\mathbf{x} . This is highly effective for the few-shot adaptation of CLIP [ 14 , 19 ] , as only the LoRA matrices ( \mathbf{A},\mathbf{B} ) are trained, leaving the vast majority of the original model parameters frozen. 2.2 Adversarial Attack Methods Adversarial attacks seek to generate a small, imperceptible perturbation \boldsymbol{\delta} that, when added to a clean image \mathbf{x}^{0} , maximizes the model’s loss. We use the following standard attack method to generate adversarial examples for our training curriculum. Projected Gradient Descent (PGD) is a powerful, iterative attack widely used as a benchmark for evaluating model robustness [ 12 ] . It takes multiple small steps of size \alpha in the direction of the gradient sign. After each step, it projects the result back into the \epsilon -ball around the original image to ensure the perturbation remains within the allowed magnitude: \mathbf{x}^{k+1}=\text{Proj}_{\|\mathbf{x}-\mathbf{x}^{0}\|_{\infty}\leq\epsilon}\left(\mathbf{x}^{k}+\alpha\cdot\text{sign}(\nabla_{\mathbf{x}}\mathcal{L}(h_{\boldsymbol{\theta}}(\mathbf{x}^{k}),y))\right) (1) 2.3 Adversarial Robustness with TRADES The goal of adversarial training is to solve a min-max optimization problem that balances performance on clean data with robustness against attacks. The TRADES framework provides a principled method for this by decomposing the objective into a clean loss and a robustness-regularization term [ 20 ] . The original TRADES loss is: \mathcal{L}_{\text{TRADES}}=\mathcal{L}(h(\mathbf{x}^{0}),y)+\beta\cdot\text{KL}(h(\mathbf{x}^{0})||h(\mathbf{x}_{\text{adv}})) (2) where KL is the Kullback-Leibler divergence. Our work uses a TRADES-inspired variant where the KL-divergence term is replaced with a cosine similarity loss between the feature embeddings of the clean and adversarial images. This serves a similar purpose by regularizing the model’s feature space, encouraging it to produce consistent representations even when the input is perturbed. 2.4 First-Order Stationary Condition (FOSC) To implement a dynamic curriculum, we need a way to measure the strength of an adversarial example. The First-Order Stationary Condition (FOSC) provides a principled criterion to quantify how well the inner maximization problem of an attack has been solved [ 18 ] . For an adversarial example \mathbf{x}^{k} , the FOSC score quantifies the potential to further increase the model’s loss. The closed-form solution is: c(\mathbf{x}^{k})=\epsilon\|\nabla_{\mathbf{x}}\mathcal{L}(h(\mathbf{x}^{k}),y)\|_{1}-\langle\mathbf{x}^{k}-\mathbf{x}^{0},\nabla_{\mathbf{x}}\mathcal{L}(h(\mathbf{x}^{k}),y)\rangle (3) We provide an intuitive derivation and further details on FOSC in the Supplementary Material (see Section 6 ). A lower FOSC score indicates a stronger , better-converged adversarial example [ 18 ] . This criterion is the cornerstone of our dynamic curriculum, enabling us to precisely control attack difficulty during training. Figure 2 : Training dynamics of our DAC-LoRA framework on the UCF101 dataset, illustrating the principles of our dynamic curriculum. Unlike a fixed-strength PGD attack, our FOSC-PGD method dynamically adjusts attack potency based on a decaying threshold, as detailed mathematically on the left. The middle plot visualizes this curriculum, showing how the required attack strength (green line) increases during training. The result, shown on the right, is a significant improvement in adversarial robustness (light blue line). 3 Methodology: Adversarial LoRA with a Dynamic Curriculum Our method synergizes the efficiency of LoRA with a robust training objective by fine-tuning the LoRA parameters on a dynamic curriculum of adversarial examples. The overall workflow is illustrated in Figure 1 . 3.1 Robust Fine-Tuning Objective We adapt the standard adversarial

Read the complete text