Robust Vision-Language Models via Tensor Decomposition: A Defense Against Adversarial Attacks

Het Patel, Muzammil Allie, Qian Zhang, Jia Chen, Evangelos E. Papalexakis · 2025

Read Robust Vision-Language Models via Tensor Decomposition: A Defense Against Adversarial Attacks by Het Patel, Muzammil Allie, Qian Zhang, Jia Chen, Evangelos E. Papalexakis (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-16163.txt.

Introduction

Vision-language models can be fooled by tiny image tweaks that humans would never notice, and most fixes demand costly retraining. Het Patel and four co-authors offer a light defense that works on any trained model as-is. Their method splits the model's image features into compact parts and rebuilds them, which washes out attack noise while keeping the meaning.

Tests with a leading model on two photo sets show clear gains in accuracy under attack. A low-rank form of the split worked best, within a narrow range of settings. The defense is cheap to add and needs no change to the model's design.

Catalog record

CREATORHet Patel, Muzammil Allie, Qian Zhang, Jia Chen, Evangelos E. Papalexakis
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE15 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-16163

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Robust Vision-Language Models via Tensor Decomposition: A Defense Against Adversarial Attacks Het Patel, Muzammil Allie, Qian Zhang, Jia Chen, Evangelos E. Papalexakis Abstract Vision language models (VLMs) excel in multimodal understanding but are prone to adversarial attacks. Existing defenses often demand costly retraining or significant architecture changes. We introduce a lightweight defense using tensor decomposition suitable for any pre-trained VLM, requiring no retraining. By decomposing and reconstructing vision encoder representations, it filters adversarial noise while preserving meaning. Experiments with CLIP on COCO and Flickr30K show improved robustness. On Flickr30K, it restores 12.3% performance lost to attacks, raising Recall@1 accuracy from 7.5% to 19.8%. On COCO, it recovers 8.1% performance, improving accuracy from 3.8% to 11.9%. Analysis shows Tensor Train decomposition with low rank (8-32) and low residual strength ( α = 0.1 − 0.2 \alpha=0.1-0.2 ) is optimal. This method is a practical, plug-and-play solution with minimal overhead for existing VLMs. 1 Introduction Vision language models (VLMs) such as CLIP [ 15 ] have revolutionized multimodal understanding by learning joint embeddings of images and text, enabling applications like image retrieval [ 8 ] , zero-shot classification [ 14 ] , and visual question answering [ 21 ] . However, VLMs remain vulnerable to adversarial attacks [ 11 , 19 ] , where imperceptible perturbations dramatically degrade performance. Ensuring robustness is essential as these models deploy in critical applications. Traditional defenses like adversarial training [ 10 ] , input preprocessing [ 20 ] , and architectural modifications [ 13 ] have significant limitations: adversarial training requires expensive retraining on large datasets, while architectural changes demand substantial redevelopment effort. For large pre-trained VLMs, these solutions are often impractical due to computational constraints or unavailable training data. We propose a novel tensor decomposition defense applicable to any pretrained VLM without retraining or modification. Leveraging the fact that adversarial perturbations introduce high-frequency noise in feature representations [ 9 ] , we decompose and reconstruct intermediate representations using low-rank tensor approximations to filter perturbations while preserving semantic content. Our contributions include: (1) a lightweight tensor decomposition defense requiring no retraining, (2) comprehensive parameter analysis, (3) empirical robustness improvements on COCO and Flickr30K, and (4) publicly available code at GitHub Link . 2 Related Work Defensive Tensorization [ 2 ] applies Tucker decomposition to express CNN weight tensors in lower-dimensional latent spaces, introducing tensor dropout to stochastically reduce tensor ranks and prevent adversarial exploitation of fixed-weight structures. Input-level defenses include SmoothVLM [ 16 ] , which uses random pixel perturbations with majority voting(achieving 95% attack reduction), and TensorSHIELD [ 4 ] , which filters adversarial noise via low-rank tensor decompositions. FARE [ 17 ] utilizes unsupervised alignment between perturbed and original images for consistent outputs withouit labeled data. (a) Original (b) Adversarial (c) Perturbation Figure 1: Adversarial attack visualization: original image, adversarial version ( \epsilon=0.0314 ), and scaled perturbation pattern. Double Visual Defense [ 19 ] improves the robustness using a two-stage adversarial training process: extensive pretraining to boost zero-shot capabilities and attack resistance, followed by fine-tuning on downstream tasks with adversarially perturbed inputs. Current approaches have notable limitations: Defensive Tensorization factorizes weight matrices without considering input variations, SmoothVLM requires multiple forward passes per query, TensorSHIELD lacks dynamic adaptation, and FARE and Double Visual Defense demand substantial computational resources and frequent dataset updates against evolving attacks. 3 Methodology We present a defense mechanism against adversarial attacks on a vision-language model utilizing tensor decomposition techniques [ 6 , 1 , 3 ] , including CP/PARAFAC [ 5 ] , Tucker [ 18 ] , and Tensor-Train Decomposition [ 12 ] . Our approach works by applying low-rank tensor decompositions to the internal representation of the CLIP model. 3.1 Problem Formulation We consider a pre-trained VLM that maps images and text to a joint embedding space. For an image-text pair (I,T) , the model produces embeddings f_{I}(I) and f_{T}(T) . During inference, the similarity between these embeddings is computed as: s(I,T)=\langle f_{I}(I),f_{T}(T)\rangle (1) where \langle\cdot,\cdot\rangle denotes the cosine similarity. An adversarial attack aims to find a perturbed image I_{adv} such that: I_{adv}=I+\delta,\text{ where }\|\delta\|_{\infty}\leq\epsilon (2) The perturbation \delta is designed to minimize the similarity between the perturbed image and its corresponding text description: \min_{\delta}s(I+\delta,T)\text{ subject to }\|\delta\|_{\infty}\leq\epsilon (3) Our goal is to defend against such attacks by restoring the correct cross-modal alignment in the presence of adversarial perturbations. 3.2 Adversarial Attack Framework We use a white-box approach known as Projected Gradient Descent (PGD) specifically on CLIP models. Given an image-text (x,t) pairs, the attack minimizes the cosine similarity between their embeddings: L_{adv}(x_{adv},t)=-\frac{f_{I}(x_{adv})\cdot f_{T}(t)}{||f_{I}(x_{adv})||\cdot||f_{T}(t)||} where f_{I} and f_{T} are the image and text encoders. The adversarial example x_{adv} is generated through iterative updates: x_{adv}^{i+1}=\Pi_{\epsilon}(x_{adv}^{i}+\alpha\cdot\text{sign}(\nabla_{x}L_{adv}(x_{adv}^{i},t))) with perturbation budget \epsilon=8/255 , step size \alpha=6/255 , and \Pi_{\epsilon} projected onto the \ell_{\infty} ball around the original image. The \ell_{\infty} -ball constraint ensures that no pixel in the adversarial image differs from the corresponding pixel in the original image by more than \epsilon , maintaining perceptual similarity while allowing effective attacks (Fig. 1 ). 3.3 Tensor Decomposition Defense Our defense mechanism simplifies CLIP vision encoder representations through three components: 1. Forward Hook Mechanism: Intercepts tensors at specific layers (final_norm, attention, MLP outputs). 2. Tensor Decomposition: Applies one of three methods to tensor T : • CP: T\approx\sum_{r=1}^{R}a_{r}\circ b_{r}\circ c_{r} (rank-1 tensor sums) • Tucker: T\approx\mathcal{G}\times_{1}A\times_{2}B\times_{3}C (core tensor + factors) • TT: T[i_{1},\ldots,i_{d}]\approx G_{1}[i_{1}]\cdots G_{d}[i_{d}] (3D core products) 3. Residual Connection: T_{\text{final}}=\alpha\cdot T+(1-\alpha)\cdot\hat{T} , where \alpha controls defense strength. 4 Experimental Setup We used CLIP (ViT-B/32) on MS-COCO and Flickr30K datasets, evaluating 1,000 pairs for parameter analysis and 3,000 for comprehensive evaluation. Attacks used PGD with \ell_{\infty} constraint: \epsilon=8/255 , step size \alpha=6/255 , and 10 steps, creating imperceptible perturbations that significantly degrade performance (Fig. 1 ). Our tensor decomposition defense, implemented with PyTorch and TensorLy [ 7 ] , uses forward hooks with random initialization, 50 iterations, and 10^{-4} tolerance. We evaluated five key factors: 1. Decomposition Method: CP, Tucker, and Tensor Train effectiveness 2. Target Layer: Final norm, attention output, and MLP layers 3. Rank Parameter: Values from 16-256 exploring capacity vs. noise filtering 4. Residual Strength: Weight \alpha (0.1-0.9) balancing original and reconstructed features 5. Multi-layer Configuration: Single layer up to all 12 transformer layers 5 Results and Analysis 5.1 Parameter Sensitivity Analysis 5.1.1 Effect of \alpha Value Figure 2 : Performance comparison across different \alpha values using Tensor Train decomposition (rank=32) on the final normalization layer. Lower \alpha values provide better defense against adversarial attacks. The effectiveness of tensor decomposition is greatly affected by the residual connection strength parameter \alpha , which balances the original and decomposed forms. As seen in 2 , there is an inverse relationship between \alpha values and model robustness in the MS-COCO and Flickr30K datasets. Performance peaks at lower \alpha values (0.1-0.3) and declines as \alpha increases. For Flickr30K, the best Recall@1 is 10.6% at \alpha=0.3 , and for MS-COCO, it is 6.2% at \alpha=0.1 . The downward trend with increasing \alpha supports that adversarial perturbations, as high-frequency components, are effectively filtered by tensor decomposition. 5.1.2 Decomposition Rank Figure 3 : Performance comparison across different rank values using Tensor Train decomposition ( \alpha=0.2 ) on the final normalization layer, lower rank values (8-32) provide greater

Read the complete text