Semantic-aware Adversarial Fine-tuning for CLIP

Jiacheng Zhang [email protected], Jinhao Li [email protected], Hanxun Huang [email protected], Sarah M. Erfani [email protected], Benjamin I.P. Rubinstein [email protected], Feng Liu [email protected] · 2026

Read Semantic-aware Adversarial Fine-tuning for CLIP by Jiacheng Zhang [email protected], Jinhao Li [email protected], Hanxun Huang [email protected], Sarah M. Erfani [email protected], Benjamin I.P. Rubinstein [email protected], Feng Liu [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-12461.txt.

Description

CLIP's zero-shot robustness can be improved by adversarial fine-tuning, but the usual attack examples may be weaker than they look. Jiacheng Zhang led this work with five co-authors. He shows that examples made by minimising cosine similarity to a hand-crafted template, such as a photo of a label, may fail to fool CLIP. That failure appears when similarity is measured with semantically richer alternatives. A fine-tuned encoder trained on those examples is then less robust than hoped.

The fix starts with a semantic-ensemble attack, generating semantic-aware examples by minimising average similarity to an ensemble of refined textual descriptions. Those descriptions come from a foundation model capturing core semantic features beyond templates, then refined to cut hallucinations. Fine-tuning CLIP's image encoder on those examples, as SAFT, beats current methods, with substantial gains in zero-shot adversarial robustness across 16 data sets.

Catalog record

CREATORJiacheng Zhang [email protected], Jinhao Li [email protected], Hanxun Huang [email protected], Sarah M. Erfani [email protected], Benjamin I.P. Rubinstein [email protected], Feng Liu [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-12461

Browse the full library · About the Tower of Babel