Semantic-aware Adversarial Fine-tuning for CLIP
Jiacheng Zhang [email protected], Jinhao Li [email protected], Hanxun Huang [email protected], Sarah M. Erfani [email protected], Benjamin I.P. Rubinstein [email protected], Feng Liu [email protected] · 2026
Read Semantic-aware Adversarial Fine-tuning for CLIP by Jiacheng Zhang [email protected], Jinhao Li [email protected], Hanxun Huang [email protected], Sarah M. Erfani [email protected], Benjamin I.P. Rubinstein [email protected], Feng Liu [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-12461.txt.
Description
CLIP's zero-shot robustness can be improved by adversarial fine-tuning, but the usual attack examples may be weaker than they look. Jiacheng Zhang led this work with five co-authors. He shows that examples made by minimising cosine similarity to a hand-crafted template, such as a photo of a label, may fail to fool CLIP. That failure appears when similarity is measured with semantically richer alternatives. A fine-tuned encoder trained on those examples is then less robust than hoped.
The fix starts with a semantic-ensemble attack, generating semantic-aware examples by minimising average similarity to an ensemble of refined textual descriptions. Those descriptions come from a foundation model capturing core semantic features beyond templates, then refined to cut hallucinations. Fine-tuning CLIP's image encoder on those examples, as SAFT, beats current methods, with substantial gains in zero-shot adversarial robustness across 16 data sets.
Catalog record
| CREATOR | Jiacheng Zhang [email protected], Jinhao Li [email protected], Hanxun Huang [email protected], Sarah M. Erfani [email protected], Benjamin I.P. Rubinstein [email protected], Feng Liu [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-12461