Safeguarding Vision-Language Models: Mitigating Vulnerabilities to Gaussian Noise in Perturbation-based Attacks

Jiawei Wang 1,* Yushen Zuo 2,* Yuanjun Chai 3 Zhendong Liu 4 Yicheng Fu 5 Yichun Feng 6,† Kin-Man Lam 2,† and 1 University of Science and Technology of China, 2 The Hong Kong Polytechnic University, 3 University of Washington, 4 Nanjing University, 5 Stanford University, 6 University of Chinese Acad · 2025

Read Safeguarding Vision-Language Models: Mitigating Vulnerabilities to Gaussian Noise in Perturbation-based Attacks by Jiawei Wang 1,* Yushen Zuo 2,* Yuanjun Chai 3 Zhendong Liu 4 Yicheng Fu 5 Yichun Feng 6,† Kin-Man Lam 2,† and 1 University of Science and Technology of China, 2 The Hong Kong Polytechnic University, 3 University of Washington, 4 Nanjing University, 5 Stanford University, 6 University of Chinese Acad (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-01308.txt.

Description

Jiawei Wang and co-authors study a blind spot in the safety of vision-language models, in a 2025 paper. These models read images and text together. Safety training guards them against jailbreak attacks. But noisy images were overlooked: even simple static noise can break many models' guardrails.

The team built a safety data set pairing aligned and misaligned image-text items. Fine-tuning on it, with noise added, cut attack success while keeping the models useful. For stronger attacks that optimize the image itself, they add a diffusion step that cleans the input first. The work shows that robustness to plain noise is a basic safety need, not a luxury, for models that see the world.

Catalog record

CREATORJiawei Wang 1,* Yushen Zuo 2,* Yuanjun Chai 3 Zhendong Liu 4 Yicheng Fu 5 Yichun Feng 6,† Kin-Man Lam 2,† and 1 University of Science and Technology of China, 2 The Hong Kong Polytechnic University, 3 University of Washington, 4 Nanjing University, 5 Stanford University, 6 University of Chinese Acad
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE81 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-01308

Browse the full library · About the Tower of Babel