Internal Activation Revision: Safeguarding Vision Language Models Without Parameter Update
Qing Li, Jiahui Geng, Zongxiong Chen, Kun Song, Lei Ma, Fakhri Karray · 2025
Read Internal Activation Revision: Safeguarding Vision Language Models Without Parameter Update by Qing Li, Jiahui Geng, Zongxiong Chen, Kun Song, Lei Ma, Fakhri Karray (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-16378.txt.
Description
Qing Li and five co-authors present a safety fix for vision-language models that works by editing their inner activations mid-answer, in this 2025 arXiv paper. Adding images to a language model weakens its safety training: pictures shift the model's internal state, and even simple jailbreaks then succeed. Their method nudges those activations back toward safe territory while the model writes.
The steering acts at two scales, whole layers and single attention heads, using vectors drawn from safe and unsafe example pairs. Across four safety test sets, attack success fell by roughly a third to a half, while helpfulness barely moved. The authors stress that no model weights were retrained to gain the improvement.
Catalog record
| CREATOR | Qing Li, Jiahui Geng, Zongxiong Chen, Kun Song, Lei Ma, Fakhri Karray |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-16378