Two Birds, One Projection: Harmonizing Safety and Utility in LVLMs via Inference-time Feature Projection
Yewon Han, Yumin Seol, EunGyung Kong, Minsoo Jo, Taesup Kim † · 2026
Read Two Birds, One Projection: Harmonizing Safety and Utility in LVLMs via Inference-time Feature Projection by Yewon Han, Yumin Seol, EunGyung Kong, Minsoo Jo, Taesup Kim † (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-14825.txt.
Description
Han and co-authors defend vision-language models from jailbreaks by projecting out a bias direction. Safety tuning often trades away skill on plain visual tasks. A cross-modal bias direction shows up across datasets here. Weak coupling between the language backbone and visual encoders breeds it. That direction drags down both safety and utility scores. Their method projects features onto the null space of the bias. One forward pass does the whole defense. Safety and utility rise together across benchmarks, they report.
Catalog record
| CREATOR | Yewon Han, Yumin Seol, EunGyung Kong, Minsoo Jo, Taesup Kim † |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56264 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.14825