Two Birds, One Projection: Harmonizing Safety and Utility in LVLMs via Inference-time Feature Projection

Yewon Han, Yumin Seol, EunGyung Kong, Minsoo Jo, Taesup Kim † · 2026

Read Two Birds, One Projection: Harmonizing Safety and Utility in LVLMs via Inference-time Feature Projection by Yewon Han, Yumin Seol, EunGyung Kong, Minsoo Jo, Taesup Kim † (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-14825.txt.

Description

Han and co-authors defend vision-language models from jailbreaks by projecting out a bias direction. Safety tuning often trades away skill on plain visual tasks. A cross-modal bias direction shows up across datasets here. Weak coupling between the language backbone and visual encoders breeds it. That direction drags down both safety and utility scores. Their method projects features onto the null space of the bias. One forward pass does the whole defense. Safety and utility rise together across benchmarks, they report.

Catalog record

CREATORYewon Han, Yumin Seol, EunGyung Kong, Minsoo Jo, Taesup Kim †
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56264
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.14825

Browse the full library · About the Tower of Babel