Steering to Say No: Configurable Refusal via Activation Steering in Vision Language Models
Jiaxi Yang, Shicheng Liu, Yuchen Yang, Dongwon Lee · 2026
Read Steering to Say No: Configurable Refusal via Activation Steering in Vision Language Models by Jiaxi Yang, Shicheng Liu, Yuchen Yang, Dongwon Lee (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-07013.txt.
Description
Jiaxi Yang and three co-authors make refusal in vision-language models adjustable to the user and the context. Fixed refusal rules miss in both directions: they refuse too little for some needs and too much for others. Their method, CR-VLM, steers activations to tune refusal instead of fixing one rule for all.
Three parts work together. First, a teacher-forced step pulls out a refusal vector that can be tuned, boosting the refusal signal. Second, a gate keeps acceptance for in-scope queries, curbing over-refusal. Third, a vision module built on counterfactuals lines up visual features with refusal needs. Tests on many data sets and models show refusals that are effective, efficient, and robust, pointing toward safety that adapts to the user.
Catalog record
| CREATOR | Jiaxi Yang, Shicheng Liu, Yuchen Yang, Dongwon Lee |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-07013