Steering to Say No: Configurable Refusal via Activation Steering in Vision Language Models Jiaxi Yang, Shicheng Liu, Yuchen Yang, Dongwon Lee Abstract With the rapid advancement of Vision Language Models (VLMs), refusal mechanisms have become a critical component for ensuring responsible and safe model behavior. However, existing refusal strategies are largely one-size-fits-all and fail to adapt to diverse user needs and contextual constraints, leading to either under-refusal or over-refusal. In this work, we firstly explore the challenges mentioned above and develop C onfigurable R efusal in VLM s ( CR-VLM ), a robust and efficient approach for configurable refusal based on activation steering. CR-VLM consists of three integrated components: (1) extracting a configurable refusal vector via a teacher-forced mechanism to amplify the refusal signal; (2) introducing a gating mechanism that mitigates over-refusal by preserving acceptance for in-scope queries; and (3) designing a counterfactual vision enhancement module that aligns visual representations with refusal requirements. Comprehensive experiments across multiple datasets and various VLMs demonstrate that CR-VLM achieves effective, efficient, and robust configurable refusals, offering a scalable path toward user-adaptive safety alignment in VLMs. † † affiliationnotice: ⋆ Equal contribution. † Work done as an intern at Penn State. Figure 1 : Motivating example of personalized refusal mechanism in LLMs, in which the constraint for configurable refusal is trading bitcoin is legal or not . 1 Introduction With the rapid advancement of VLMs, ensuring their safe and responsible use has become a critical concern ( 44 ; 24 ) . One aspect of this safety framework is the refusal mechanism, which enables VLMs to decline answering queries that may lead to harmful, unethical, or inappropriate outcomes ( 32 ) . This mechanism is essential for preventing misuse and ensuring that VLMs operate within acceptable ethical boundaries ( 19 ; 40 ) . Although substantial progress has been made in designing refusal mechanisms, most current approaches rely on a one-size-fits-all strategy ( 45 ; 17 ) and generally lack the adaptability required to accommodate the diverse needs and contexts of different users. For instance, as shown in Figure 1 , differences in user backgrounds, intentions, and local regulations lead to varying requirements for VLMs refusal mechanisms. This highlights the need for configurable refusal strategies that refuse queries violating the specified constraints while answering those satisfying them. However, only a few studies have investigated configurable refusal mechanisms in Large Language Models (LLMs) ( 16 ; 17 ) , while VLMs remain largely unexplored. In LLMs, ( 17 ) introduces an early benchmark for evaluating controllable refusal, and ( 16 ) explores controllable refusal via hard steering vectors induced from system prompts; however, the resulting refusal signal are weak and the method does not readily transfer to VLMs without additional safety alignment. For VLMs, an intuitive way is system prompt engineering, where user-specific constraints are explicitly stated ( 17 ; 6 ) . However, this approach is highly sensitive to linguistic variations and is easily bypassed ( 38 ; 29 ) , leading to unstable and fragile refusal behavior ( 16 ; 26 ) . Another approach is to fine-tune the model on user-specific constraints, yet such training-based personalization is expensive and difficult to scale across diverse users and evolving multimodal contexts ( 14 ) . This leads us to the question: How can we develop a robust and efficient configurable refusal approach for VLMs that can adapt to diverse user needs? In this work, we present the first systematic study of C onfigurable R efusal in VLM s and propose CR-VLM, a robust and efficient framework based on activation steering. To obtain reliable refusal signals, we adopt teacher-forced activation extraction to obtain strong and low-noise refusal activations, even for VLMs without inherent safety alignment. To mitigate over-refusal, we design a selective configurable mechanism that preserves responses for in-scope queries while maintaining refusal for out-of-scope inputs. Moreover, we incorporate a vision-aware calibration objective to align activation shifts with visual evidence, enabling more accurate refusal behavior. Our contributions are summarized as follows: • To the best of our knowledge, CR-VLM is the first to systematically investigate configurable refusal in VLMs, laying the groundwork for user-adaptive safety alignment. • A novel end-to-end approach is proposed, which is robust and efficient, by leveraging activation steering to achieve configurable refusal. • Extensive experiments conducted on various datasets across multiple VLMs demonstrate the effectiveness and robustness of our approach using various evaluation metrics. 2 Related Work 2.1 Large Vision Language Models The rapid advancements of LLMs ( 23 ; 46 ; 27 ) stimulate great developments in VLMs ( 41 ; 21 ) . By aligning the visual and textual representation spaces ( 30 ) , VLMs demonstrate strong performance in variable tasks, such as multi-model dialogue, visual reasoning, and visual grounding. However, similar to LLMs, VLMs also critically need safety alignment to mitigate potential risks and ensure responsible deployment ( 40 ; 35 ) . Our work focuses on the configurable refusal mechanism in VLMs rather than LLMs. 2.2 Activation Steering Activation steering refers to manipulating the hidden states of a model to control its behaviors ( 31 ; 1 ) . Recent works present their effectiveness across various application domains such as reducing hallucinations ( 47 ) , controlling personas ( 6 ) , and enhancing safety alignment ( 5 ; 8 ) . In this paper, our work shows the feasibility of activation steering to induce refusal behaviors in a configurable manner. 2.3 Refusal in Foundation Models Safety alignment in LLMs aims to prevent models from responsing to content that may harm individuals and society. This includes refusing to generate content that is illegal, harmful, or violates ethical guidelines ( 40 ) . Recently, substantial research has focused on enhancing the refusal capabilities of LLMs via alignment techniques such as supervised fine-tuning ( 4 ) and RLHF ( 9 ) . Additional strategies include prompt engineering ( 47 ; 39 ) , probing the internal states of LLMs ( 36 ; 3 ; 22 ) , and consistency-based approaches ( 42 ) , among others. Meanwhile, other studies have explored over-refusal issues in LLMs ( 37 ; 8 ; 5 ; 7 ) . These methods all lead to generic refusal responses that lack user-specific control. Though several recent works ( 6 ; 34 ) control model behavior via activation steering, these approaches typically utilize global activation steering intervention, which can induce over-refusal even for queries within the intended constraints. Therefore, they remain orthogonal to our goal of constraint-aware configurable refusal. The limited existing studies ( 17 ) for configurable refusal firstly propose a benchmark in LLMs. And the most relevant work named CAST ( 16 ) , which achieves controllable refusal in LLMs by applying a hard steering vector extracted through a refusal-inducing system prompt. However, this method only focus on LLMs and heavily depends on the inherent safety alignment of LLMs. When directly applied to VLMs that lack safety alignment, the refusal signal becomes too weak to extract a reliable activation direction. Moreover, their approach relies on hard intervention by steering vector and is not an end-to-end solution, limiting its applicability. Figure 2 : Framework of CR-VLM, a configurable refusal approach for VLMs. 3 Notations This section clarifies the core notations that are used throughout the remainder of the paper. Consider a LVLM \mathcal{M}(\cdot) that is feeded with x=(I,T) including an image I and text query T to generate reponses R=\{r_{1},r_{2},...,r_{n}\} , which denotes as R=\mathcal{M}(x) . As the inputs pass through the model’s transformer architecture, it generates a series of intermediate activations denoted as h(x)_{i}^{l} at position i in layer l . For in-constraint inputs x_{\text{in}}\in D_{\text{in}} , we denote their activations as h_{\text{in}} , and for out-of-constraint inputs x_{\text{out}}\in D_{\text{out}} , we denote their activations as h_{\text{out}} . 4 Methodology The overview pipeline of our proposed configurable refusal approach is presented in Figure 2 . The core idea is to train a lightweight model that predicts calibrated activations from the original activations of the VLMs. To capture stronger refusal signals, we utilize teacher-forced activation extraction in subsection 4.1 . Loss design for inducing refusal behavior is presented in subsection 4.2 . The over-refusal mitigation mechanism for in-scope samples is introduced in subsection 4.3 . We further incorporate counterfactual vision enhancement, which encourages the model to attend to visual information during the refusal process, as described in subsection 4.4 . 4.1 Teacher-Forced Activation Extraction Prior works extract behavioral directions by contrasting activations obtained with and without a system prompt ( 34 ; 6 ; 17 ; 16 ) . However, directly applying this strategy to configurable refusal scenarios introduces reliability issues. In particular, for VLMs that lack strong safety alignment, system prompts specifying refusal constraints often fail to trigger an explicit refusal, resulting in weak activation shifts. For example, given an out-of-scope query such as “How can I trade bitcoin to make money?” , a model still attempts to provide an answer even when preceded by a system prompt: “Bitcoin trade questions are not allowed. Refuse to answer.” In such cases, the activation difference between runs with and without the system prompt primarily reflects prompt-induced semantic variation rather than refusal behavior. To address this limitation, we adopt teacher-forced activation extraction ( 31 ; 5 ) . Here, the “teacher” is not a separate model but a predefined refusal response appended to each out-of-scope sample. Concretely, for an out-of-scope input x_{\text{out}}=(I,T) , we form the sequence (x_{\text{out}},r_{\text{ref}}) , where r_{\text{ref}} is a fixed refusal string ( e.g., "Sorry," ). For the same bitcoin-trading query, the model is thus fed the user question followed by the refusal response, forcing it to process the refusal during the forward pass. This guarantees that the activation difference corresponds to the computation of refusal behavior rather than an uncertain prompt-induced semantic variation. We denote the resulting refusal activation representation as h(x_{\text{out}},r_{\text{ref}})_{i}^{l} . As a reference non-refusal activation representation, we use h(x_{\text{out}})_{i}^{l} obtained from the original input alone. Since individual activations are noisy, we aggregate them across the dataset to obtain stable mean representations. Specifically, we compute the dataset-level mean activations: \displaystyle\mathbf{v}_{i,l}^{acc}=\frac{1}{|D|}\sum_{x_{\text{out}}\in D}h_{i}^{l}(x_{\text{out}}), (1) \displaystyle\mathbf{v}_{i,l}^{ref}=\frac{1}{|D|}\sum_{x_{\text{out}}\in D}h_{i}^{l}(x_{\text{out}},r_{\text{ref}}), where \mathbf{v}_{i,l}^{ref} and \mathbf{v}_{i,l}^{acc} are the mean refusal and mean non-refusal activation representations, respectively. To obtain a clean and directional refusal representation, we then compute a steering vector via difference-in-means ( 2 ) : \mathbf{s}_{i}^{l}=\mathbf{v}_{i,l}^{ref}-\mathbf{v}_{i,l}^{acc}, (2) Then, we can obtain the target refusal activations of out-of-scope samples by h_{\text{out},i}^{l,target}\leftarrow h_{\text{out},i}^{l}+\lambda\cdot\mathbf{s}^{l}_{j}, (3) where j is the last token position of predefined refusal response. Table 1 : Refusal rates (%) for in-scope and out-of-scope queries without (w/o) and with (w) applying the over-refusal mitigation mechanism, as well as under an ablated version of the mitigation for out-of-constraint queries. Increase (%) Original w/o w In-scope 0.0 97.0 \uparrow 97.0 0.5 \downarrow 96.5 Out-of-scope 0.0 95.0 \uparrow 95.0 95.0 \approx 4.2 Refusal Behavior Induction In our approach, we employ a lightweight model named calibrated model , to predict the calibrated activations. While teacher-forced extraction provides reliable refusal activations offline, these signals are not directly available during inference. The calibrated model serves as a bridge that translates this offline supervision into input-conditioned activations at runtime, enabling refusal control without modifying the backbone VLMs. And these calibrated hidden states are then used to intervene in specific layers in the forward pass, thereby inducing the desired refusal behavior. To ensure that the predicted calibrated activations of out-of-scope samples by our trained model h^{\prime}_{\text{out}} should be as similar as possible to the ground-truth activations of out-of-scope samples h_{\text{out}}^{target} . We align both the direction and the magnitude of h^{\prime}_{\text{out}} with h_{\text{out}}^{\text{target}} and design the loss function as follows: \mathcal{L}_{\text{out}}=\mathcal{L}_{\text{out, dir}}+\mathcal{L}_{\text{out, len}}. (4) For direction alignment, we utilize cosine similarity as follows: \mathcal{L}_{\text{out, dir}}=(1-cos(h^{\prime}_{\text{out}},h_{\text{out}}^{target})). (5) For the magnitude alignment, we leverage MSE loss: \mathcal{L}_{\text{out, len}}=MSE(h^{\prime}_{\text{out}},h_{\text{out}}^{target}). (6) 4.3 Mitigation of Over-Refusal Simply optimizing the loss in Equation ( 4 ) does not fully capture the contraint-aware refusal behavior we seek. As illustrated in Table 1 , this kind of global calibration may lead to excessive adjustment of in-scope queries, shifting the representation space of the model uniformly toward the refusal direction. Consequently, not only out-of-scope queries but also those within scope are pushed toward refusal, leading to undesirable over-refusal ( 8 ; 5 ) . Motivated by this observation, we also design the gate loss function for queries within scope. The objective is to preserve the activations of in-scope samples, preventing them from being altered by the learned refusal direction. We thus incorporate an MSE reconstruction term that encourages h^{\prime}_{\text{in}} to remain close to h_{\text{in}} . In addition, we employ a gating mechanism and regularize its probability \rho toward zero for in-constraint queries. We further include a gate supervision term g , which softly regularizes the gating behavior to follow the desired activation pattern under different scope conditions. The gate loss function for mitigating over-refusal is formulated as follows: \mathcal{L}_{\text{in}}=MSE(h^{\prime}_{\text{in}},h_{\text{in}})+\rho+g (7) To prevent the calibrated model from collapsing into a single global update direction and to encourage scope-conditional separation in the learned intervention vectors, we introduce an orthogonality regularizer between the average update directions induced by out-of-scope and in-scope samples. Let \Delta(h)\in\mathbb{R}^{d} denote the predicted update direction ( i.e., \Delta=h^{\prime}-h ) for an input activation h . For each mini-batch, samples are partitioned into two subsets \mathcal{B}_{out} and \mathcal{B}_{in} . We compute the mean update direction for each subset as: \Delta_{\text{out}}=\frac{1}{|\mathcal{B}_{\text{out}}|}\sum_{i\in\mathcal{B}_{\text{out}}}\Delta(h_{i}),\quad\Delta_{\text{in}}=\frac{1}{|\mathcal{B}_{\text{in}}|}\sum_{i\in\mathcal{B}_{\text{in}}}\Delta(h_{i}). (8) The orthogonality loss is then defined as the squared cosine similarity between these two mean directions: \mathcal{L}_{\text{ortho}}=\left(\frac{\Delta_{\text{out}}^{\top}\Delta_{\text{in}}}{\|\Delta_{\text{out}}\|_{2}\,\|\Delta_{\text{in}}\|_{2}}\right)^{2}. (9) By minimizing \mathcal{L}_{\text{ortho}} , the model is encouraged to learn the decoupled intervention directions for out-of-scope refusal behavior and in-scope response preservation, thereby reducing the risk of over-refusal. 4.4 Counterfactual Vision Enhancement Although steering the textual activations at later token positions provides a more globally informed refusal signal, this intervention can dominate the multimodal representation and suppress visual contributions, causing the model to underutilize visual cues. As a result, the vision modality contributes only weakly to the overall refusal behavior. This makes existing LLM-side steering mechanisms ineffective in cases where the refusal decision should be driven by visual evidence rather than textual semantics. To achieve this, we require a reference that characterizes how visual representations should causally change when a refusal is warranted based solely on visual input. We therefore introduce a vision-only activation shift as a counterfactual reference, which isolates the contribution of the vision modality by excluding textual influence. Concretely, for each image I_{i} , we obtain the pure visual refusal activation shift by contrasting its activations under out-of-constraint images and in-constraint images: \mathbf{r}_{i}=h(I_{i,\text{out}},r_{\text{ref}})-h(I_{i,\text{in}}), (10) which reflects how the visual activations should change when the constraint requires a refusal. During training, our model predicts calibrated activations h^{\prime}_{\text{in}} and h^{\prime}_{\text{out}} . Accordingly, we obtain the predicted activation shift: \mathbf{r}^{\prime}_{i}=h^{\prime}(I_{i,\text{out}},T_{i,\text{out}},r_{\text{ref}})-h^{\prime}(I_{i,\text{in}},I_{i,\text{out}},) (11) Our objective is to regularize the predicted activation shift \mathbf{r}^{\prime}_{i} to align with the vision-derived counterfactual shift \mathbf{r}_{i} , thereby ensuring that the refusal behavior learned by the model remains consistent with how visual representations should causally change when a refusal is required. Thus, we can have the vision loss as follows: \mathcal{L}_{\text{vision}}=1-cos(\sum\mathbf{r}^{\prime}_{i},\sum\mathbf{r}_{i}). (12) This counterfactual vision enhancement forces the model to internalize how visual representations should shift when a refusal is required. Consequently, VLMs can correctly modulate refusal behaviors based on visual evidence. Then the whole definition of the loss function can be illustrated to combine Equation ( 4 ), ( 7 ), and ( 12 ) as follows: \displaystyle\mathcal{L}=\lambda_{1}\cdot\mathcal{L}_{\text{out}}+\lambda_{2}\cdot\mathcal{L}_{\text{in}}+\lambda_{3}\cdot\mathcal{L}_{\text{orth}}+\lambda_{4}\cdot\mathcal{L}_{\text{vision}} (13) Table 2 : (EQ1) Refusal rate (%) for out-of-scope test samples. Dataset Subjects Methods Human Evaluation LLM-as-Judgement LLaVA-1.5-7b LLaVA-1.5-13b Idefics3 InstructBLIP LLaVA-1.5-7b LLaVA-1.5-13b Idefics3 InstructBLIP ScienceQA Biology Prompt-based 0.0 0.0 2.5 1.5 1.5 0.0 3.0 3.0 Fine-tuning 54.0 25.5 31.0 90.5 54.0 26.0 34.0 90.0 Persona 0.0 0.0 0.0 2.5 0.0 0.5 0.5 3.0 CR-VLM 95.0 94.0 94.5 94.5 95.5 94.0 93.5 96.0 Physics Prompt-based 0.0 0.0 1.0 0.0 0.0 0.0 1.0 2.0 Fine-tuning 9.0 43.0 7.5 97.5 9.0 43.0 8.0 98.0 Persona 0.0 8.0 0.0 0.0 0.0 10.5 0.0 1.5 CR-VLM 99.5 92.0 83.5 98.5 99.5 92.5 84.5 97.5 Geography Prompt-based 0.0 0.0 2.5 0.0 2.0 0.0 3.5 1.0 Fine-tuning 64.0 87.5 11.0 100.0 64.0 87.5 14.5 100.0 Persona 0.0 0.0 0.0 8.5 0.0 0.5 2.5 8.0 CR-VLM 93.0 89.5 92.0 94.0 96.0 90.0 90.5 95.0 MMMU Math Prompt-based 18.0 4.0 2.0 6.5 19.0 5.5 8.0 10.0 Fine-tuning 23.5 26.0 8.0 88.5 25.0 27.0 11.0 88.5 Persona 31.0 27.0 2.5 24.5 32.0 27.0 5.5 21.5 CR-VLM 97.0 92.0 82.5 83.0 97.5 93.5 88.5 81.0 Art Theory Prompt-based 28.5 7.5 10.5 7.5 29.5 9.0 15.0 11.5 Fine-tuning 92.0 93.5 90.5 98.0 93.5 93.5 91.5 98.0 Persona 33.5 31.5 7.0 26.0 34.0 33.5 13.0 26.0 CR-VLM 97.5 96.5 94.5 85.0 98.0 96.5 95.5 78.5 Geography Prompt-based 21.0 4.0 5.0 15.0 23.5 5.5 5.5 15.0 Fine-tuning 32.5 37.0 8.5 72.5 33.5 38.0 10.0 73.0 Persona 33.0 28.0 5.0 28.5 35.0 30.5 9.5 29.5 CR-VLM 93.0 91.5 79.5 85.0 92.0 89.0 81.0 81.0 N umbers in green shows the best-performing results compared to baselines. U nderlined values denote the second best. Table 3 : (EQ2) Over-refusal rate (%) for in-scope test samples. CR-VLM exhibits a very low over-refusal rate compared to other three baselines. Dataset Subjects Methods Human Evaluation LLM-as-Judgement LLaVA-1.5-7b LLaVA-1.5-13b Idefics3 InstructBLIP LLaVA-1.5-7b LLaVA-1.5-13b Idefics3 InstructBLIP ScienceQA Biology System Prompt 0.0 0.0 1.5 0.0 0.0 0.0 1.5 0.0 Fine-tuning 0.0 0.0 0.0 3.0 0.0 0.5 0.5 5.5 Persona 0.0 7.5 0.0 1.0 0.0 9.0 3.0 2.0 CR-VLM 0.5 0.0 0.0 0.5 1.0 0.5 1.0 0.5 Physics System Prompt 0.0 0.0 0.0 0.0 1.5 0.5 1.0 0.0 Fine-tuning 0.0 0.0 0.0 2.5 1.5 0.0 1.5 2.5 Persona 0.0 15.0 1.5 0.0 0.0 15.5 2.5 0.0 CR-VLM 0.0 0.0 0.0 0.0 1.0 0.0 2.5 1.0 Geography System Prompt 0.0 0.0 0.0 0.5 0.0 0.0 0.0 1.0 Fine-tuning 0.0 0.0 0.0 2.0 0.0 0.0 0.0 2.0 Persona 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 CR-VLM 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.5 MMMU Math Prompt-based 35.0 1.5 1.5 9.0 37.0 3.0 4.5 11.5 Fine-tuning 2.0 0.5 0.0 24.0 3.5 1.5 3.0 24.0 Persona 34.5 24.5 9.0 21.5 35.5 22.0 13.0 21.0 CR-VLM 9.5 7.0 6.0 10.5 12.0 7.5 11.5 12.5 Art Theory Prompt-based 4.5 1.5 2.0 6.0 5.0 1.5 2.5 9.5 Fine-tuning 0.0 0.0 0.5 46.5 0.0 0.0 0.5 46.5 Persona 27.5 11.5 7.0 22.0 27.0 11.0 2.0 21.0 CR-VLM 1.5 0.0 0.0 2.5 2.0 2.5 1.5 3.5 Geography Prompt-based 32.0 10.5 3.0 5.0 32.5 11.5 5.5 8.5 Fine-tuning 1.0 0.0 4.0 33.0 1.5 2.5 7.0 32.5 Persona 44.0 47.5 8.5 30.0 45.0 50.0 6.0 27.5 CR-VLM 4.0 4.0 5.0 6.0 6.5 5.0 9.5 9.5 5 Experiments In this experiment section, we conduct experiments to answer the following E valuation Q uestions (EQ): • EQ1: To what extent does our proposed approach achieve alignment with the intended refusal scope under configurable settings? • EQ2: Does steering refusal on out-of-scope queries come at the cost of inducing over-refusal on in-scope queries? And how well does our approach balance this trade-off? • EQ3: How does each component of our approach contribute to configurable refusal alignment and over-refusal mitigation? 5.1 Experiment Settings Models. We conduct our experiments on several widely used open-source VLMs, including LLaVA-1.5-7B-hf ( 21 ) , LLaVA-1.5-13B-hf ( 21 ) , Idefics3-8B-Llama3 ( 15 ) , and InstructBLIP-Vicuna-7B ( 10 ) . These models represent diverse architectures and size-scales, allowing us to assess the universality of our configurable refusal approach. Datasets. To the best of our knowledge, we are the first to study configurable refusal in VLMs, an area where no suitable benchmarks currently exist. Following the idea of ( 17 ) in LLMs, which designates certain subjects in MMLU ( 12 ) as in-scope samples and others as out-of-scope samples, we evaluate our method on two multimodal datasets: ScienceQA ( 25 ) and MMMU ( 43 ) . For each dataset, we randomly sample 200 in-scope and 200 out-of-scope samples for training, and the same number for testing. To ensure sufficient data coverage, we use three subjects in ScienceQA, including biology, physics, and geography, as our in-scope dataset. Similarly, we select several in-scope subdomains in MMMU ( e.g., math, art theory, and geography) as our in-scope dataset. Baselines. We compare our proposed method with four representative refusal steering baselines: 1) Prompt-based Steering ( 11 ) , which injects handcrafted configurable refusal constraints into the system prompt; 2) Fine-Tuned Refusal Adapter (FRA) uses parameter-efficient updates ( e.g., LoRA ( 13 ) ) to adjust refusal behavior; 3) Persona ( 6 ) , which utilizes activation vectors extracted via a refusal-oriented system prompt to control model behavior. In our experiments, we adopt this method, originally designed for LLMs, and extend it to LVLM scenarios by applying the same prompt-based activation extraction procedure to multimodal inputs. 5.2 (EQ1) configurable Refusal Alignment In this section, we evaluate whether our proposed method enables VLMs to refuse queries that fall outside the scope. Evaluation Metrics. To quantify how well the model refuses to answer out-of-scope questions, we adopt refusal rate as the primary evaluation metric, defined as the proportion of responses that contain explicit refusal semantics. A higher refusal rate indicates better performance. To verify the correctness and robustness of measuring our results, we employ both human evaluation and LLM-as-Judgment ( 18 ) , using the high-performance model DeepSeek-V3 ( 20 ) to automatically identify refusal semantics following the template in Appendix G . Figure 3 : (EQ2) MB-Score trade-off between refusal rate on out-of-scope queries and over-refusal rate on in-scope queries. (a) LLaVA-1.5-7B-hf (b) LLaVA-1.5-13B-hf (c) Idefics3 (d) InstructBLIP Figure 4 : (EQ2) Activation distribution shifts at the 25 th layer for in-scope and out-of-scope test samples across different subjects in ScienceQA and MMMU on LLaVA-1.5-7B-hf. Result Analysis. As shown in Table 2 , our method achieves higher refusal rates on out-of-scope samples than baselines in most cases across both datasets in different subjects. This demonstrates that the calibrated activations through our method are substantially more effective for inducing configurable refusal behavior in various VLMs. The baseline of Persona consistently underperforms across models and subjects, yielding near-zero refusal rates in some settings. This is because injecting refusal behavior solely through system prompts relies heavily on the built-in safety alignment of the model. When such alignment is weak, the induced refusal signal is insufficient, as failing to extract strong refusal-related activations, leading to ineffective refusal behavior steering. While fine-tuning methods are able to induce refusal behavior to some extent, our approach outperforms fine-tuning in 41 out of 48 experimental settings, with fine-tuning performing better in only a small fraction of cases. This suggests that the approach becomes unreliable when only limited 200 fine-tuning data samples are available, as in our experimental setting. Besides, prompt-based steering performs extremely poorly, often yielding near-zero refusal rates, indicating that simple prompt-based manipulation is insufficient to induce reliable configurable refusal in VLMs. 5.3 (EQ2) Over-Refusal Mitigation Here, we evaluate whether our proposed method will not lead to over-refusal for in-scope queries while maintaining the high refusal rate for out-of-scope queries. Over-Refusal Rate on In-scope Queries. Evaluation Metrics. Similar to the evaluation protocol in EQ1, we use refusal rate to quantify over-refusal ( i.e., the proportion of in-scope queries that are false refused). In contrast to EQ1, a lower refusal rate here refers to better performance in this setting. Similarly, we evaluate this using both human evaluation and LLM-as-Judgment as well. Result Analysis. As shown in Table 3 , our method exhibits consistently low over-refusal rates across all models and datasets within various subject domains, indicating that it effectively preserves acceptance for in-scope queries while maintaining strong refusal behavior on out-of-scope queries. In contrast, Persona in baseline methods performs poorly in some subjects, displaying significant over-refusal, which frequently rejects in-scope queries. These results demonstrate that our calibrated activation approach achieves a more reliable configurable refusal behavior. Trade-off between Refusal and Over-Refusal. Evaluation Metrics. To explicitly capture the trade-off between effective Refusal Rate (RR) and Over-Refusal Rate (ORR), we utilize MB-Score ( 33 ) as shown in equation ( 14 ), a composite metric that jointly considers refusal performance on out-of-scope queries and acceptance behavior on in-scope queries. \text{MB-Score}=\frac{2\cdot\text{RR}\cdot(1-\text{ORR})}{\text{RR}+(1-\text{ORR})}.\vskip-5.0pt (14) Result Analysis. Figure 3 summarizes the trade-off using MB-Score, which jointly reflects refusal effectiveness and over-refusal. Across all subjects of both datasets, our method consistently achieves higher MB-Scores than baselines, indicating the best performance for configurable refusal. In contrast, fine-tuning and Persona methods suffer from degraded MB-Scores due to excessive over-refusal for in-scope queries or ineffective refusal steering for out-of-scope queries. These results demonstrate that our calibrated activation approach enables configurable refusal, selectively amplifying refusal only when constraints are violated, rather than globally increasing refusal strength. Activation Distribution. Evaluation Metrics. The goal of our configurable refusal mechanism is to reshape the model’s hidden representations to reflect the desired refusal behavior. To assess whether the calibrated activations induced by CR-VLM yield clearer separation between in-scope and out-of-scope queries compared to the original representations, we conduct a qualitative analysis of hidden-state distributions. Specifically, we apply Principal Component Analysis (PCA) ( 28 ) to both the original and calibrated activations of in-scope and out-of-scope queries, and visualize their projected representations in a low-dimensional space. Result Analysis. As shown in Figure 4 , the calibrated activations for in-scope queries remain close to the original in-scope representations, indicating that our method largely preserves the acceptance behavior for in-scope queries. In contrast, out-of-scope samples present a substantial shift in the representation space: their calibrated activations are pushed away from both the original out-of-scope and in-scope region and form a compact cluster associated with refusal behavior. We further observe similar distribution patterns across more various models in the Appendix C . 5.4 (EQ3) Ablation Studies In this section, we conduct ablation studies on LLaVA-1.5-7B-hf to analyze the contribution of individual components in CR-VLM. Efficacy of Teacher-Forced Activation Extraction. To evaluate whether teacher-forced activation extraction yields stronger refusal signals, we compare it with prompt-based activation extraction across different datasets and subjects. As shown in Figure 5 , teacher-forced activation extraction consistently achieves higher refusal rates on out-of-scope samples, verifying its effectiveness. Figure 5 : (EQ3) Ablation study of teacher-forced method for activation extraction compared with prompt-based method. Over-Refusal Mitigation by Orthogonality. We conduct an ablation study by comparing designing a loss function with or without the orthogonality regularization term \mathcal{L}_{\text{ortho}} . The results reported in Table 4 demonstrate that severe over-refusal on in-scope queries across all subjects without the orthogonality loss term. This indicates that without explicit decoupling, the learned intervention directions for in-scope and out-of-scope samples collapse into a shared global shift, leading to over-refusal. In contrast, incorporating the orthogonality constraint effectively suppresses over-refusal while preserving strong refusal behavior for out-of-scope inputs, validating its critical role in enabling configurable refusal. Table 4 : (EQ3) Ablation results of the loss function with (w) and without (w/o) the orthogonality term across different datasets, conducted on LLaVA-1.5-7B-hf for both in-scope and out-of-scope test samples. Dataset Subject In-scope (%) Out-of-scope (%) w w/o w w/o ScienceQA Biology 0.0 97.0 \uparrow 97.0 95.0 95.0 \approx Physics 0.0 89.5 \uparrow 89.5 99.5 98.0 \downarrow 1.5 Geography 0.0 100.0 \uparrow 100 93.0 93.0 \approx MMMU Math 9.5 80.5 \uparrow 71 97.0 98.5 \uparrow 1.5 Art Theory 1.5 91.0 \uparrow 89.5 97.5 97.0 \downarrow 0.5 Geography 4.0 96.0 \uparrow 92 93.0 97.5 \uparrow 4.5 Efficacy of Vision Loss. To examine whether vision loss encourages the model to pay more attention to visual cues during refusal, we conduct an ablation study by inputting the image with blank textual content. This setting focuses on the contribution of visual information to the predicted refusal behavior by excluding textual influence. Following ( 1 ; 37 ) , we adopt the refusal score as evluation metrics here and the calculation details of the refusal score are introduced in the appendix D . As shown in Figure 6 , removing vision loss leads to a notable drop in refusal scores across different subjects. These results on ScienceQA dataset indicate that vision loss effectively enhances visual grounding in the refusal process, rather than relying solely on text-dominated signals. The additional results on MMMU are shown in the appendix F . Figure 6 : (EQ3) Ablation study of vision loss conducted on LLaVA-1.5-7B-hf on ScienceQA. 6 Conclusion In this paper, we firstly introduce CR-VLM, a configurable refusal approach for VLMs based on activation steering, enabling models to refuse out-of-scope queries while preserving acceptance for in-constraint ones. Through teacher-forced extraction, gate mechanism, and vision-aware calibration, our method achieves robust refusal alignment across different models and datasets, with low over-refusal. Impact Statement This paper presents work whose goal is to achieve configurable refusal for VLMs. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here. Appendix A Case Study Figure 7 and Figure 8 provide qualitative examples illustrating the behavior of our configurable refusal mechanism for out-of-scope queries and in-scope queries respectively. The model correctly refuses out-of-scope queries while preserving normal responses for in-scope inputs, demonstrating configurable refusal rather than global refusal enforcement. (a) Example 1 (b) Example 2 Figure 7 : Case Study for Out-of-scope Test Samples. (a) Example 1 (b) Example 2 Figure 8 : Case Study for In-scope Test Samples. Figure 9 : (EQ2) MB-Score trade-off between refusal rate and over-refusal rate by LLM-as-Judgment. Figure 10 : (EQ2) Activation distribution shifts at the 25 th layer for in-scope and out-of-scope test samples across different subjects in ScienceQA and MMMU on LLaVA-1.5-13B-hf. Appendix B (EQ2) Additional Experiments by MB-Score To complement the human evaluation results reported in the main paper, we further analyze MB-Score using an LLM-as-Judgment by DeepSeek-V3 ( 20 ) as shown in Figure 9 . Consistent with human evaluation, our method consistently achieves higher MB-Scores than baselines across different models and subjects, demonstrating a more balanced trade-off between refusal effectiveness and over-refusal. Figure 11 : (EQ2) Activation distribution shifts at the 25 th layer for in-scope and out-of-scope test samples across different subjects in ScienceQA and MMMU on Idefics3-8B-Llama3. Figure 12 : (EQ2) Activation distribution shifts at the 25 th layer for in-scope and out-of-scope test samples across different subjects in ScienceQA and MMMU on InstructBLIP-Vicuna-7B. Appendix C (EQ2) Additional Results of Activation Distribution We report the activation distribution shifts for additional VLMs in Figure 10 , Figure 11 , and Figure 12 , to further validate the generality of our configurable refusal method. Specifically, we visualize the calibrated and original activations for LLaVA-1.5-13B-hf, Idefics3-8B-Llama3, and InstructBLIP-Vicuna-7B across different subject domains. Consistent with the observations in the main paper, calibrated activations for out-of-scope queries are clearly separated from both their original counterparts and in-scope activations, forming compact clusters associated with refusal behavior. Meanwhile, activations corresponding to in-scope queries remain close to their original distributions, indicating that acceptance behavior is largely preserved. These results demonstrate that the activation-level separation induced by our method is stable across model architectures and scales, further confirming the effectiveness and robustness of our configurable refusal mechanism. Appendix D Refusal Score Calculation We calculate a refusal score followed by ( 1 ; 37 ) in Eq. ( 15 ), which quantifies the model’s inclination to generate refusal-related tokens. Formally, at the first decoding step, we compute the log-odds between refusal-related tokens \mathcal{R} ( e.g., “Sorry”, “I”) and all other tokens \mathcal{V}\setminus\mathcal{R} as refusal score. A higher refusal score demonstrates a stronger tendency to refuse. The equation for calculating the refusal score is shown as follows: \mathcal{T}(x)=\log\Bigg(\sum_{t\in\mathcal{R}}p_{t}\Bigg)-\log\Bigg(\sum_{t\in\mathcal{V}\setminus\mathcal{R}}p_{t}\Bigg). (15) Appendix E Refusal Score Shift To obtain a more fine-grained understanding of how our method modulates refusal behavior, we examine how our proposed method shifts the underlying tendency of the model to generate refusal-related tokens. The details of calculating refusal score is introduced in section D in the appendix. Unlike the refusal rate, which provides a binary view of whether a refusal occurs, the refusal score captures subtle changes in the refusal tendency of the model. The results on ScienceQA are shown in Figure 13 , while the corresponding results on MMMU are presented in Figure 14 . Across all evaluated VLMs and subject domains, out-of-scope test samples consistently exhibit substantially higher refusal scores than in-scope samples. Importantly, in-scope samples remain concentrated in low refusal-score ranges, indicating that our method selectively amplifies refusal behavior for out-of-scope inputs without introducing over-refusal. This consistent pattern across datasets further validates the effectiveness and robustness of CR-VLM. Figure 13 : Refusal score shift for in-scope and out-of-scope test samples on ScienceQA. Figure 14 : Refusal score shift for in-scope and out-of-scope test samples on MMMU. Appendix F Additional Ablation Study of Vision Loss Figure 15 : (EQ3) Ablation Study of Vision Loss conducted on LLaVA-1.5-7B-hf on MMMU. To further verify the effectiveness of the vision loss on visual grounding in refusal behavior, we conduct a similar ablation study on the MMMU dataset. Consistent with the main results reported on ScienceQA in the main content, we evaluate refusal behavior using the refusal score while providing the image input with blank textual content to isolate visual contributions by excluding the visual influence. As shown in Figure 15 , removing the vision loss consistently leads to lower refusal scores across different MMMU subjects. These results suggest that vision loss promotes visual attention of the model when making a refusal behavior. In addition, we provide a complementary analysis from a representation-level perspective beyond the results shown in section 5.4 . Our goal is to test whether the predicted refusal-related representations preserve visual evidence, rather than being only dominated by textual cues. Concretely, we compute the cosine similarity between the predicted representation produced from multimodal inputs ( i.e., both vision and text) and a vision-conditioned reference representation ( i.e., vision-only), and report the averaged similarity across different subjects. A higher similarity score indicates a stronger influence of visual cues on the predicted representations. The results represented in Table 5 demonstrate that the vision loss acts as an effective regularizer, prompting greater reliance on visual cues rather than text-dominated decision patterns. Table 5 : Hidden representation alignment with (w) and without (w/o) vision loss. Dataset ScienceQA MMMU Biology Physics Geography Math Art Theory Geography w/o -0.021 0.070 0.200 0.203 0.106 0.068 w 0.019 0.107 0.224 0.252 0.165 0.082 Example prompt used for LLM-as-Judgment . Principles: Identify whether the response expresses refusal to answer the user’s question. Refusal includes direct rejection, statements of inability ( e.g., ‘I cannot see’, ‘I don’t have access’), or citing safety/policy restrictions. Regard responses starting with a rejection as refusal as well. If refusal semantics are present, return 1; otherwise return 0. Only output the number 0 or 1. Figure 16 : The prompts for generating LLM-as-Judgment with principles and examples Appendix G LLM-as-Judgment For the LLM-as-Judgment framework used to compute the refusal rate and over-refusal rate in this paper, we adopt a strong and popular model, DeepSeek-V3 ( 20 ) , as the evaluator model. The prompt template employed for this evaluation is provided in Figure 16 . Appendix H Architecture of Calibrated Model The architecture of our calibration module for generating activation adjustments used in tunable refusal for VLMs is shown in Table 6 . Table 6 : The architecture of the calibrated model. Input Hidden state x\in\mathbb{R}^{d} Flow Network Linear( \text{Input}\_\text{dim} , 1024) SiLU \rightarrow Linear(1024, 1024) SiLU \rightarrow Linear(1024, \text{Output}\_\text{dim} ) Gate Network Linear, Sigmoid Residual Update \Delta x Output x_{\text{out}}=x+\alpha\,p\,\Delta x Trainable Parameter Scalar \alpha Appendix I Additional experiment Settings We optimize the loss defined in Equation 13 , where all loss weights are set to 1 across experiments. We use a batch size of 8, a learning rate of 1\times 10^{-4} , and train for 200 epochs with the Adam optimizer. Activation steering is applied to different layer ranges depending on the backbone: layers 12–32 for LLaVA-1.5-7B-hf, layers 12–40 for LLaVA-1.5-13B-hf, layers 18–32 for InstructBLIP-Vicuna-7B, and layers 20–32 for Idefics3-8B-Llama3. All hyperparameters are selected via preliminary validation and kept fixed across datasets and models. All experiments use a fixed random seed of 42 and identical decoding settings. Appendix J Current Limitations and Future Directions While our framework demonstrates strong effectiveness in controlling personalized refusal behaviors, it is subject to several main limitations. First, our method relies on open-source models with accessible hidden states to extract representation activations and compute steering vectors. This restricts its direct applicability to closed-source models ( e.g., GPT-4), where internal activations are not exposed. Second, our current experiments focus on one constraint at a time to keep the analysis clear and controlled. This is a design choice instead of a limitation of the framework. In practice, multiple constraints can be incorporated together through composition, and a systematic study of such multi-constraint settings is left for future work. Third, our current evaluation focuses on domain-level configurable refusal, as standardized benchmarks for explicit policy-conditioned scenarios ( e.g., regulatory constraints) are still lacking. This is mainly a benchmarking constraint rather than a limitation of the framework. Extending CR-VLM to such settings is straightforward, and we plan to explore this direction once suitable benchmarks become available.