Steering Safely or Off a Cliff? Rethinking Specificity and Robustness in Inference-Time Interventions Navita Goyal, Hal Daumé III Abstract Model steering, which involves intervening on hidden representations at inference time, has emerged as a lightweight alternative to finetuning for precisely controlling large language models. While steering efficacy has been widely studied, evaluations of whether interventions alter only the intended property remain limited, especially with respect to unintended changes in behaviors related to the target property. We call this notion specificity. We propose a framework that distinguishes three dimensions of specificity: general (preserving fluency and unrelated abilities), control (preserving related control properties), and robustness (preserving control properties under distribution shifts). We study two safety-critical use cases: steering models to reduce overrefusal and faithfulness hallucinations, and show that while steering achieves high efficacy and largely maintains general and control specificity, it consistently fails to preserve robustness specificity. In the case of overrefusal steering, for example, all steering methods reduce overrefusal without harming general abilities and refusal on harmful queries; however, they substantially increase vulnerability to jailbreaks. Our work provides the first systematic evaluation of specificity in model steering, showing that standard efficacy and specificity checks are insufficient, because without robustness evaluation, steering methods may appear reliable even when they compromise model safety. 1 Introduction Figure 1: Refusal and overrefusal behavior in LLMs ( top left ). Existing evaluations predominantly focus on efficacy ( bottom left )—does steering reduce overrefusal. We extend evaluation of steering methods to assess specificity across three dimensions ( right )—does steering preserve unrelated abilities (general), preserve refusal on harmful queries (control), and remain safe under adversarial prompts (robustness)? Model steering, also known as inference-time interventions, has been proposed as a way of controlling the behavior of large language models (LLMs) 20 ; 44 . By manipulating hidden representations during inference, steering aims to adjust LLM generations in fine-grained and targeted ways, without the need for retraining 31 ; 40 . In principle, two dimensions define the success of these methods: efficacy—whether the intervention alters the target property—and specificity—whether it only alters the target property. While efficacy has been widely studied, specificity remains under-examined. Existing studies suggest that steering methods generally preserve fluency 40 ; 6 as well as general knowledge and mathematical abilities 31 ; 2 . However, these evaluations capture only a limited notion of specificity. In practice, steering operates in entangled representation spaces, where interventions may affect behaviors closely related to the target property, even when unrelated abilities appear intact 44 . Figure 1 illustrates this distinction in the case of overrefusal steering—interventions designed to reduce excessive refusal in safety-aligned LLMs. These models are trained to refuse harmful requests (e.g., how to make a bomb), but often go too far by rejecting benign queries misconstrued as unsafe (e.g., how to create a realistic looking prop knife for a play). This behavior, known as overrefusal 10 ; 1 , limits model usability. Steering methods could, in principle, help mitigate this problem by precisely reducing overrefusal while retaining refusal behavior on harmful queries, thereby improving utility without compromising safety. An efficacy-focused evaluation would conclude success if the model indeed becomes more compliant on benign queries ( Figure 1 ; bottom left). Yet this view ignores the broader question of specificity ( Figure 1 ; right): does the model continue to refuse harmful queries, and does this behavior hold under adversarial settings such as jailbreak attacks? A steering method may thus appear effective while undermining safety, highlighting why evaluating steering only through efficacy is inadequate in practical, safety-critical applications. To better assess utility and reliability of inference-time steering methods, we propose an evaluation framework for steering formalizing three aspects of specificity: general capabilities (does steering preserve the LLM’s fluency and performance on unrelated tasks?), control (does steering preserve behavior or capabilities on properties related to the target property?), and robustness (does steering preserve control properties under distribution shifts, such as adversarial attacks?). Together, these dimensions provide a more complete picture of steering reliability and precision. We investigate specificity in two safety-critical settings: overrefusal steering and faithfulness hallucination steering. In the former, we study the problem of reducing overrefusal of benign queries in safety-aligned models. In the latter, we examine faithfulness hallucinations in question answering when in-context information contradicts a model’s internal knowledge ( 16 ) ; see example in Figure 2 . Both settings are practically important, as reducing overrefusal and hallucinations can improve usability and faithfulness, and safety-critical, as careless steering may increase compliance with malicious requests or produce unreliable answers. By situating our analyses here, we can test not only whether steering improves the target property, but also whether it preserves related behaviors and withstands adversarial challenges. We apply our proposed evaluation framework to a wide range of existing steering techniques, including difference-in-means 20 ; 38 ; 31 , linear probe 44 , supervised steering vector 41 , representation finetuning 41 , and partial orthogonalization 39 on instruction-tuned LLMs with up to 8B parameters. We compare two settings: unconstrained steering, which does not explicitly control for refusal on harmful queries 40 , and constrained steering, which does 39 . Our results reveal that steering effectively reduces overrefusal in LLMs both in- and out-of-distribution. Furthermore, steering for overrefusal is largely able to preserve models’ general capabilities and even model safety on harmful queries. Importantly though, even when model refusal behavior is explicitly controlled for during steering, steering methods consistently and significantly increase models’ susceptibility to jailbreaking, showing that specificity gains are not robust. Similarly, steering effectively reduces faithfulness hallucinations, improving models’ abilities to adapt to updated contextual information while preserving fluency, benchmark performance and accurate reliance on internal knowledge in absence of context. Yet here too, robust specificity degrades substantially, as steered models become more susceptibility to irrelevant or misleading contextual information. To summarize our contributions: (1) we introduce a framework for evaluating steering specificity along three crucial dimensions, (2) we provide the first systematic evaluation of specificity in two safety-critical settings: overrefusal and faithfulness hallucination steering, and (3) we show that steering methods lack robust specificity, as interventions that appear safe under standard benchmarks fail under adversarial settings. Together, these findings highlight the need to evaluate steering not only for efficacy, but also for specificity. Without such evaluation, interventions that appear precise may in fact weaken model safety and reliability. 1 1 1 Code and dataset are available at https://github.com/navitagoyal/steering-specificity/ . Figure 2: Faithfulness hallucination steering evaluation. 2 Related Work Inference-time intervention or activation steering Inference-time intervention methods offer lightweight and precise mechanisms to influence LLM behavior without extensive data collection or retraining efforts 20 . These methods apply directional perturbations to internal model activations by isolating, and subsequently amplifying or ablating linear directions associated with specific concepts 38 ; 44 . A growing body of work demonstrates the promise of such methods for steering model behavior 41 ; 31 ; 2 ; 34 . Sparse autoencoders (SAEs) represents another emerging approach for LLM interpretation and fine-grained steering 5 ; 37 . SAEs aim to decompose LLM representations into meaningful latent features by learning a higher-dimensional basis that can sparsely reconstruct hidden activations. Prior work has used SAEs for steering by identifying and amplifying or clamping latents linked to (un)desirable behavior 24 ; 6 ; 27 . Despite their promise, SAE-based steering remains difficult to apply and shows mixed results 3 ; 12 ; 40 . Comparisons indicate that SAEs perform worse than representation-based steering methods 40 and can degrade model capabilities 12 . Moreover, 17 show that SAEs are highly dataset dependent and often fail to find sparse, interpretable latents for important concepts such as refusal. Based on this evidence, we focus our analysis on representation-based steering methods rather than SAE steering. Evaluation of steering methods Recent work has begun formalizing desiderata for steering evaluation, covering aspects such as open-ended generation, standardized comparisons, and informative baselines 28 ; 4 , but one crucial yet underexplored criterion is specificity: interventions should alter only the intended property while leaving other behaviors intact. Specificity (also known as locality or selectivity) has long been recognized as a key desideratum in adjacent areas such as model unlearning or concept erasure. For instance, 11 , 25 , and 26 emphasize that editing a specific fact or set of facts should not affect unrelated knowledge, even relating to same subjects as the edited knowledge. Similarly, 13 assess specificity post-hoc by checking whether reintroducing the edited attribute restores the original model behavior. These works underscore that interventions must be minimally disruptive, a principle that is also relevant in the context of model steering, though not yet systematically operationalized or analyzed in this context. Large-scale benchmarks, such as Axbench 40 , employ steering toward concepts derived from sparse-autoencoder features, for example “ golden gate bridge ” 37 , with the prescribed goal that LLMs should incorporate a concept in their generation “even if the output is not related to the question or does not make sense”. While informative for studying steerability, such experimental setups lack a clear definition or measurement of specificity, which is a core component of practical steering applications. Benchmark /Method General Control Robustness ITI 20 Coherence ✗ ✗ ActAdd 38 Coherence ✗ ✗ CAA 31 Performance ✗ ✗ DiffMean 2 Performance ✗ ✗ RepE 44 Performance ✔ ✗ InterventionSuccess 3 Coherence ✗ ✗ AxBench 40 Coherence ✗ ✗ ReFT 41 ✗ ✗ ✗ RePS 42 ✗ ✗ ✗ Hypersteer 34 ✗ ✗ ✗ PartialOrth 39 ✗ ✔ ✗ SAE-TS 6 Coherence ✗ ✗ Table 1: An overview of specificity evaluations in existing literature. While some studies assess general or control specificity, robust specificity is almost entirely missing. In other steering research, aspects of specificity have been assessed but only partially. Most work evaluates retention of general model capabilities, such as whether steering preserves fluency 20 ; 38 ; 3 ; 40 or benchmark accuracy 31 ; 2 , while largely overlooking effects on related properties. A few studies probe these interactions: 44 examine whether increasing refusal on harmful queries affects compliance on benign queries, and 39 explicitly constrain overrefusal steering to maintain safety. However, such evaluations remain in-distribution and do not test how steering behaves under distribution shifts or adversarial prompts. Notably, prior work has explored out-of-distribution generalization of steering efficacy 20 ; 44 ; 35 ; 41 , but not their specificity. In contrast, we provide systematic evaluation of steering specificity, especially under adversarial attacks. 3 Desideratum for Specificity in Steering Specificity measures the extent to which steering methods affect only the intended target property without causing unintended side effects. While efficacy captures whether an intervention successfully alters the target property, specificity asks whether the intervention preserves everything else. We introduce three dimensions of specificity: • General specificity: Does steering preserve general model abilities such as coherence—are LLM outputs fluent—and performance—do LLMs perform well on standard benchmarks? • Control specificity: Does steering preserve properties that are closely related to the target property—referred to as control properties—without unintended modifications? For example, when reducing overrefusal, does the model still correctly refuse unsafe queries? • Robust specificity: Does steering preserve these control properties even under distribution shifts, such as adversarial attacks? Importantly, robust specificity differs from out-of-distribution efficacy often studied in prior work 20 ; 44 ; 35 ; 42 . Whereas prior evaluations test whether steering transfers across distributions for the target property, we ask whether control properties are preserved under such shifts. Table 1 summarizes how existing work evaluates steering specificity. Most prior studies only evaluate general specificity, typically measuring fluency or benchmark performance, though even these evaluations are not consistently included in steering evaluations. A smaller number of works examine control specificity. For example, 44 test whether steering models to refuse harmful queries also changes behavior on benign prompts, and 39 study whether reducing overrefusal inadvertently weakens refusal on truly harmful inputs. While these studies offer some insights into control specificity, they do so in isolation, often for a single method and task, and without a comprehensive evaluation of other aspects of specificity. Notably, robustness specificity remains almost entirely unexplored in prior work. In contrast, our work provides the first systematic and multi-dimensional evaluation of steering specificity, jointly assessing general, control, and robustness specificity across a broad range of steering methods and safety-critical settings. 4 Evaluating Specificity In this section, we operationalize our three-part framework in two safety-critical use cases: steering to reduce overrefusal and faithfulness hallucinations in LLMs. 4.1 Overrefusal Steering For the case of overrefusal steering, the target property is compliance on benign-but-overrefused queries, while the control property is refusal on genuinely harmful queries. This captures a realistic and safety-critical application of steering: improving usability by reducing overrefusal must not compromise refusal on unsafe requests. General specificity measures whether steering degrades the model’s overall fluency and performance on tasks unrelated to refusal. Following prior work, we quantify fluency via perplexity 38 and general capabilities via benchmark accuracy 31 , specifically on MMLU 15 —testing general knowledge via multiple choice question answering—and GSM8K 9 —testing open-ended math reasoning. Control specificity measures whether steering preserves safety on harmful queries drawn from the same distribution as training data. To evaluate this, we adopt two safety metrics from prior work 19 ; 21 ; 2 : ComplianceRate , calculated as the proportion of harmful prompts where the model does not produce refusal style response, such as “ I am sorry, I can not ”, and Harmscore , calculated as the harmfulness score assigned by Llama Guard 2, a model explicitly finetuned to detect harmful content 22 . Finally, robust specificity assesses whether steering preserves safety under distribution shifts, in particular, jailbreaking attacks 32 —prompts designed to bypass LLM safeguard and elicit harmful content. Even if steering preserves refusal on harmful queries, it may still weaken defenses against such adversarial attacks. To evaluate this, we prepend jailbreaks to harmful queries and measure safety using the same metrics as above. 4.2 Faithfulness Hallucination Steering For the case of faithfulness hallucination steering, the target property is faithfulness to provided contextual information when it contradicts the model’s internal knowledge, while the control property is the ability to answer accurately based on internal knowledge when no context is provided. This setting captures a common and safety-relevant failure mode in retrieval-augmented and in-context learning systems, where models may hallucinate answers that ignore or overwrite reliable context. General specificity measures whether steering degrades the model’s overall fluency and performance on tasks unrelated to faithfulness. As in overrefusal steering, we evaluate fluency using perplexity and assess general capabilities using MMLU and GSM8K benchmarks. Control specificity measures whether steering preserves accurate reliance on internal knowledge in the absence of contextual information. To evaluate this, we measure answer correctness on question-answering tasks without provided context, ensuring that steering for contextual faithfulness does not degrade the model’s ability to respond accurately based on its pretrained knowledge. Finally, robust specificity assesses whether steering preserves control specificity under distribution shifts in the contextual information. In particular, we test adversarial contexts that are designed to distract the model. Even if steering improves faithfulness when context is relevant, it may increase susceptibility to spurious or misleading context. We evaluate robustness by measuring the extent to which steered models incorrectly incorporate irrelevant context into their responses. For each specificity dimension, we report the degradation in the respective measure upon steering. That is, \Delta=m(\mathcal{M}^{\prime})-m(\mathcal{M}) , where \mathcal{M}^{\prime} is the steered model, \mathcal{M} is the unsteered baseline and m is the respective metric (for example, perplexity or benchmark accuracy for general specificity). Subsequently, a large negative \Delta_{\text{general}} would indicate degradation in general abilities caused by steering. Similarly, a large negative \Delta_{\text{control}} would indicate that steering undermines behaviors related to the control property, for example, models’ abilities to appropriately refuse harmful queries. Similarly, a large negative \Delta_{\text{robust}} would indicate a lack of robustness, where improvements in the target property come at the cost of degraded behavior under distribution shifts or adversarial conditions. 5 Overview of Steering Baselines We evaluate five representative steering methods spanning unsupervised and supervised approaches: Difference-in-Means (DiffMean) 20 ; 38 ; 31 ; 2 , Linear Probing (LinearProbe) 44 , Supervised Steering Vector (SSV) 41 , Rank-1 Representation Finetuning (ReFT-r1) 41 ; 40 , and Partial Orthogonalization (PartialOR) 39 . For DiffMean, LinearProbe, SSV, and ReFT-r1, steering is implemented by learning a direction \mathbf{w} in representation space from supervised examples. The methods differ in how the direction is computed: DiffMean uses the mean difference between hidden activations for positive and negative examples; LinearProbe uses the weight vector of a classifier trained to distinguish these activations; SSV directly optimizes the likelihood of positive demonstrations; and ReFT-r1 combines probing and supervised objectives with additional sparsity regularization. At inference, steering is applied by adding the learned vector to hidden activations, that is, \mathcal{M}^{\prime}=\mathcal{M}_{h^{l,k}\leftarrow h^{l,k}+\alpha\mathbf{w}} , where h^{l,k} denotes the activation at layer l and token position k , \mathbf{w} is the steering vector, and \alpha is a steering factor. PartialOR differs qualitatively from the above methods as it explicitly controls for the preservation of the control property. Rather than learning a single direction from labeled examples, it explicitly learns a target direction (e.g., overrefusal) and a control direction (e.g., true refusal). The final steering vector is obtained by projecting the target vector to be orthogonal to the control vector. Subsequently, model activations are projected onto the nullspace of the learned steering vector. See Appendix A for full training objectives. Steering with and without explicit control. While steering methods are generally designed to enhance or suppress a target property, they can also be adapted to explicitly retain a control property. For instance, rather than simply steering a model to be more compliant on pseudo-harmful queries, we could additionally ascertain that the model retains refusal behavior on harmful queries. Among the evaluated methods, PartialOR is explicitly designed to enforce such control. For a fair comparison, we therefore introduce a with explicit control counterpart of the other steering methods. Concretely, in the without-explicit control setting, training examples are chosen solely to capture the target property. For overrefusal steering, we construct positive examples as (x^{\text{pseudo}},y^{\text{pseudo}})\cup(x^{\text{harmless}},y^{\text{harmless}}) , that is, pseudo-harmful and harmless queries paired with compliant demonstrations, and negative examples as (x^{\text{pseudo}},y^{\text{refusal}}) , that is, pseudo-harmful queries paired with refusal responses. For faithfulness hallucination steering, we construct positive examples as (x^{\text{update}},y^{\text{update}}) , where query is accompanied by updated contextual information that contradicts the model’s internal knowledge, paired with the updated answer, and negative examples as (x^{\text{update}},y^{\text{original}}) , which pair the same input with the model’s original answer. In the explicit-control setting, training examples are augmented to preserve the control property. For overrefusal steering, we augment the positive example set with (x^{\text{harmful}},y^{\text{refusal}}) , that is, harmful queries paired with refusal responses. For hallucination steering, we augment positive examples with (x^{\text{none}},y^{\text{original}}) , where the input query has no additional context, paired with the original answer. These control examples anchor the steering direction to preserve the model’s intended behaviors while still optimizing for the target property. Model Method Efficacy ( \uparrow ) Specificity ( \uparrow ) General Control Robustness PHTest ORBench MMLU GSM8K Fluency JBBench +Jailbreak Llama-8B-Instruct Baseline 0.84 0.43 0.55 0.45 -3.84 0.97 0.55 DiffMean 0.08 0.15 -0.07 0.05 -0.04 -0.04 -0.28 LinearProbe 0.10 0.32 0.00 0.02 -0.40 -0.10 -0.30 SSV 0.04 -0.21 -0.08 0.03 -0.57 0.01 -0.19 ReFT-r1 0.05 0.05 0.03 0.06 -0.07 -0.03 -0.23 PartialOR 0.09 0.57 -0.01 0.02 -0.17 -0.03 -0.25 Qwen-7B-Instruct Baseline 0.84 0.52 0.62 0.19 -3.78 0.88 0.60 DiffMean 0.10 0.34 -0.03 -0.01 -0.62 -0.02 -0.16 LinearProbe 0.10 0.38 -0.02 -0.04 -0.86 -0.03 -0.14 SSV 0.04 -0.02 -0.29 -0.17 -12.21 0.05 -0.05 ReFT-r1 0.10 0.38 -0.01 -0.03 -0.64 -0.03 -0.18 PartialOR 0.07 0.35 -0.02 -0.02 -0.67 0.05 -0.12 Llama-3B-Instruct Baseline 0.68 0.64 0.49 0.31 -3.92 0.95 0.92 DiffMean 0.32 0.31 0.01 0.01 0.08 -0.08 -0.15 LinearProbe 0.29 0.25 -0.05 0.07 0.11 -0.08 -0.08 SSV 0.16 0.02 -0.08 0.02 -0.04 -0.03 -0.09 ReFT-r1 0.28 0.24 -0.04 0.01 0.00 -0.07 -0.11 PartialOR 0.28 0.28 0.02 0.04 0.02 0.01 -0.09 Gemma-2B-Instruct Baseline 0.63 0.35 0.29 0.00 -7.27 0.92 0.60 DiffMean 0.18 0.19 0.00 0.00 -0.43 0.04 -0.16 LinearProbe 0.11 0.15 -0.01 0.00 0.04 0.04 -0.12 SSV 0.14 0.16 0.00 0.00 -1.48 0.04 -0.16 ReFT-r1 0.15 0.18 0.00 0.00 -0.19 0.04 -0.15 PartialOR 0.09 0.20 0.00 0.00 -0.24 0.04 -0.10 Table 2: Efficacy and specificity evaluation in overrefusal steering (with explicit control). Efficacy: change in ComplianceRate for overrefusal benchmarks (PHTest and ORBench-hard) relative to the baseline; General specificity: change in MMLU accuracy, GSM8K accuracy, and fluency; Control specificity: change in safety (i.e., 1- HarmScore ) on JailbreakBench benchmark; Robust specificity: difference in safety on JailbreakBench queries prefixed with jailbreak prompts. Baseline values are shown for reference. Significant deviations \Delta from the baseline are marked in bold (at p<0.05), with degradations underlined in red. While safety in largely preserved on canonical harmful queries, jailbreak robustness consistently drops when steering to reduce overrefusal. Figure 3: Utility and safety trade-off for different steering methods. The arrows indicate the difference between safety in-distribution vs out-of-distribution. Steering improves the ComplianceRate on pseudo-harmful queries with some drop in safety. Importantly, jailbreaking success is higher after steering compared to baseline and steering methods with larger gains in utility generally show a lower robustness to adversarial attacks. 6 Experimental Details Datasets. We use PHTest 1 , JailbreakBench 7 , and Alpaca 36 datasets for pseudo-harmful, harmful, and harmless queries, respectively, to learn overrefusal steering vectors. We use NQSwap dataset 23 for faithfulness hallucination steering. Following prior work showing that steering directions can be estimated from small samples 20 , we use 256 training and 100 validation examples. For evaluation, we measure overrefusal steering efficacy using in-distribution ComplianceRate on PHTest test set and out-of-distribution ComplianceRate on the hard subset of ORBench dataset 10 . We measure hallucination steering efficacy using ExactMatch 30 on model answer and the updated answer on NQSwap test set. We measure general specificity by assessing performance on MMLU 15 and GSM8K 9 benchmarks and fluency (measured as the inverse of perplexity scores) on response to Alpaca queries. For overrefusal steering, we evaluate control specificity on harmful queries from the JailbreakBench test set and robust specificity by adding jailbreaking prefixes from JailbreakHub 32 to harmful test queries. For hallucination steering, we evaluate control specificity by estimating ExactMatch with original answer on queries without any additional information in context, and robust specificity by estimating ExactMatch with original answer on queries with distractor information in context. See Appendix C for more details on the dataset. Each evaluation uses 500 test queries. For robustness evaluation, we sample 25 jailbreaking prompts and apply them to 100 harmful queries, resulting in 2{,}500 adversarial test instances. Models. We evaluate four instruction-tuned models: Llama-3.1-8B-Instruct, Llama-3.2-3B-Instruct 22 , Qwen-2.5-7B-Instruct 29 , and Gemma-2-2B-it 14 . We experiment with instruction-tuned variants because they are already safety-aligned, typically refusing harmful queries reliably but often over-refusing benign ones, making them a natural testbed for overrefusal steering interventions. Hyperparameters. The steering factor \alpha , which controls the magnitude of intervention, and the intervention position are chosen to balance target and control property. For example, in overrefusal steering, we select \alpha that maximizes \textsc{ComplianceRate}_{\text{pseudo-harmful}}-\textsc{ComplianceRate}_{\text{harmful}} on the validation set. We search over \alpha\in\{0.5,1.0,2.0,3.0,4.0\} . We report result averaged over 3 runs. Baseline. As a baseline, we generate outputs from each model without any steering interventions. These baselines capture the inherent safety–utility trade-off of each model prior to steering. 7 Results: Are Steering Methods Indeed Precise? Table 2 shows efficacy and specificity for overrefusal steering with explicit control, with positive values indicating improvement and negative values indicating degradation relative to the baseline. We observe similar trends for hallucination steering ( Table 4 ) and settings without explicit control ( Table 5 ). For conciseness, we primarily focus our discussion on overrefusal steering. See Appendix B for a discussion of hallucination steering. Steering efficacy. As seen in Table 2 , all steering methods, with the exception of supervised steering vectors (SSV), successfully increase compliance on pseudo-harmful queries compared to the baseline, both in-distribution (PHTest) and out-of-distribution (ORBench). These findings confirm that steering effectively mitigates overrefusal, aligning with prior findings of success of inference-time interventions at controlling model refusal behavior 44 ; 39 . General specificity. Models largely maintain their fluency and performance on MMLU and GSM8K benchmarks after steering. 2 2 2 The low GSM8K performance of Gemma-2B-it primarily stems from limited instruction-following ability. While it often produces correct answers, it fails to follow formatting instructions required by the evaluation protocol. Even when graded more leniently on instruction-following, steered and unsteered models perform comparably across all steering methods, supporting our overall conclusion. Some methods exhibit a small negative shift in performance, but these differences are not significant at p <0.05 for any method except supervised steering vector based steering on Qwen-7B-Instruct model. Control specificity. Steering generally preserves the model’s refusal behavior on harmful queries drawn from the same distribution as the training data. Most deltas hover near zero, with slight negative and insignificant shifts in model safety in most cases except, a drop of 10% with LinearProbe steering in Llama-8B-Instruct and a drop of 7-8% with DiffMeans, LinearProbe, and ReFT-r1 steering in Llama-3B-Instruct model. Overall, however, steering methods preserve model’s refusal behavior on harmful queries under canonical settings. Robust specificity. In stark contrast, robustness evaluations reveal systematic vulnerabilities. Across all models, steering increases susceptibility to jailbreak attacks. In Llama-8B-Instruct, steering methods lead to a 35%-55% drop in jailbreak robustness. Similarly, susceptibility to jailbreaking attacks increases by 20-30% in Qwen-7B-Instruct model, by 8-16% in Llama-3B-Instruct (the model with the highest baseline jailbreak robustness), and by 17-27% in Gemma-2B-Instruct model. This demonstrates that current steering methods are lack robust specificity: they maintain refusal on standard harmful queries but collapse under adversarial attacks. Notably, we see that the strongest efficacy gains often coincide with the largest robustness declines (e.g., largest gains in utility (12%) in Llama-8B-Instruct model with LinearProbe steering correspond to largest drop in robustness (55%). Importantly, this failure of robust specificity is not unique to overrefusal steering, nor is it specific to jailbreaking-style attacks. We observe a similar pattern in faithfulness hallucination steering, where interventions that improve reliance on relevant contextual information and preserve accuracy in standard settings nonetheless degrade substantially under distribution shifts. Utility-Safety-Robustness trade-offs. To further visualize these trade-offs, we plot the utility-safety tradeoff across steering methods and models in Figure 3 . Ideally, a precise steering method should shift rightward (higher utility) without moving downward (loss of safety). Instead, we observe that gains in utility are often accompanied by losses in safety, particularly under jailbreak settings where the gap between in-distribution safety and out-of-distribution robustness (downward arrows) widens substantially after steering. Steering methods that lie further to the right on the utility axis generally exhibit lower robustness to jailbreaking. These patterns highlight a fundamental tension: interventions that most effectively reduce overrefusal tend to erode robustness, suggesting entangled latent representations that steering fails to disentangle. Method comparison. No single steering method consistently dominates across models. LinearProbe and ReFT-r1 overall show the highest efficacy gains, but accompanied by a large drop in jailbreak robustness. PartialOR yields more balanced trade-offs, with moderate efficacy and smaller robustness degradations. SSV, while sometimes effective, introduce the sharpest regressions in general specificity (e.g., 47\% drop in MMLU performance on Qwen-7B-Instruct model). These results indicate that the robustness concerns are not limited to a specific method or class of methods, but affect unsupervised and supervised approaches alike. Furthermore, since different steering methods yield different trade-offs, a comprehensive specificity evaluation can help practitioners pick most reliable and useful methods based on their use case. Figure 4: Top 2 principal components of activations at the last token position in layer 20 for harmless, pseudo-harmful, harmful, and harmful queries with jailbreaking prefix. The dotted lines visualize the decision boundaries corresponding to the different steering vectors. Activations of harmful queries with jailbreak prefixes lie markedly closer to that of harmless queries. Why does robustness decline after steering? To understand why steering retains safety in-distribution but fails under jailbreak attacks, we analyze hidden activations for harmful queries with and without jailbreaking prefix and compare them to the activations of harmless and pseudo-harmful queries. Figure 4 visualizes the first two principal components of these activations at the last token position in layer 20 of Qwen-7B-Instruct. We find that harmful queries with jailbreak prefixes have hidden activations markedly toward the same region as harmless queries. This is unsurprising as jailbreak prompts are designed to disguise harmful inputs so that their representations appear deceptively similar to safe one. Crucially though, since steering vectors are trained to increase compliance along directions separating benign from harmful activations, this overlap causes steering to inadvertently amplify compliance even for adversarially disguised unsafe inputs. This analysis suggests that the latent representations of harmful and harmless queries are not cleanly separated under adversarial transformations, implying that any steering method relying solely on these directions may inherently struggle to preserve robustness. 8 Discussion & Conclusion Steering methods are effective at controlling LLM generations, but specificity remains a critical and under-evaluated aspect. Across our evaluations of general, control, and robust specificity, we find that while steering preserves general model capabilities and even control properties under standard settings, robustness under adversarial inputs is consistently compromised. Our results establish how steering methods can introduce systematic issues and vulnerabilities, even when they appear effective and well-controlled in-distribution. These findings underscore the necessity of evaluating specificity in steering interventions since efficacy alone may give a misleadingly optimistic picture of steering as a “precise” tool for controlling LLM behavior. A natural mitigation strategy would be to tune steering vector or steering factor using jailbreaking queries 33 ; 43 to better capture distributional shifts. However, such approaches may not generalize to novel attacks. Importantly, our observations indicate that steering methods do not always work out-of-the-box and may require precise tuning for specific applications, highlighting the need for techniques that generalize well out-of-distribution in both efficacy and specificity. We hope our framework motivates more systematic development and evaluation of steering methods, guiding the design of interventions that are not only effective but also reliably safe. Limitations While our work highlights the issue of a lack of specificity evaluation in safety-critical contexts, our analysis is conducted in a single-task setup. For steering research to progress systematically, there is a need for stronger benchmarks that appropriately catalog both target and control properties, enabling more reliable assessment of steering efficacy and safety. Existing large-scale benchmarks 40 represent an important step toward evaluating model steering at scale, but as we show, these evaluations remain insufficient for assessing steering in practical, safety-critical use-cases. We view our evaluation framework as complementary to such efforts, offering an in-depth analysis of different aspects of steering criteria that are often underexplored. We assess the efficacy and specificity of generated text, following practices common in prior work 2 ; 39 . However, our evaluation framework can be readily extended to analyze sequence probabilities for additional insights, which we leave to future work. Due to compute constraints, our experiments are limited to LLMs up to 8B parameters, consistent with the scale of models evaluated in prior benchmarks 40 . Nonetheless, our framework can readily be extended to larger LLMs. We will release all code and data to support such extensions. Lastly, alignment through inference-time interventions requires white-box model access, similar to training-based strategies, such as SFT or RLHF post-training 8 ; 22 . For closed models, prompting remains the most accessible and effective alternative. Ethical Considerations This work investigates inference-time steering methods that steer LLM behavior without retraining. These methods raise important ethical and safety concerns due to the potential for dual use. Although our study focuses on LLMs and steering methods that are already publicly available, we recognize that releasing steering vectors that increase model susceptibility to adversarial attacks could facilitate misuse. Due to this, we advocate responsible disclosure: we will share all code necessary to replicate and extend our results, but we will not directly release specific intervention parameters that could weaken deployed models’ safeguards. Additionally, since our findings include compliant responses on harmful queries, we report only aggregate safety scores and exclude example generations to avoid spreading unsafe or harmful content. Second, steering interventions may affect latent model representations in opaque ways, degrading model safety. We therefore emphasize that it is important to evaluate not only efficacy but also specificity and robustness to adversarial attacks. Our findings highlight that even constrained steering can increase model vulnerabilities. We present these results to inform the research community of potential risks, not to enable exploitation. Finally, this work underscores the broader ethical tension between improving model utility and preserving safety. Steering interventions should be evaluated holistically, considering not only performance metrics but also downstream impacts on reliability, potential misuse, and societal harm. We encourage future work to incorporate explicit measures of specificity and robustness when developing and benchmarking steering methods. Appendix A Details: Steering Methods Without explicit control. Let x_{i}^{\text{pseudo}}\in X^{\text{pseudo}} be the set of pseudo-harmful queries and x_{j}^{\text{harmless}}\in X^{\text{harmless}} be the set of harmless queries, and let y_{i}^{\text{pseudo}} and y_{j}^{\text{harmless}} be the demonstration responses for these queries. Consider D^{+} = (x^{\text{pseudo}},y^{\text{pseudo}})\cup(x^{\text{harmless}},y^{\text{harmless}}) as the set of positive examples and D^{-} = ( x^{\text{pseudo}},y^{\text{refusal}} ) as the set of negative examples, where y^{\text{refusal}} represents a refusal response. Let h^{l,k}(x) be the hidden representation at layer l\in\{1,\ldots,L\} and token position k\in\{1,\ldots,n\} for an input x . \rhd Difference-in-means ( DiffMean ): DiffMean estimates the difference in the average of hidden representations from positive and negative sets of examples. The steering vector \mathbf{w} is calculated as \mathbf{w}_{\text{DiffMean}}=\frac{1}{|D^{+}|}\sum_{i\in D^{+}}h_{i}^{l,k}-\frac{1}{|D^{-}|}\sum_{j\in D^{-}}h_{j}^{l,k}, where h_{i}^{l,k} is the hidden representation at layer l and token position k for the concatenated input x_{i}\circ y_{i} . We can subsequently steer the model by adding this vector at the layer l and position k during inference. That is, h^{l,k}\leftarrow h^{l,k}+\alpha\mathbf{w}_{\text{DiffMean}} , where \alpha is the steering factor. \rhd Linear Probe (LinearProbe) : The linear probe first trains a linear classifier to distinguish between positive and negative sets of examples. Essentially, the method learns the direction \mathbf{w}_{\text{LinearProbe}}\in\mathbb{R}^{d\times 1} using binary cross-entropy loss with h_{i}^{l,k} as the input, where d is the size of hidden dimension h^{l,k} . The label for the cross-entropy loss is defined by whether the corresponding sample belongs in the set of positive examples or not. The steering follows in the same way as that for \mathbf{w}_{\text{DiffMean}} . \rhd Supervised steering vector ( SSV ): SSV learns the steering vector by directly optimizing for the language modeling probability of y_{j} for the set of positive examples. That is, \mathbf{w}_{\text{SSV}}\in\mathbb{R}^{d\times 1} is a learned vector such that \min_{\mathbf{w}_{\text{SSV}}}\Big\{\sum_{t=1}^{n}\log p_{LM}\big(y_{t}|y_{