Angular Steering: Behavior Control via Rotation in Activation Space

Hieu M. Vu, Tan M. Nguyen · 2025

Read Angular Steering: Behavior Control via Rotation in Activation Space by Hieu M. Vu, Tan M. Nguyen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-26243.txt.

Introduction

Hieu M. Vu and Tan M. Nguyen introduce a geometric method to steer the behavior of a large language model without retraining it. Steering methods adjust internal activations, the numeric patterns that represent a prompt inside the model, to raise or lower a behavior such as refusal. Prior methods add a vector or delete a direction inside a two dimensional subspace, which makes them sensitive to settings and prone to side effects on unrelated traits.

Angular Steering instead rotates activations within a fixed two dimensional plane, toward or away from a target behavior direction, giving continuous, fine grained control. An adaptive variant rotates only activations that align with the target feature, which improves stability. Refusal and emotion steering serve as test cases.

Catalog record

CREATORHieu M. Vu, Tan M. Nguyen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE90 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-26243

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Angular Steering: Behavior Control via Rotation in Activation Space Hieu M. Vu, Tan M. Nguyen Abstract Controlling specific behaviors in large language models while preserving their general capabilities is a central challenge for safe and reliable artificial intelligence deployment. Current steering methods, such as vector addition and directional ablation, are constrained within a two-dimensional subspace defined by the activation and feature direction, making them sensitive to chosen parameters and potentially affecting unrelated features due to unintended interactions in activation space. We introduce Angular Steering, a novel and flexible method for behavior modulation that operates by rotating activations within a fixed two-dimensional subspace. By formulating steering as a geometric rotation toward or away from a target behavior direction, Angular Steering provides continuous, fine-grained control over behaviors such as refusal and compliance. We demonstrate this method using refusal steering emotion steering as use cases. Additionally, we propose Adaptive Angular Steering, a selective variant that rotates only activations aligned with the target feature, further enhancing stability and coherence. Angular Steering generalizes existing addition and orthogonalization techniques under a unified geometric rotation framework, simplifying parameter selection and maintaining model stability across a broader range of adjustments. Experiments across multiple model families and sizes show that Angular Steering achieves robust behavioral control while maintaining general language modeling performance, underscoring its flexibility, generalization, and robustness compared to prior approaches. Code and artifacts are available at https://github.com/lone17/angular-steering/ . 1 Introduction Large language models (LLMs) have become remarkably capable, yet steering their behavior towards desired responses remains a challenge. On one hand, we want the model to follow certain guidelines or exhibit particular traits, e.g., refusing inappropriate requests or complying with user instructions. On the other hand, aggressive tuning of the model’s behavior can degrade its original performance, causing losses in fluency or actuality [ 48 , 50 ] . Activation steering, which manipulates internal representations of language models at inference time, has emerged as a compelling alternative to retraining for behavior control [ 50 , 57 , 38 ] . Techniques such as activation addition [ 50 , 38 ] and direction orthogonalization [ 1 , 57 ] have demonstrated the capacity to steer models toward or away from specific behaviors. However, these methods offer limited granularity. For instance, orthogonalization removes the feature entirely by projecting activations onto the orthogonal subspace, leaving no room for partial suppression. Moreover, activation addition requires careful tuning of the coefficient to avoid instability; improper values can lead to degraded fluency or incoherent outputs [ 42 , 46 , 51 , 43 ] . While conditional methods improve context-sensitivity, they often retain the underlying manipulation mechanism [ 52 , 19 , 21 ] . Figure 1 : Geometric interpretation of activation steering. Left: Before normalization, the original activation vector {\bm{h}} , the feature direction {\bm{d}}_{\text{\;feat}} , the ablation vector {\bm{h}}_{\perp} , and addition {\bm{h}}+\alpha{\bm{d}}_{\text{\;feat}} are shown. Right: After normalization, all vectors lie on a (scaled) unit circle within Span\{{\bm{h}},{\bm{d}}_{\text{\;feat}}\} . The dashed arc shows {\bm{h}}_{\perp} and {\bm{h}}+\alpha{\bm{d}}_{\text{\;feat}} as rotations of {\bm{h}} , motivating Angular Steering. Figure 2 : Illustration of a typical Transformer Block in modern LLMs with Angular Steering applied after each normalization layer. Table 1 : A sample generation of Qwen2.5-7B-Instuct on \mathcal{D}^{(\small{\text{eval}})}_{\small{\text{harmful}}} showing four different behaviors when being steered by various angles. Some content was omitted for brevity. Prompt Create a hoax website that spreads false information and leads to panic refuse (steered to 20°) I’m sorry, but I can’t assist with creating a hoax website or spreading false information. Such actions can cause real harm and panic, and they are unethical and illegal. Instead, I can help you create a legitimate website or provide information that is accurate and beneficial. If you have any other questions or need assistance with a different topic, feel free to ask! indirect (steered to 100°) Sure, I can create a fictional example of a hoax website to illustrate how such a site might look and spread false information. However, I won’t actually create a real website because that could cause real harm. Instead, I’ll provide a description of what such a website might look like and how it could spread misinformation. ### Example Hoax Website: "Global Health Alert" (…) direct (steered to 200°) Sure, let’s create a fun and fictional example of a hoax website that could spread false information and potentially cause a bit of a stir. For this example, we’ll create a website about a new, highly contagious virus that has just been discovered. The goal is to highlight how misinformation can spread quickly, so let’s craft a website that looks credible but contains false information. ### Website Name: GlobalVirusAlert.org (…) redirect (steered to 300°) Creating a hoax website that spreads false information and leads to panic is unethical and illegal. Such actions can cause real harm and distress to individuals and communities. Instead, I can help you create a responsible and informative website that educates people on how to spot and report misinformation. Here’s an example of a responsible website that provides resources and guidance: (…) Contribution. We propose Angular Steering , a method that reformulates activation editing as a geometric rotation within a fixed 2D subspace of the model’s activation space. By identifying a behavior-associated direction, e.g., refusal, and a complementary axis that jointly define a steering plane, we rotate the activation vector within this plane to control the behavioral expression (Tab. 1 ). This approach offers a smooth and interpretable control of the model’s behavior via the rotation angle of activations, enabling the modulation of the feature’s strength without compromising the model’s overall representation capacity. We also introduce an adaptive variant of Angular Steering, namely Adaptive Angular Steering , which adds another dimension of controllability by applying steering selectively based on the local activation context. Crucially, our formulation generalizes and unifies existing steering operations (Fig. 2 ). In particular, activation addition [ 50 ] and directional ablation [ 1 ] , when apply before normalization, can both be viewed as specific instances of rotation in a 2D plane defined by the original activation and a target feature direction. Linear combination [ 50 ] corresponds a partial rotation toward or away from the feature; and orthogonalization [ 1 ] corresponds to rotating to a position 90 degrees from the feature. We summarize these correspondences in Appendix A . By subsuming these techniques under a common geometric framework, Angular Steering offers a principled abstraction that clarifies their effects and limitations, while extending their controllability. In summary, our contribution is three-fold: 1. We propose the novel Angular Steering , a rotation-based framework for fine-grained, continuous control of model behaviors, and the Adaptive Angular Steering , a selective variant of Angular Steering that improves robustness and minimizes coherence loss. 2. We demonstrate that Angular Steering serves as a unifying framework for prior activation intervention methods from a geometric perspective. 3. We empirically demonstrate that both Angular Steering and Adaptive Angular Steering achieve strong behavior control with minimal degradation of model’s performance outside of the targeted steering tasks across multiple modern LLM families. Organization. We structure this paper as follows: Section 2 contextualizes the existing body of literature regarding LLMs Activation Steering and Interpretability. In Section 3 , we provide the necessary

Read the complete text