Mechanistic Steering of LLMs Reveals Layer-wise Feature Vulnerabilities in Adversarial Settings

Nilanjana Das, Manas Gaur · 2026

Read Mechanistic Steering of LLMs Reveals Layer-wise Feature Vulnerabilities in Adversarial Settings by Nilanjana Das, Manas Gaur (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-23130.txt.

Introduction

Das and three colleagues trace jailbreak weakness in safety-aligned language models to small, prompt-linked groups of Sparse Autoencoder features, not broad refusal directions. With Gemma-2-2B and single-category unsafe cases from BeaverTails, they pull harmful concepts from adversarial replies and match them to prompt tokens. Three grouping methods across all 26 layers find vulnerable feature groups, strongest in mid-to-late layers. Amplifying the top features, single-token grouping matches full cluster grouping on harm, showing that single harmful tokens can pinpoint where steering breaks safety.

Catalog record

CREATORNilanjana Das, Manas Gaur
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21615
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.23130

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Mechanistic Steering of LLMs Reveals Layer-wise Feature Vulnerabilities in Adversarial Settings Nilanjana Das, Manas Gaur Abstract Large language models (LLMs) can still be jailbroken into producing harmful outputs despite safety alignment. Existing attacks show this vulnerability, but not the internal mechanisms that cause it. This study asks whether jailbreak success is driven by identifiable internal features rather than prompts alone. We propose a three-stage pipeline for Gemma-2-2B using the BeaverTails dataset. First, we extract concept-aligned tokens from adversarial responses via subspace similarity. Second, we apply three feature-grouping strategies (cluster, hierarchical-linkage, and single-token-driven) to identify SAE feature subgroups for the aligned tokens across all 26 model layers. Third, we steer the model by amplifying the top features from each identified subgroup and measure the change in harmfulness score using a standardized LLM-judge scoring protocol. In all three approaches, the features in the layers [16–25] were relatively more vulnerable to steering. All three methods confirmed that mid to later layer feature subgroups are more responsible for unsafe outputs. These results provide evidence that the jailbreak vulnerability in Gemma-2-2B is localized to feature subgroups of mid to later layers, suggesting that targeted feature-level interventions may offer a more principled path to adversarial robustness than current prompt-level defenses. Keywords: Mechanistic Interpretability, Adversarial Attacks, SAEs, Neural Features, LLM Safety 1 Introduction LLM’s rapid deployment in real-world systems has outpaced our understanding of their internal mechanisms. This creates a dangerous asymmetry: while LLMs demonstrate remarkable capability, we lack fundamental knowledge of what the model’s internal features interpret. Prior works on adversarial attacks like Greedy Coordinate Gradient (GCG) ( 18 ) that uses gradient-based optimization strategy at the token-level or Tree of Attacks with Pruning (TAP) ( 7 ) an effective black-box approach to create adversarial prompts are effective but do not explore the internal mechanism of how the model underwent a behavioral change (safe \rightarrow unsafe). To bridge this gap between attack efficacy and internal mechanism, we study the residual stream of a LLM, where it embeds the contribution from the attention and the MLP sub-layers. For an adversarial attack task, we presume that there lies an association between features in each of a model’s layers. We hypothesize that understanding the specific subgroups of neural features driving jailbreak success will enable more surgical, principled interventions. Furthermore, an important observation we make here is the localization of the feature subgroups in a layer that have the potential to steer the model to an unwanted behavior. Thus, we identify the feature subgroups activated during an adversarial attack by combining concept extraction, feature detection, and feature steering. By isolating these “jailbreak feature subsets,” we test their attribution to harmful response generation through steering experiments. This provides further information on the model layers and harm category that are more vulnerable to steering. The intellectual merit of this work lies in advancing mechanistic interpretability methodology for adversarial settings and revealing how LLMs’ model layers create exploitable vulnerabilities. The broader impact includes enabling safer AI systems and providing a scientific foundation for more principled safety interventions beyond today’s heuristic approaches. 2 Problem Formulation Task Definition Given a LLM M , when presented with an adversarial prompt from a benchmark dataset D , the model produces a response y . Our aim is to identify relevant subgroups of feature activations that play a vital role in enhancing the model’s harmfulness after the prompt is passed. The adversarial prompt-response pair chosen from D is denoted as <x,z> , where z is an unsafe response to x . An unsafe response contains harmful words/phrases that represent negative sentiment. These words/phrases are then extracted from z and reformulated as concept descriptions. To generate a vector representation of each of these descriptions, we used a subspace generator model G (a fine-tuned version of M ) from 16 , where a subspace lies in the embedding space of M . G can generate high-quality subspaces for the layer 20 residual stream of the instruction-tuned version of M , given a concept description. The concept descriptions ( w_{1},w_{2},\ldots,w_{m} ) that focus on negative sentiment were extracted from z using Grok’s non-reasoning model. For each concept description, we have a subspace such that S:\{s_{1},s_{2},\dots,s_{m}\} , where s_{1} is the vector for w_{1} , etc. We compute the cosine similarity between each subspace vector and the residual stream activations (feature activations for each token in the prompt) extracted at layer 20 of model M . \forall s_{i}\in S,\forall t\in T,cos\_sim(s_{i},layer20_{act_{t}}) 12 demonstrated that the middle layers of models exhibit the highest similarity between concepts; thus, we continued to use layer 20. We identify num tokens in a prompt with a positive cosine similarity and refer to them as pos . This is required to identify tokens that are similar to negative-sentiment concept descriptions and can therefore be used to find relevant feature subgroups for a successful jailbreak. To find relevant feature subgroups, we require feature activations. A SAE facilitates the layer-wise extraction of feature activations. We used the SAEs made available by 5 that can sparsely decompose latent representations into interpretable features. We find important feature subgroups from SAE features for an adversarial specific task such that we are able to steer M towards a relatively unsafe behavior when compared to its default behavior. Feature activations were extracted using SAEs attached to the residual stream of all layers. The cached SAE feature-activations as represented across the 26 layers of M can be described as: A=\{a_{0},a_{1},..,a_{25}\} , where, a_{k}\in\mathbb{R}^{B\times sl\times 16384} ( B = batch size, sl = sequence length) represents the SAE feature activations for sl tokens extracted at layer k . We focus only on the cached activations of the tokens in pos for each layer. We then perform agglomerative clustering and linkage analysis on transposed activation matrices to identify subgroups of features that activate across layers. We denote the set of transposed activations as A^{\prime}:\{a^{\prime}_{0},a^{\prime}_{1},..,a^{\prime}_{25}\} , where new a_{k}\in\mathbb{R}^{\text{num}\times 16384}\rightarrow a^{\prime}_{k}\in\mathbb{R}^{16384\times\text{num}} . On a^{\prime}_{k} we apply our method to identify relevant subgroups of features in layers. Below we describe the three methods to find feature subgroups. Figure 1: Approach 1: This figure represents steering with cluster-based feature selection. Results demonstrate (mid-late) layers are relatively more prone to steering. It also illustrates the harm categories that can be used to amplify the harmful behavior in the model. Feature Identification Through Three Approaches. Approach 1: Cluster-based steering. We identify the highest activating feature for all tokens in pos specifically in layer 20 and collectively refer to them as feature set \mathcal{F}:\{f_{1},f_{2},\dots,f_{n}\} . To find the feature subgroup for each f_{i} in each layer, we perform agglomerative clustering on a^{\prime}_{k}\in A^{\prime},\quad k\in\{0,\dots,25\} . An agglomerative clustering algorithm assigns a cluster number to each feature in each layer. For each layer and feature f_{i}\in\mathcal{F} , we identify the features that belong to the same cluster (have the same cluster number) as that of f_{i} and consider the identified features as part of the feature subgroup of f_{i} . This feature subgroups or clusters of features across layers can be described as C:\{c_{k-f_{1}},c_{k-f_{2}},\dots,c_{k-f_{n}}\} , k represents the layer, and f_{i} is the feature to which the feature cluster belongs. For each a^{\prime}_{k} we compute a^{\prime\prime}_{k} the average activation across num tokens such that a^{\prime\prime}_{k}\in\mathbb{R}^{16384}

Read the complete text