DEAL : Disentangling Transformer Head Activations for LLM Steering
Li-Ming Zhan, Bo Liu, Zexin Lu, Yujie Feng, Chengqiang Xie, Xiao-Ming Wu ✉ · 2025
Read DEAL : Disentangling Transformer Head Activations for LLM Steering by Li-Ming Zhan, Bo Liu, Zexin Lu, Yujie Feng, Chengqiang Xie, Xiao-Ming Wu ✉ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-08359.txt.
Introduction
Li-Ming Zhan and four co-authors present a framework in two thousand twenty-five for steering large language models from the inside. Steering at inference time changes a model's answers without retraining it. The hard part is finding which internal modules drive the target behavior.
Their framework, REAL, trains a small autoencoder on each attention head or layer's activity. A shared codebook splits that activity into behavior-relevant and irrelevant parts. A classification score then rates how well each module separates aligned from violating responses, guiding both module choice and steering strength. Across eight models and nine data sets, REAL beat a leading steering method by about twenty percent on average. Its chosen modules also transferred to new domains.
Catalog record
| CREATOR | Li-Ming Zhan, Bo Liu, Zexin Lu, Yujie Feng, Chengqiang Xie, Xiao-Ming Wu ✉ |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-08359
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DEAL : Disentangling Transformer Head Activations for LLM Steering Li-Ming Zhan, Bo Liu, Zexin Lu, Yujie Feng, Chengqiang Xie, Xiao-Ming Wu ✉ Abstract Inference-time steering aims to alter the response characteristics of large language models (LLMs) without modifying their underlying parameters. A critical step in this process is the identification of internal modules within LLMs that are associated with the target behavior. However, current approaches to module selection often depend on superficial cues or ad-hoc heuristics, which can result in suboptimal or unintended outcomes. In this work, we propose a principled causal-attribution framework for identifying behavior-relevant attention heads in transformers. For each head, we train a vector-quantized autoencoder (VQ-AE) on its attention activations, partitioning the latent space into behavior-relevant and behavior-irrelevant subspaces, each quantized with a shared learnable codebook. We assess the behavioral relevance of each head by quantifying the separability of VQ-AE encodings for behavior-aligned versus behavior-violating responses using a binary classification metric. This yields a behavioral relevance score that reflects each head’s discriminative capacity with respect to the target behavior, guiding both selection and importance weighting. Experiments on seven LLMs from two model families and five behavioral steering datasets demonstrate that our method enables more accurate inference-time interventions, achieving an average relative improvement of 20% (up to 81.5%) over the ITI method DBLP:conf/nips/0002PVPW23 on the truthfulness-steering task. Furthermore, the heads selected by our approach exhibit strong zero-shot generalization in cross-domain truthfulness-steering scenarios. 1 Introduction As large language models (LLMs) demonstrate increasingly versatile capabilities, understanding and controlling their internal decision-making processes has become more challenging. Consequently, steering the behavior of LLMs to ensure they produce outputs with desirable properties—such as truthfulness, appropriate refusals, and minimal hallucinations—has become a primary focus in both research and practice ( yang2024behavior ; perez2022discovering ; goyal2024llmguard ) . Inference-time activation engineering ( arditi2024refusal ; DBLP:conf/nips/0002PVPW23 ; Zou2023RepresentationEA ; lee2024programming ) has emerged as a promising approach for guiding the behaviors of LLMs, due to its high efficiency and strong generalization capability. Works in this area aim to steer LLM behavior by applying additive perturbations to their intermediate activations. The typical process involves two main steps: (i) identifying internal modules (such as attention heads) that are linked to the target behavior, and (ii) constructing a steering vector to modify the original activations accordingly. Accurately identifying which modules are relevant to the target behavior is crucial for effective steering. However, due to the complexity of LLM internals, existing methods often rely on superficial linear correlations ( DBLP:conf/nips/0002PVPW23 ) , ad-hoc empirical cues ( DBLP:conf/nips/YinYD24 ) , or computationally expensive cross-validation ( zhang-etal-2024-truthx ; rimsky-etal-2024-steering ) to select intervention modules. Therefore, a principled, effective, and efficient framework for module selection is essential for advancing inference-time steering. To address this gap, we present DEAL — D isentangling Transformer h E ad A ctivations for L LM Steering —a novel framework that disentangles behavior related representation from head activations and computes head-wise confidence scores to facilitate LLM steering. Attention heads govern how LLMs generate text, enabling diverse functions such as induction DBLP:conf/naacl/CrosbieS25 ; olsson2022context and long-range factual retrieval wu2024retrieval . However, these functions are encoded within the heads’ hidden activations in a highly entangled manner DBLP:conf/acl/MoneaPJCEKPP024 ; ferrando2024primer , hindering the extraction of behavior-specific features from the activations. We address this challenge by learning a disentangled latent space that separates each head activation vector into behavior-relevant and behavior-irrelevant components. Specifically, for each attention head, we train a vector-quantized autoencoder (VQ-AE) that divides the latent space into several subspaces—some capturing behavior-relevant features and others capturing behavior-irrelevant features. Each subspace is quantized by a shared learnable codebook. The VQ-AE is optimized using a standard reconstruction loss alongside an auxiliary supervised contrastive loss, which encourages separation between positive and negative behaviors. After training, the VQ-AE produces a discrete, sequential code for each head activation. To quantify the behavioral relevance of each head, we fit an autoregressive prior p_{\theta} over the discrete codes, estimating the likelihood that a given code aligns with the target behavior. For each head, we assess the separability of these likelihoods for behavior-aligned versus behavior-violating responses using a binary classification metric (e.g., AUC-ROC), yielding a behavioral relevance score that quantifies each head’s discriminative ability. This score is then used to guide both head selection and importance weighting. To demonstrate the effectiveness of DEAL , we evaluated it on seven LLMs from two leading open-source families— Llama and Qwen . We applied our head-selection strategy to two prevalent attention variants: standard multi-head attention (e.g., Llama-7B, Llama2-7B, Llama2-13B-Chat) and the more efficient grouped-query attention (GQA) ainslie2023gqa (e.g., Qwen2.5-7B, Llama3.1-8B-Instruct). DEAL integrates seamlessly with current behavior-steering methods, including the mean-difference technique ITI DBLP:conf/nips/0002PVPW23 and the fine-tuning approach LoFiT DBLP:conf/nips/YinYD24 . On the truthfulness-steering task, DEAL achieved an average relative improvement of 20\% (up to 81.5%) over ITI across the seven LLMs. Moreover, the heads selected by DEAL exhibit strong generalization in zero-shot, cross-domain truthfulness-steering scenarios. 2 Preliminary Activation engineering ( DBLP:conf/nips/0002PVPW23 ; rimsky-etal-2024-steering ) is an emerging paradigm aimed at directing desired behaviour in LLMs without the need for fine-tuning. This technique injects perturbations into the intermediate activations of LLMs to influence their outputs. It typically involves three steps: (i) select a set of behaviour-specific attention heads; (ii) for each selected head, compute a steering vector from its activations; and (iii) during decoding, add this steering vector to the head’s original activation, thereby biasing the model toward the desired behaviour. Step 1: Head Selection. In a decoder–only LLM, each transformer layer typically employs multiple parallel self-attention heads, enabling each head to specialize in distinct patterns, such as induction and long-range fact retrieval ( wu2024retrieval ) . We denote the selected subset of attention heads as \mathcal{G}\subseteq\{(l,i)\mid l\in\{1,\dots,L\},\,i\!\in\!\{1,\dots,H\}\} , where l is the layer index, L is the number of transformer layers, h is the head index within each layer, and H is the number of heads per layer. Notably, |\mathcal{G}|\ll LH . The selection of attention heads for intervention is crucial ( DBLP:conf/nips/YinYD24 ) . A suitable choice can elicit the desired behavior, while an incorrect selection may significantly degrade performance. In this study, we focus on developing a principled and effective head selection method. Step 2: Steering Vector Derivation. For an input token sequence \{x_{1},x_{2},\ldots,x_{T}\} , the head-specific context vector at time step t is computed as \mathbf{h}_{t}^{(l,i)}=\mathrm{Attention}\!\bigl(\mathbf{Q}_{t}^{(l,i)},\mathbf{K}_{\leq t}^{(l,i)},\mathbf{V}_{\leq t}^{(l,i)}\bigr), (1) where \mathbf{Q}^{(l,i)},\mathbf{K}^{(l,i)},\mathbf{V}^{(l,i)}\!\in\!\mathbb{R}^{T\times d_{h}} denote the query, key, and value projections, respectively, and d_{h} is the dimensionality per head. Given a behavior-contrastive dataset \mathcal{D}=\{(q_{i},a_{i},y_{i})\}_{i=1}^{N} , where q_{i} denotes a query, a_{i} is its associated answer, y_{i}\in\{0,1\} indicates whether the pair exhibits the target behavior (e.g., truthfulness), and N is the dataset size. For each query, the corpus contains both compliant and non-compliant answers, allowing the formation of a positive behavior subset \mathcal{D}^{+}=\{(q_{i},a_{i})|y_{i}=1\} and a negative behavior subset \mathcal{D}^{-}=\{(q_{i},a_{i})|y_{i}=0\} . The steering vector \mathbf{v}^{(l,i)} for head (l,i) can be derived using the mean difference method, applied to the activations from \mathcal{D}^{+} and \mathcal{D}^{-} respetively ( lee2024programming ; DBLP:journals/corr/abs-2308-10248 ; rimsky-etal-2024-steering ; DBLP:conf/nips/0002PVPW23 ) , or by employing the fine-tuning method ( DBLP:conf/nips/YinYD24 ) . Step 3: Steering. During decoding, the activation of each selected head is perturbed by: \hat{\mathbf{h}}_{t}^{(l,i)}=\mathbf{h}_{t}^{(l,i)}+\epsilon\,\mathbf{v}^{(l,i)},\quad(l,i)\in\mathcal{G}, (2) where \mathbf{v}^{(l,i)} is the steering vector for head (l,i) and \epsilon\in\mathbb{R}^{+} controls