Distributional Surgery for Language Model Activations
Bao Nguyen, Binh Nguyen, Duy Nguyen, Viet Anh Nguyen · 2025
Read Distributional Surgery for Language Model Activations by Bao Nguyen, Binh Nguyen, Duy Nguyen, Viet Anh Nguyen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-15758.txt.
Introduction
Bao Nguyen, Binh Nguyen, Duy Nguyen, and Viet Anh Nguyen present a two stage guard for language model outputs in a 2025 paper. Models can slip into harmful or toxic text, and the team aim to catch and fix such slips from inside. First, an ensemble of layer by layer classifiers learns to detect bad content from the model's activations. The training uses a smoothed version of a risk aware score.
For flagged content, steering policies then adjust the attention heads at each layer. The policies are computed with a principled optimization that promises minimal disturbance while guaranteeing the edit works with high probability. Across several models and data sets, the method beat baselines at cutting undesirable output. The authors frame activation surgery as a precise, internal alternative to output filtering.
Catalog record
| CREATOR | Bao Nguyen, Binh Nguyen, Duy Nguyen, Viet Anh Nguyen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-15758
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Distributional Surgery for Language Model Activations Bao Nguyen, Binh Nguyen, Duy Nguyen, Viet Anh Nguyen Abstract Language models, while capable of generating remarkably coherent and seemingly accurate text, can occasionally produce undesirable content, including harmful or toxic outputs. In this paper, we present a new two-stage approach to detect and mitigate undesirable content generations by rectifying activations. First, we train an ensemble of layerwise classifiers to detect undesirable content using activations by minimizing a smooth surrogate of the risk-aware score. Then, for detected undesirable contents, we propose layerwise distributional steering policies that transform the attention heads. These policies are computed through principled semidefinite programming, which aims to minimally perturb the attention distribution while probabilistically guaranteeing the effectiveness of the editions. Empirical evaluations across multiple language models and datasets show that our method outperforms baselines in reducing the generation of undesirable output. 1 Introduction Language models (LMs) have demonstrated a remarkable ability to understand and generate human-like documents Radford et al. (2019) ; Brown et al. (2020) ; Touvron et al. (2023a) ; Touvron et al. (2023b) ; Jiang et al. (2023) ; Dubey et al. (2024) . However, inspection of their output often reveals undesirable generation, including inaccurate or toxic texts Ji et al. (2023) ; Rawte et al. (2023) ; Xu et al. (2024) . Meanwhile, developing effective strategies to control the generation process of LMs remains a significant challenge Tonmoy et al. (2024) . Researchers have proposed numerous methods for controllable text generation in language models Zhang et al. (2023) ; Li et al. (2024a) . These approaches primarily include model editing and supervised fine-tuning. Both methods, however, require altering the model weights using a subset of text samples, which can result in unstable representations for other text instances Hase et al. (2024) . Additionally, these approaches typically require substantial computational resources. To address these limitations, activation intervention has emerged as a promising alternative for controllable text generation Subramani et al. (2022) ; Hernandez et al. (2023) ; Li et al. (2024b) . This approach involves altering the model activations responsible for undesirable output during inference. Previous research has identified interpretable directions within the activation space of language models that play a causal role during inference. Studies by Burns et al. (2022) and Moschella et al. (2023) suggest that these directions can be manipulated to adjust model behavior in a controlled manner. This body of work suggests that the internal representations of language models are structured in a way that enables fine-grained control over generated text. Drawing inspiration from these findings, activation intervention frameworks operate on the premise that the information needed to guide the model toward generating a target sentence is already encoded within the model . The hidden information is extracted as latent vectors and then used to steer the generation toward producing desirable outputs. The preliminary success of these activation intervention methods motivates our approach to improving the quality and controllability of language model generation. Problem Statement. We consider a language model consisting of L layers, each layer has H heads, each head has dimension d . For example, for Llama-2, we have L=32 , H=32 , and d=128 . The training dataset is denoted by \mathcal{D}=(x_{i},y_{i}^{*})_{i=1,\ldots,N} , the i -th text is denoted by x_{i} , and its ground truth label is y_{i}^{*}\in\{0,1\} , where the label 1 (positive) represents the un desirable text, and the label 0 (negative) represents the desirable text. Our goal is twofold: (i) detect an undesirable text, and (ii) modify an undesirable text into a desirable text. The activations for a text x_{i} at layer \ell\in\{1,\ldots,L\} is denoted by a_{\ell,i} . The activation at layer \ell+1 is the output of the operation: \displaystyle a_{\ell+1,i} \displaystyle=a^{\mathrm{mid}}_{\ell,i}+\mathrm{FFN}(a^{\mathrm{mid}}_{\ell,i}), (1) \displaystyle a^{\mathrm{mid}}_{\ell,i} \displaystyle=a_{\ell,i}+\sum_{i=1}^{H}Q_{\ell h}\textrm{Att}(P_{\ell h}a_{\ell,i}). Here, P_{\ell h}\in\mathbb{R}^{d\times dH} is the projection matrix that maps the output of each layer to the d dimensional head space, \mathrm{Att} is the attention operator Vaswani et al. (2017) , Q_{\ell h}\in\mathbb{R}^{dH\times d} is the pull-back matrix and \mathrm{FFN} is the feed-forward layer. Each a_{\ell,i} is a concatenation of headwise activations a_{\ell h,i} for h=1,\ldots,H . Inspired by Li et al. (2024b) , we aim to perform intervention at some selected a_{\ell h,i} , the activations for head h of layer \ell , if we detect that the activation is from undesirable content. Contributions. We contribute an activation intervention method to detect and rectify the undesirable generation of LM. We call our method RADIANT ( R isk- A ware D istributional I ntervention Policies for Language Models’ A ctivatio n s). RADIANT comprises two components: 1. A layerwise probe: at each layer, we train a classifier to detect undesirable content from the layer’s activations. We train a risk-aware logistic classifier for each head that balances the false positive and false negative rates. Then, we aggregate these headwise classifiers’ predictions using a voting mechanism to form a layerwise classifier. We then identify one layer where the probe delivers the most reasonable predictive performance. This optimal classifier serves as the detector of undesirable content. 2. A collection of headwise interventions: given the optimal layer for the layerwise probe found previously, we find for each head in that layer an optimal headwise intervention policy. We choose a simple linear map for this intervention policy that minimizes the magnitude of editing while delivering sufficient distributional guarantees that the undesirable-predicted activations will be edited into desirable-predicted activations. We show that this linear map can be computed efficiently using semidefinite programming. 1.1 Related Works Controllable generation. Controllable text generation methods aim to alter the outputs of large language models in a desired way. One possible approach is model editing Wang et al. (2023a) ; Zhang et al. (2024) , which involves modifying the parameters of a model to steer its outputs. For example, Meng et al. (2022) involves identifying specific middle-layer feedforward modules that correspond to factual knowledge and then altering these weights to correct or update the information encoded by the model. Other notable methods include fine-tuning techniques such as Supervised Fine-Tuning (SFT, Peng et al. 2023 ; Gunel et al. 2020 ) and Reinforcement Learning from Human Feedback (RLHF, Ouyang et al. 2022a ; Griffith et al. 2013 ). Probing. Probing is a well-established framework to assess the interpretability of neural networks Alain and Bengio (2016) ; Belinkov (2022) . Probing techniques have been applied to understand the internal representations of transformer architectures in language models such as BERT and GPT. For example, Burns et al. (2022) proposed an unsupervised probing method that optimizes consistency between positive and negative samples. Marks and Tegmark (2023) computes the mean difference between true and false statements and skews the decision boundary by the inverse of the covariance matrix of the activations. Activation interventions. Activation intervention at inference time is an emerging technique for controllable generation Turner et al. (2023) ; Li et al. (2024b) ; Singh et al. (2024) ; Yin et al. (2024) . Unlike model editing or fine-tuning techniques, inference time intervention does not require altering the model parameters. Li et al. (2024b) proposed a headwise intervention method for eliciting truthful generated answers of a language model. They first train linear probes on each head of the language model, then shift the