Understanding and Mitigating Dataset Corruption in LLM Steering

Cullen Anderson, Narmeen Oozeer, Foad Namjoo, Remy Ogasawara, Amirali Abdullah, Jeff M. Phillips · 2026

Read Understanding and Mitigating Dataset Corruption in LLM Steering by Cullen Anderson, Narmeen Oozeer, Foad Namjoo, Remy Ogasawara, Amirali Abdullah, Jeff M. Phillips (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-03206.txt.

Introduction

Cullen Anderson, Narmeen Oozeer, Foad Namjoo, and colleagues test how poisoned examples can hijack an AI model's steering. Contrastive steering finds a trait direction by comparing the model's inner signals on paired examples. It then shifts those signals along that line at runtime. Safety teams use it a lot, yet nobody had checked how it holds up when the example set is rigged.

A modest share of rigged examples leaves steering mostly intact. Past a clear threshold, though, attackers can plant strong unwanted side effects on purpose. A look at the shapes of distinct attack types points to guards. The weak step is a plain average over many signals. Swapping in a robust form of that average cancels most of the harm.

Catalog record

CREATORCullen Anderson, Narmeen Oozeer, Foad Namjoo, Remy Ogasawara, Amirali Abdullah, Jeff M. Phillips
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46002
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.03206

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Understanding and Mitigating Dataset Corruption in LLM Steering Cullen Anderson, Narmeen Oozeer, Foad Namjoo, Remy Ogasawara, Amirali Abdullah, Jeff M. Phillips Abstract Contrastive steering has been shown as a simple and effective method to adjust the generative behavior of LLMs at inference time. It uses examples of prompt responses with and without a trait to identify a direction in an intermediate activation layer, and then shifts activations in this 1-dimensional subspace. However, despite its growing use in AI safety applications, the robustness of contrastive steering to noisy or adversarial data corruption is poorly understood. We initiate a study of the robustness of this process with respect to corruption of the dataset of examples used to train the steering direction. Our first observation is that contrastive steering is quite robust to a moderate amount of corruption, but unwanted side effects can be clearly and maliciously manifested when a non-trivial fraction of the training data is altered. Second, we analyze the geometry of various types of corruption, and identify some safeguards. Notably, a key step in learning the steering direction involves high-dimensional mean computation, and we show that replacing this step with a recently developed robust mean estimator often mitigates most of the unwanted effects of malicious corruption. 1 Introduction As large language models (LLMs) and the resulting chatbots and agents get more and more integrated into scientific and everyday activities, the need to mechanistically understand them and tune them for various purposes becomes more crucial. Contrastive steering – where one can find an activation layer in the LLM which captures a behavior, and linearly modify to control that behavior – has become a central tool in this enterprise. These methods have been shown to be surprisingly effective in controlling a wide variety of behaviors ( 45 ; 31 ) , and as a result are being integrated into widely-used products ( 7 ) . As such, it is crucial to understand the robustness of these methods. In particular, in this paper, we study the robustness of the main input to this contrastive steering process: the datasets used to train them. How do changes in the datasets used to train steering affect their performance? To investigate this, we denote data which correctly models the behavior desired to steer as inliers and data that have been corrupted as outliers . Then, we investigate three predominant forms of data corruption: • Random corruption where the outliers are random. This is the most benign level of corruption, where bad data appear in the training set that do not correspond to the behavior, and there is no describable pattern. Many of the training data for steering are automatically generated, and this process may be faulty. • Mislabeling corruption where the outliers fit the data distribution, but the label as having or not having the behavior is flipped. This corruption is common ( 24 ) and corresponds to the well-studied Massart noise ( 6 ) . • Coordinated Behavior Corruption where the outliers are coordinated to represent a particular other behavior. This coordinated corruption has multiple stronger effects: It can be harder to remove. It can also more strongly pull the learned steering direction away from the intended behavior. And finally, it can induce a secondary (unwanted) behavioral effect from steering. While some manifestations of these corruptions may be accidental or due to datasets just becoming stale, we remark that large companies and services that rely on steering within LLMs should be cognizant of this potential attack. Due to the quickly developing nature of this useful mechanism, the creation of steering datasets has not always been carefully reviewed or protected. This paper aims to evaluate and highlight this potential issue. Regardless of such corruption, the steering infrastructure can potentially adapt to such poor or manipulated data. We leverage that the core mechanism for learning a steering vector is the difference of means ( 8 ; 34 ) where at the most effective layer, (1) the activations of responses labeled with a behavior and labeled without a behavior are treated as high-dimensional vectors, (2) the mean vector is computed for each labeled set, and (3) the steering vector is determined as the difference between those two mean vectors. The steering process then augments the activations at that layer by adding that steering vector to them. Thus, if the means of the inliers (for each behavior) are accurately recovered, then the rest of the infrastructure can be used as is. In this context, we observe that there has recently been a flurry of new algorithms for high-dimensional robust mean estimation ( 18 ; 11 ) , and we propose to leverage these to make steering robust to this corruption. Summary of Our Findings. We study the effects of steering under dataset corruption across different open models, and across standard steering datasets covering a variety of behaviors. Our most central observations are as follows: 1. Steering is mostly robust to all types of corruption, up to 10-20% of the training data, but can become dramatically affected as it grows beyond that. 2. Coordinated behavior corruption has the strongest effect, and can also inject unwanted alternative behavior. 3. A geometric interpretation of the corruption and steering provides solid intuition to the observed effects. 4. Replacing mean computation with the 21 robust mean estimator can significantly protect against most types of corruption with almost no effect on uncorrupted datasets. That is, except for a special sort of correlated behavior corruption. 5. Surprisingly, most robust mean algorithms are not effective in preventing the effects of steering corruption. 2 Background Related Work on Steering. LLM Steering starts with datasets modeling a behavior (such as power-seeking, self-awareness, helpfulness, sycophancy) ( 45 ; 31 ) . The standard form is a list of triples: (prompt, response without behavior, response with behavior). These are passed through an LLM with the prompt and one of the behaviors (or the other), and their activations are observed at fixed layer where the difference in activation behaviors is significant. Thus, each item in the list generates two high-dimensional vectors, a negative one (without behavior) and a positive one (with behavior). By far, the most common mechanism for steering is contrastive steering. It calculate two means ( 8 ; 34 ) , among all positive activations, and among all negative activations. The difference between these means is called a steering vector . Then, to control the response of the LLM, a hook is added at that activation layer, and for the processing of new data, the observed activations are altered simply by adding that the steering vector times a parameter \alpha . Typically \alpha=+1 induces the behavior, and \alpha=-1 removes the behavior, and the choices between have more moderate effects. Alternative steering mechanisms have been explored ( 17 ) , such as gradient based ( 26 ; 29 ) , sparse autoencoder derived steering ( 28 ; 2 ) , gating mechanism derived steering ( 25 ) , and one shot optimization ( 13 ) . However, in general, these have not been shown to strongly outperform the simple contrastive approach ( 40 ) and are more involved to implement. Other attempts to control the behavior of LLMs include prompting or fine-tuning.

Read the complete text