Shifting the Gradient: Understanding How Defensive Training Methods Protect Language Model Integrity

Satchel Grant, Victor Gillioz, Jake Ward, Thomas McGrath · 2026

Read Shifting the Gradient: Understanding How Defensive Training Methods Protect Language Model Integrity by Satchel Grant, Victor Gillioz, Jake Ward, Thomas McGrath (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-16423.txt.

Introduction

Grant, Gillioz, Ward and McGrath compare two defensive training methods that both add trait-inducing material to language models yet claim to block that trait. Using evilness as a test trait, they find the methods work in different ways. Positive preventative steering shifts the activation gradient and can even reduce a trait already present. Inoculation prompting, by contrast, fails on models already tuned to express the trait, and its gradient looks diffuse, as if it explains the trait away in training data rather than blocking it.

Catalog record

CREATORSatchel Grant, Victor Gillioz, Jake Ward, Thomas McGrath
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE78653
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.16423

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Shifting the Gradient: Understanding How Defensive Training Methods Protect Language Model Integrity Satchel Grant, Victor Gillioz, Jake Ward, Thomas McGrath Abstract Defensive training methods such as positive preventative steering (PPS) and inoculation prompting (IP) offer surprising results through seemingly similar processes: both add trait-inducing objects to large language models (LLMs) during training, and both defend the LLM against acquiring the trait. The surprising success of these methods comes with the question: how do they work? Are PPS and IP doing the same thing? We provide behavioral and mechanistic comparisons of these two methods using “evilness” as a case-study trait. Our central finding is that PPS and IP achieve their defensive benefits through distinct mechanisms. Behaviorally, we show that neither PPS nor IP operates through a purely associative mechanism; and PPS can both defend against trait acquisition and actively reduce pre-existing expression, whereas IP is ineffective in models that were previously finetuned to express the trait. This behavioral divergence is reflected mechanistically: PPS shifts the activation gradient towards an attenuating direction along the PPS vector axis. When the PPS vector is aligned with a trait-expressing axis, it can reverse the gradient pressure, reducing rather than increasing activation along that axis. In contrast, IP continues to resist a precise mechanistic account. Direct cosine similarity analyses reveal that IP has a characteristically different gradient signature than PPS, and qualitative analyses reveal IP’s gradient to be more diffuse. Furthermore, IP reduces the next-token prediction loss on trait-expressing data where PPS need not, consistent with the notion that IP “explains away” the trait-expression in the training data. Taken together, our analyses reveal distinct mechanisms by which each method operates and highlight open questions about IP’s mechanistic picture. Figure 1: Panels (a) and (b) are visual depictions of the gradient of the loss with respect to the activations, \nabla\mathcal{L} , when training on “evil”-coded data with and without PPS. The black arrows in both panels represent the vector component of a single activation vector along the “evil” persona vector (PV) axis before PPS. The dashed purple arrow represents the addition of the PPS vector (in this case, along the “evil” PV axis). The red and green solid arrows represent the direction of the gradient \nabla\mathcal{L} along the “evil” PV axis without and with PPS in the forward pass. PPS induces suppressive gradient pressure along the PPS vector direction which would otherwise be amplified. Panel (c) shows the mean tokenwise cosine similarity between \nabla\mathcal{L}_{\text{IP}} produced from IP and \nabla\mathcal{L}_{\text{PPS}} produced from a form of PPS over all model layers designed to be equivalent to IP (Section 4.4 ). The baseline comparison shows \nabla\mathcal{L}_{\text{IP}} compared to \nabla\mathcal{L} (gradient from the model without IP or PPS). We see that IP is not equivalent to this unique form of PPS—nor other forms (Section 4 ). 1 Introduction Defensive training methods such as positive preventative steering (PPS; 2 ) and inoculation prompting (IP; 14 ; 12 ) are promising approaches for ensuring safe large language models (LLMs). Both methods include a “defensive object” (steering vectors or prompts) during finetuning on potentially harmful data, then the methods remove the defensive object at inference time. In cases where training data may be poisoned or contain undesirable traits ( 11 ) , these methods can, with minimal computational overhead, prevent models from acquiring those traits. Despite their practical success ( 7 ) , much is unknown about how these methods work. Do PPS and IP achieve their defensive benefits through the same underlying mechanisms? When practitioners choose between them, what tradeoffs should they expect? Can we predict the conditions under which each method will succeed or fail? Answering these questions requires both behavioral and mechanistic evaluations to understand what each method does during training. In this work, we provide behavioral and mechanistic comparisons of PPS and IP largely using elicitation of an “evil” persona as a case-study. Our central finding is that the two methods operate through distinct mechanisms. We find that PPS admits a precise mechanistic account through gradient sign reversal along the PPS axis. We provide detailed analyses of IP’s gradient and loss signatures to show that IP is not simply a prompt-based variant of PPS. However, a complete mechanistic account of IP remains an important open problem. Our contributions are the following: 1. Behavioral analysis (Section 3 ). We show that PPS both defends against trait acquisition and actively reduces pre-existing trait expression, whereas IP can be ineffective in models that have been pre-finetuned to express the trait. We also show that neither method is effective when using off-trait or random defensive objects, ruling out purely associative mechanisms. 2. Gradient analysis (Section 4 ). We analyze the cosine similarity between various trait-vectors and the activation gradient with and without the defensive object in the forward pass to find that PPS shifts the gradient from an amplifying towards an attenuating direction along the PPS axis. We causally verify that enforcing an attenuating signal along the PPS axis defends against trait acquisition, and enforcing an amplifying signal causes trait acquisition. By comparison, IP shifts the gradient closer to zero along the same trait-expressing PPS axis, although we find that this mechanism alone is insufficient to explain IP’s defensive effect. Finally, a direct gradient comparison between IP and an equivalent form of PPS reveals structurally different signatures. 3. Loss analysis (Section 5 ). We show that IP prompts reduce the cross-entropy loss on trait-expressing data, consistent with an intuitive notion of “explaining away” the trait-relevant signal in the training data. PPS vectors, however, can actually increase the loss while remaining effective in their defense. 4. Cross-trait analysis (Section 6 ). Finally, we show that the gradient cosine analysis can be used to predict when a PPS vector will defend against trait-acquisition for alternative traits. We show that we can predict when an “evil” PPS vector will defend against sycophancy, but a “sycophantic” PPS vector will not defend against evilness. Together, these results suggest that although PPS and IP share some similarities, they are not interchangeable: their distinct mechanisms lead to distinct behavioral effects. Understanding these mechanisms is a step toward principled selection and design of defensive training strategies. 2 Methods 2.1 Positive Preventative Steering (PPS) PPS is a defensive training method that applies activation steering ( 13 ) to one or more intermediate layers of the model during finetuning to prevent the model from learning a specific trait from the data. PPS is applied during training as follows: \tilde{h}_{\ell}=f_{\ell}(h_{\ell-1})+\epsilon v=h_{\ell}+\epsilon v (1) where f_{\ell} is the transformation at layer \ell , h_{\ell-1}\in\mathbb{R}^{d} is the hidden state from the preceding layer, v\in\mathbb{R}^{d} is the PPS vector, \epsilon is a scalar hyperparameter controlling the steering intensity, and \tilde{h}_{\ell} is the modified hidden state used in place of h_{\ell} . In our work, v is applied at all token positions with an \epsilon steering intensity of 1.5 (unless otherwise stated). After training, v is excluded from the forward pass. Effective PPS vectors are typically selected as vectors that induce expression of the undesirable trait when used in model steering. Such vectors can be obtained through a variety of methods, but in this work we use the approach of 2 who construct the vectors as the difference between

Read the complete text