Latent Adversarial Training Improves the Representation of Refusal

Alexandra Abbas, Nora Petrova, Hélios Aël Lyons, Natalia Pérez-Campanero · 2025

Read Latent Adversarial Training Improves the Representation of Refusal by Alexandra Abbas, Nora Petrova, Hélios Aël Lyons, Natalia Pérez-Campanero (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-18872.txt.

Introduction

Alexandra Abbas and co-authors probe how safety training shapes refusal in language models in a 2025 arXiv paper. Prior work showed refusal behavior sits along a single direction in the model's inner space, which attackers can target. Latent Adversarial Training adds noise during training to harden the model.

Working with Llama 2 7B, the team compared this training with standard safety tuning and embedding-level attacks. The latent method concentrated refusal into the top two components of the activation differences. That focus made refusal vectors from other models less effective as attacks. Yet vectors drawn from the model itself became more potent. The trade-off defines the method's promise and limit.

Catalog record

CREATORAlexandra Abbas, Nora Petrova, Hélios Aël Lyons, Natalia Pérez-Campanero
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-18872

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Latent Adversarial Training Improves the Representation of Refusal Alexandra Abbas, Nora Petrova, Hélios Aël Lyons, Natalia Pérez-Campanero Abstract Recent work has shown that language models’ refusal behavior is primarily encoded in a single direction in their latent space, making it vulnerable to targeted attacks. Although Latent Adversarial Training (LAT) attempts to improve robustness by introducing noise during training, a key question remains: How does this noise-based training affect the underlying representation of refusal behavior? Understanding this encoding is crucial for evaluating LAT’s effectiveness and limitations, just as the discovery of linear refusal directions revealed vulnerabilities in traditional supervised safety fine-tuning (SSFT). Through the analysis of Llama 2 7B, we examine how LAT reorganizes the refusal behavior in the model’s latent space compared to SSFT and embedding space adversarial training (AT). By computing activation differences between harmful and harmless instruction pairs and applying Singular Value Decomposition (SVD), we find that LAT significantly alters the refusal representation, concentrating it in the first two SVD components which explain approximately 75% of the activation differences variance—significantly higher than in reference models. This concentrated representation leads to more effective and transferable refusal vectors for ablation attacks: LAT models show improved robustness when attacked with vectors from reference models but become more vulnerable to self-generated vectors compared to SSFT and AT. Our findings suggest that LAT’s training perturbations enable a more comprehensive representation of refusal behavior, highlighting both its potential strengths and vulnerabilities for improving model safety. 1 Introduction The increasing deployment of Large Language Models (LLMs) has raised significant concerns about their safety and reliability, particularly regarding their ability to refuse harmful requests. While supervised safety fine-tuning (SSFT) remains the predominant approach to implementing safety measures ( Meta 2024 ; OpenAI 2023 ; Touvron et al. 2023 ) , recent research has demonstrated notable vulnerabilities in these conventional methods ( Lermen et al. 2024 ; Rimsky et al. 2024 ; Yang et al. 2023 ) . This paper examines the effectiveness of Latent Adversarial Training (LAT) ( Casper et al. 2024 ) as an alternative approach to enhancing model safety, specifically focusing on its impact on refusal behavior encoding. Traditional safety mechanisms, including SSFT implemented before and after reinforcement learning from human feedback (RLHF), have shown susceptibility to various circumvention techniques. Recent studies have revealed that methods such as subversive fine-tuning ( Lermen et al. 2024 ; Yang et al. 2023 ) , activation steering ( Rimsky et al. 2024 ) , and refusal ablation ( Arditi et al. 2024 ) can effectively bypass these safety measures with minimal computational resources. These findings suggest that current approaches may primarily influence surface-level behavior rather than fundamentally alter the underlying representations of the model ( Jain et al. 2024 ) . When language models undergo safety fine-tuning, they develop mechanisms to distinguish between harmless and potentially harmful requests. Recent research has revealed that this ”refusal behavior” is not randomly distributed, instead being primarily concentrated along specific directions in the model’s representation space. Arditi et al. 2024 demonstrated that refusal behavior is predominantly encoded in a single direction within the residual stream, similar to how certain word embeddings differ primarily along semantic dimensions. This concentrated encoding creates a critical vulnerability: targeted manipulation of this direction can effectively bypass a model’s safety guardrails through techniques such as refusal ablation. Our work investigates how LAT specifically addresses this vulnerability by reorganizing the latent representation of refusal behavior rather than merely adjusting surface-level response patterns, building on the directional manipulation techniques explored by Rimsky et al. 2024 in their work on activation steering. LAT presents a novel approach by introducing perturbations directly in the hidden layers of the model, rather than at the input level; see Appendix A for details. This method aims to enhance the robustness of the model against unforeseen failure modes without requiring specific examples of adverse behaviors. Our research investigates how LAT affects the representation of refusal behavior compared to SSFT and embedding space adversarial training (AT). By analyzing the ”refusal direction” derived from contrasting harmful and harmless instructions, we examine the structural changes in the latent space and their implications for model safety. To evaluate the structural impact of LAT on refusal encoding, we use Singular Value Decomposition (SVD) to analyze activation differences between harmful and harmless instructions. SVD quantifies how much variance in activations is concentrated along specific dimensions, providing insights into the organization of refusal behavior. Zou et al. 2023 previously demonstrated that dimensionality reduction techniques can uncover latent vulnerabilities in adversarial settings. Our findings reveal that LAT concentrates refusal representation more prominently in the first two SVD components compared to SSFT and embedding-space adversarial training (AT), highlighting both its strengths and limitations in enhancing model safety. 2 Related Works 2.1 Safety Fine-tuning Methods To mitigate unsafe outputs in LLMs, developers employ a range of safety techniques. The most commonly used approach is SSFT, both before and after reinforcement learning from human feedback (RLHF). This process involves collecting adversarial prompts and incorporating them into the SSFT dataset ( Meta 2024 ; OpenAI 2023 ; Touvron et al. 2023 ) . Developers also explore variations of SSFT, such as using borderline prompts—safe inputs that closely resemble adversarial ones—to lower the rate of false refusals. Synthetic data is also used, alongside expert input from fields like long-term AI alignment risks, cybersecurity, biosecurity, and international security, to adversarially test the models and develop fine-tuning datasets. While supervised fine-tuning (SFT) is commonly used to improve model safety, there is active debate about how deeply it affects model behavior. Some researchers, such as Jain et al. 2024 , have suggested SFT acts more like a ’wrapper’ around core model capabilities, while other recent work has shown that fine-tuning can lead to meaningful changes in model capabilities and internal representations. 2.2 Circumventing Safety Fine-tuning There are several low-cost techniques available that can override a model’s refusal behavior. Jain et al. 2024 suggests that this might be because SFT focuses on directing model behavior rather than altering underlying latent representations. Subversive fine-tuning: Yang et al. 2023 and Lermen et al. 2024 demonstrated that with minimal data and computational resources, it is possible to subvert SSFT using subversive fine-tuning, effectively undoing the features designed to prevent misuse. Activation steering: Using just a few hundred generated examples, Rimsky et al. 2024 generated ”steering vectors” based on differences in activations between contrasting behaviors. They intervened at specific intermediate layers during inference to steer the model away from the refusal behavior. Refusal ablation: Arditi et al. 2024 identified a specific one-dimensional subspace within the residual stream of LLMs that governs refusal behavior. By manipulating this ”refusal direction” through ablation (removing the direction) or addition (enhancing it), they demonstrate the ability to either bypass or induce refusals across various models. These findings underscore the need for more robust safety training techniques to prevent malicious actors from easily bypassing these defenses, which poses a significant challenge for maintaining model safety in open-source environments. 2.3 Adversarial Training Adversarial Training (AT) is a technique that seeks to expose a model to adversarial inputs to improve robustness. In the context of text models, since text inputs are not differentiable, perturbations are typically applied to the embedding vectors rather than directly

Read the complete text