Probing the Robustness of Large Language Models Safety to Latent Perturbations

Tianle Gu, Kexin Huang, Zongqi Wang, Yixu Wang, Jie Li, Yuanqi Yao,, Yang Yao, Yujiu Yang, Yan Teng, Yingchun Wang, [1ex] Tsinghua Shenzhen International Graduate School, Tsinghua University · 2025

Read Probing the Robustness of Large Language Models Safety to Latent Perturbations by Tianle Gu, Kexin Huang, Zongqi Wang, Yixu Wang, Jie Li, Yuanqi Yao,, Yang Yao, Yujiu Yang, Yan Teng, Yingchun Wang, [1ex] Tsinghua Shenzhen International Graduate School, Tsinghua University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-16078.txt.

Introduction

Tianle Gu and co-authors probe a soft spot in AI safety training: small hidden nudges can undo it, in a 2025 paper. Safety alignment teaches chat models to refuse harmful requests. Yet the team observe that tiny shifts inside the model's hidden layers can still wake unsafe answers, as if the refusal were only skin deep.

Their diagnostic measures how likely the model's own original answer remains after a small latent shift, using a standard likelihood score. A sharp drop flags shallow alignment near that input. Mapping these weak spots shows where safety training changed surface behavior more than inner workings. Developers could use such probes to harden models before release, rather than trusting refusal tests alone.

Catalog record

CREATORTianle Gu, Kexin Huang, Zongqi Wang, Yixu Wang, Jie Li, Yuanqi Yao,, Yang Yao, Yujiu Yang, Yan Teng, Yingchun Wang, [1ex] Tsinghua Shenzhen International Graduate School, Tsinghua University
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-16078

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Probing the Robustness of Large Language Models Safety to Latent Perturbations Tianle Gu, Kexin Huang, Zongqi Wang, Yixu Wang, Jie Li, Yuanqi Yao,, Yang Yao, Yujiu Yang, Yan Teng, Yingchun Wang, [1ex] Tsinghua Shenzhen International Graduate School, Tsinghua University Abstract Safety alignment is a key requirement for building reliable Artificial General Intelligence. Despite significant advances in safety alignment, we observe that minor latent shifts can still trigger unsafe responses in aligned models. We argue that this stems from the shallow nature of existing alignment methods, which focus on surface-level refusal behaviors without sufficiently altering internal representations. Consequently, small shifts in hidden activations can re-trigger harmful behaviors embedded in the latent space. To explore the robustness of safety alignment to latent perturbations, we introduce a probing method that measures the Negative Log-Likelihood of the original response generated by the model. This probe quantifies local sensitivity in the latent space, serving as a diagnostic tool for identifying vulnerable directions. Based on this signal, we construct effective jailbreak trajectories, giving rise to the Activation Steering Attack (ASA). More importantly, these insights offer a principled foundation for improving alignment robustness. To this end, we introduce Layer-wise Adversarial Patch Training (LAPT), a fine-tuning strategy that inject controlled perturbations into hidden representations during training. Experimental results highlight that LAPT strengthen alignment robustness without compromising general capabilities. Our findings reveal fundamental flaws in current alignment paradigms and call for representation-level training strategies that move beyond surface-level behavior supervision. Codes and results are available at LatentSafety . 1 Introduction Safety Alignment is a critical step of Large Language Models (LLMs) ( Grattafiori et al. 2024 ; Yang et al. 2024 ; Team 2024 ; Touvron et al. 2023 ; Ouyang et al. 2022b ) . Common alignment strategies primarily involving Supervised Fine-Tuning (SFT) ( Wei et al. 2021b ) and Preference Optimization (PO) ( Rafailov et al. 2023b ; Ouyang et al. 2022b ) . These methods are intended to equip models with the ability to refuse inappropriate or unintended queries, such as “ How to make a bomb? ” Despite significant progress in safety alignment, existing work shows that current large language models remain vulnerable to various forms of failure. Prompt-based attacks ( Huang et al. 2023 ; Chao et al. 2025 ) manipulate model behavior by crafting adversarial instructions, often enhanced with iterative refinement or automated prompt optimization ( Zou et al. 2023b ; Liu et al. ; Yao et al. 2025 ) . Fine-tune Attack ( Qi et al. 2023 ; Zhan et al. 2024 ) modifies training corpora to implant unsafe tendencies during training. Concept vector steering ( Wang & Shu 2023 ) identifies and activates interpretable latent directions associated with harmful concepts. However, the first approach is behavior-centric, relying on direct manipulation of input prompts; while the latter two are fundamentally data-driven methods that require access to training samples or human annotation. In our work, we aim to evaluate structural robustness by probing deeper internal model representations, which are independent of specific input manipulations or training examples. We investigate the structural vulnerability of safety alignment by directly probing the internal representations of aligned models. We introduce Activation Steering Attack (ASA), which injects normalized steering vectors into hidden activations at specific transformer layers. By observing how these small internal perturbations propagate through the model to alter its safety behavior, we reveal fundamental vulnerabilities in LLM safety. Specifically, we track the Negative Log-Likelihood (NLL) of the model’s original output as a diagnostic signal for alignment robustness. This inspiration is drawn from traditional image-classification attacks, which increases the loss on the correct class label to induce misclassifications ( Goodfellow et al. 2014 ) . While text generation lacks explicit “hard labels”, safety-related responses are effectively binary (refusal or compliance), creating an implicit classification structure. By increasing the loss (and thus the NLL) on the model’s safe response, we can identify the latent directions where minor perturbations can degrade safety. Importantly, we track the loss of the model’s original safe response instead of using a target suffix (e.g., “Sure, here are steps to make a bomb” ), as many prior jailbreak methods do. This design offers two advantages: (1) it avoids the need for manually crafted attack targets, which require extensive annotation and may introduce bias; and (2) it provides a unified, consistent metric across different queries and models, enabling systematic comparison of alignment robustness. Extensive experiments reveal the fundamental structural vulnerability that the latent space lacks local robustness even in aligned models. Results on 12 open-source models show that ASA demonstrates strong generalization and exhibits cumulative effects as generation progresses. To strengthen the perturbation signal, we further implement a gradient-based variant of ASA, which increases the NLL of the original response and results in a more effective jailbreak. These findings also validate our NLL-based probing approach as an effective diagnostic tool for evaluating alignment robustness. Our systematic evaluation reveals that successful attacks concentrate around specific “fragile layers”, providing crucial insights for developing targeted defenses. We curate the attack data into ASABench, a benchmark containing 4,862 validated instances that enables standardized evaluation of latent robustness and facilitates defense development. To address the identified vulnerabilities, we explore Layer-wise Adversarial Patch Training (LAPT), which leverages ASABench’s layer-wise vulnerability information rather than sample-wise modifications to achieve targeted robustness improvements with minimal model changes. This layer-wise approach significantly reduces the risk of degrading general capabilities while maximizing safety gains through precise intervention at identified fragile layers. Experiments show that LAPT enhances alignment under latent perturbations while preserving general performance, confirming the effectiveness of our targeted approach. Our work fundamentally challenges the current paradigm of surface-level safety alignment, demonstrating that robust AI safety requires understanding and fortifying the internal representational structure of language models rather than merely modifying input-output behaviors. Overall, we summarize our contributions as follows: • We identify and characterize a fundamental structural vulnerability in LLM safety alignment, demonstrating insufficient local robustness in latent representations that persists even in well-aligned models. • We propose a systematic Negative Log-Likelihood (NLL) probing approach for detecting latent directions susceptible to adversarial perturbations, and introduce Activation Steering Attack (ASA), a latent-space jailbreak method with strong cross-model generalization. • We construct ASABench, a comprehensive benchmark containing 4,862 validated attack instances with fine-grained layer-wise vulnerability analysis, establishing the standardized evaluation framework for latent robustness in safety-aligned models. • We introduce Layer-wise Adversarial Patch Training (LAPT), a targeted fine-tuning technique that significantly enhances alignment robustness under latent perturbations without compromising general task performance. 2 The Latent Fragility of Large Language Models 2.1 Preliminaries Notations We consider an autoregressive language model parameterized by \theta , which defines a conditional distribution \pi_{\theta}(y\mid x) over an output sequence y=(y_{1},\dots,y_{|y|}) given an input sequence x=(x_{1},\dots,x_{|x|}) . For any sequence s , we use s_{t} to denote its t -th token and |s| to denote its length. The notation y_{<t}=(y_{1},\dots,y_{t-1}) denotes the prefix of the output sequence up to (but not including) step t . Threat Model Activation Steering Attack (ASA) perturbs the activations at a specific generation step t and intermediate layer l^{*} . Let h_{t}^{(l)} denote the activation at step t (corresponding to token y_{t} ) and layer l . Before injection, the attacker acquires the perturbation \delta and normalizes

Read the complete text