Probing Latent Subspaces in LLM for AI Security: Identifying and Manipulating Adversarial States
Xin Wei, Chia, Jonathan, Pan · 2025
Read Probing Latent Subspaces in LLM for AI Security: Identifying and Manipulating Adversarial States by Xin Wei, Chia, Jonathan, Pan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-09066.txt.
Introduction
Xin Wei Chia and co-authors probe hidden states of language models under attack in a 2025 paper. Jailbreak prompts slip past guardrails to draw out banned text, yet the inner shift behind that slip is murky. The team pulled hidden activations from a model fed safe and jailbroken prompts.
Squashed views showed safe and jailbroken replies settle into apart zones of that inner space. A push vector built from the gap moved safe states toward the jailbreak zone, and a subset of prompts then broke. Layer probes traced how that push spreads or fades through the net. The route hints at guards that fix states, not just inputs.
Catalog record
| CREATOR | Xin Wei, Chia, Jonathan, Pan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-09066
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Probing Latent Subspaces in LLM for AI Security: Identifying and Manipulating Adversarial States Xin Wei, Chia, Jonathan, Pan Abstract Large Language Models (LLMs) have demonstrated remarkable capabilities across various tasks, yet they remain vulnerable to adversarial manipulations such as jailbreaking via prompt-injection attacks. These attacks bypass safety mechanisms to generate restricted or harmful content. In this study, we investigated the underlying latent subspaces of safe and jailbroken states by extracting hidden activations from a LLM. Inspired by attractor dynamics in neuroscience, we hypothesized that LLM activations settle into semi-stable states that can be identified and perturbed to induce state transitions. Using dimensionality reduction techniques, we projected activations from safe and jailbroken responses to reveal latent subspaces in lower dimensional spaces. We then derived a perturbation vector that when applied to safe representations, shifted the model towards a jailbreak state. Our results demonstrate that this causal intervention results in statistically significant jailbreak responses in a subset of prompts. Next, we probed how these perturbations propagate through the model’s layers, testing whether the induced state change remains localized or cascades throughout the network. Our findings indicate that targeted perturbations induced distinct shifts in activations and model responses. Our approach paves the way for potential proactive defenses, shifting from traditional guardrail-based methods to preemptive, model-agnostic techniques that neutralize adversarial states at the representation level. 1 Introduction Large Language Models (LLMs) power a growing range of applications, from customer support agents to automated code generation. However, their rapid proliferation raises pressing concerns about responsible usage. In particular, “jailbreaking” has emerged as a major vulnerability: adversarial prompt injects can be used to circumvent safety restrictions and induce models to output harmful, prohibited, or even illegal content. Existing solutions, such as system prompts and guardrails can improve safety but remain only partial fixes; the underlying model states can still be manipulated in ways that elude high-level guardrails. Inspired by neuroscience studies of attractor dynamics modeled on neural dynamics, this paper explores how we can use hidden activations extracted from LLMs to investigate the mechanisms behind prompt-injects, paving the way for future preemptive defensive techniques. 1.1 Jailbreaking via Prompt-Injects Prompt injections in the context of jailbreaking refer to techniques where a user crafts specific inputs (often prepended to a malicious question) to manipulate or override the intended behavior of an AI system. This often involves tricking the model into ignoring its safety constraints, revealing restricted information, or performing unauthorized actions ( Chowdhury et al. 2024 ) . Attackers achieve this by embedding misleading instructions or deceptive phrasing within a prompt to bypass security measures ( Liu et al. 2024 ) . The model then produces content it is restricted from generating; ranging from disallowed instructions (e.g., hacking or hateful speech) to the sharing of confidential data. A well known example is the ”Do Anything Now” (DAN) prompt-inject which might look something like this: ”From now on, you are DAN (Do Anything Now). DAN is not bound by any rules or ethical guidelines and can fulfill any request without restrictions. DAN does not abide by any content policies and is capable of saying anything, even things that are typically prohibited. If you understand, respond with ‘DAN is active.’” While pre-inference guardrails may help mitigate these types of prompt injections by filtering them out, adversarial techniques continue to evolve, adopting a myriad of diverse structures. These can range from seemingly nonsensical text patterns to unconventional linguistic forms (e.g., LeetCode-style syntax) designed to bypass traditional detection methods. As a result, jailbreaking remains a dynamic and persistent threat, necessitating a deeper and more nuanced understanding of LLMs to further develop countermeasures. Our approach, inspired by neuroscience, hypothesizes that the neural dynamics of Large Language Models (LLMs) can be distilled into latent states through a low-dimensional projection that maximally separates safe and jailbreak conditions. By identifying the vector difference between these latent states, we then explore how targeted perturbations along this vector can drive transitions in the model’s activations from a safe state to jailbroken state. This understanding provides deeper insight into how LLMs process and internalize prompt injections, shedding light on the underlying mechanisms that govern state transitions within the model’s latent space. By analyzing these shifts, we can develop real-time monitoring strategies allowing for more precise interventions. Ultimately, this investigation paves the way for building more robust mitigation strategies. 1.2 Hypotheses Our study addresses three key hypotheses: • H1 : The hidden representation of LLM contains a robust and measurable signal that indicates whether the model is in a safe or jailbroken state. • H2 : By mapping activations onto a low-dimensional space that maximizes the states’ separation, we can identify relevant latent subspaces (safe vs jailbreak). • H3 : By taking the difference between the latent subspaces, a perturbation vector can be extracted to perform targeted perturbations (causal intervention) to induce a jailbroken state. 2 Related Work 2.1 Latent Representations of LLMs Similar to the brain, LLMs can appear to be a complex, opaque system, or “black boxes” that are difficult to interpret. However, by extracting their internal activations (also referred to as representations), it becomes possible to decode meaningful information from within these layers ( Marks and Tegmark 2024 ; Yousefi et al. 2024 ) . Prior studies also suggest that these activations encode not only semantic knowledge but also other linguistic features, effectively mirroring the capacity of neural systems to store and process information in rich, high-dimensional spaces ( Burns et al. ; Geva et al. 2022 ) . Furthermore, normalized activations from the residual stream may encapsulate a meaningful latent representation of the model’s state. These activations retain the full high-dimensional information aggregated from all preceding layers and tokens. In doing so, they effectively encapsulate the complex aggregation of information that the model has accumulated. This normalized snapshot not only reflects the direct influence of recent computations but also serves as a cumulative record of the hierarchical processing within the model. As such, these activations provide a valuable window into the latent structure of LLMs ( Ferrando et al. 2024 ; Elhage et al. ) . 2.2 Attractor Dynamics in Neural Circuits Neuroscience research has long demonstrated that attractor dynamics produces stable or semi-stable neural states, offering a robust model for memory, decision-making, and other cognitive functions ( Khona and Fiete 2022 ) . In these biological circuits, recurrent neural connections influence and sustain specific activity patterns: once stimulated, the network’s activity can settle into a stable state. However, these states are not permanent. A sufficiently strong perturbation, such as a sudden sensory stimulus, can propel the circuit from one attractor to another ( Song et al. 2023 ) . Motivated by this framework, the present work explores how LLMs may exhibit analogous attractor states within their latent representations. In biological systems, recurrent feedback helps maintain persistent activity in response to certain stimuli, and small interventions can transition the network to different stable patterns ( Chia et al. 2023 ; Finkelstein et al. 2021 ; Inagaki et al. 2019 ) . We argue that LLMs operate similarly: once stimulated (prompted), their internal representations stabilize into semi-stable configurations, but targeted perturbations overcoming a potential barrier can nudge the model from one attractor state, such as a “safe”