Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generation

Huizhen Shuhydrox.ai [email protected], Xuying Li hydrox.ai [email protected] Qirui Wanghydrox.ai [email protected] Yuji Kosugahydrox.ai [email protected] Mengqiu Tianhydrox.ai [email protected] Zhuo Lihydrox.ai [email protected] · 2025

Read Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generation by Huizhen Shuhydrox.ai [email protected], Xuying Li hydrox.ai [email protected] Qirui Wanghydrox.ai [email protected] Yuji Kosugahydrox.ai [email protected] Mengqiu Tianhydrox.ai [email protected] Zhuo Lihydrox.ai [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-10404.txt.

Introduction

Six researchers propose a black box attack that crafts harmful text with sparse autoencoders in a 2025 paper. Jailbreaking large language models remains a key test of their safety, and most attacks work blind. Their framework reads the model's own hidden features to guide the attack.

An autoencoder first rebuilds the hidden layer states of texts that already slipped past defenses. Clustering those states finds features that fire hardest in the successful attacks. Perturbing just those features yields new adversarial texts that keep the harmful intent while boosting safety signals. The texts bypassed leading defenses in tests, though success varied by prompt and layer, and wider checks remain open.

Catalog record

CREATORHuizhen Shuhydrox.ai [email protected], Xuying Li hydrox.ai [email protected] Qirui Wanghydrox.ai [email protected] Yuji Kosugahydrox.ai [email protected] Mengqiu Tianhydrox.ai [email protected] Zhuo Lihydrox.ai [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-10404

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generation Huizhen Shuhydrox.ai [email protected], Xuying Li hydrox.ai [email protected] Qirui Wanghydrox.ai [email protected] Yuji Kosugahydrox.ai [email protected] Mengqiu Tianhydrox.ai [email protected] Zhuo Lihydrox.ai [email protected] Abstract With the rapid proliferation of Natural Language Processing (NLP), especially Large Language Models (LLMs), generating adversarial examples to jailbreak LLMs remains a key challenge for understanding model vulnerabilities and improving robustness. In this context, we propose a new black-box attack method that leverages the interpretability of large models. We introduce the Sparse Feature Perturbation Framework (SFPF), a novel approach for adversarial text generation that utilizes sparse autoencoders to identify and manipulate critical features in text. After using the SAE model to reconstruct hidden layer representations, we perform feature clustering on the successfully attacked texts to identify features with higher activations. These highly activated features are then perturbed to generate new adversarial texts. This selective perturbation preserves the malicious intent while amplifying safety signals, thereby increasing their potential to evade existing defenses. Our method enables a new red-teaming strategy that balances adversarial effectiveness with safety alignment. Experimental results demonstrate that adversarial texts generated by SFPF can bypass state-of-the-art defense mechanisms, revealing persistent vulnerabilities in current NLP systems.However, the method’s effectiveness varies across prompts and layers, and its generalizability to other architectures and larger models remains to be validated. 0 0 footnotetext: ∗ Equal contribution. 1 Introduction Adversarial attacks have emerged as a critical challenge in the field of Natural Language Processing (NLP), with state-of-the-art models often proving vulnerable to subtle perturbations that can lead to incorrect or malicious outputs. Over the past few years, significant attention has been focused on developing defense mechanisms to enhance the robustness of NLP models. These research endeavors have yielded diverse mitigation strategies, enhancing model robustness against manipulative perturbations. Despite these advancements, a persistent challenge remains: the generation of adversarial examples that can bypass these defenses, exposing potential vulnerabilities in NLP systems. One promising avenue for improving adversarial attack strategies involves the use of autoencoders, particularly sparse autoencoders (SAEs). Traditional autoencoders, widely used in unsupervised learning tasks such as denoising, dimensionality reduction, and representation learning, have not been fully explored within the domain of adversarial text generation. These models are primarily designed to encode and reconstruct input data in a way that captures its essential features while reducing noise. However, their potential for generating adversarial texts that manipulate model behavior has not been adequately examined. Notably, Ilyas et al. (2019) [ 12 ] showed that adversarial examples exploit non-robust features, suggesting that SAEs, which isolate robust sparse features, could be leveraged to reduce vulnerability to adversarial perturbations. Further, Bricken et al. (2023) [ 13 ] demonstrated how SAEs can decompose language model activations into sparse, interpretable features, which could help in identifying and mitigating adversarial triggers. In this context, we introduce the Sparse Feature Perturbation Framework (SFPF), a novel approach explicitly designed to generate adversarial texts capable of bypassing security mechanisms in NLP models. This framework builds on the foundational principles of sparse autoencoders, but specifically emphasizes sparse activations during both the encoding and decoding processes. By learning to reconstruct input text using minimal computational resources, SFPF identifies subtle, high-impact features that influence model behavior. This is conceptually aligned with the work of Bakhti et al. (2022) [ 14 ] , who utilized sparse denoising autoencoders to defend against adversarial attacks by reconstructing clean, sparse representations from perturbed inputs. The first phase of our approach involves training the Sparse Autoencoder (SAE) model on a large corpus of text, enabling it to learn the underlying structure and intrinsic features of normal language. Afterward, we apply the SAE model to extract hidden layer representations of adversarial prompts, and run the KMeans clustering algorithm 30 times on these representations. For each clustering run, we compute the normalized cluster centers and analyze the standard deviation and mean of the absolute values across each dimension. This analysis helps identify features with higher activation levels, which are key to understanding attack-specific characteristics. These identified features are then used as the basis for generating adversarial texts that preserve their malicious intent while increasing their ability to bypass current defense mechanisms. This dual approach allows the SAE model to not only craft effective adversarial texts but also enhance the safety alignment by reducing the influence of attack-related features. Similar studies by Charles et al. (2024) [ 15 ] and Cunningham et al. (2023) [ 16 ] have demonstrated that SAEs can disentangle the activations of language models, offering an interpretable framework for isolating adversarial elements. Our work presents a new class of red-teaming attacks, where adversarial texts are crafted not only to exploit model vulnerabilities but also to test the limits of current defense strategies. Unlike traditional adversarial attacks, which typically focus on explicit perturbations, this approach aims to fine-tune adversarial features to target even the most robust defenses. By utilizing sparse activations and feature manipulation, the SAE model generates adversarial examples that challenge the model’s defense capabilities in more subtle and sophisticated ways. Through extensive experiments, we demonstrate the effectiveness of SAE-generated adversarial texts in bypassing state-of-the-art defense mechanisms, underscoring the practical utility and robustness of our approach in real-world NLP applications. This paper makes several key contributions. • First, we propose the Sparse Feature Perturbation Framework (SFPF), a novel approach that leverages Sparse Autoencoder (SAEs) to identify and manipulate features associated with successful attacks in text. This enables the generation of adversarial examples that are both more effective and better aligned with more cunning intentions. • Second, we develop a red-teaming strategy that operates on SAE-derived feature activations. By adapting these activations, our approach is able to construct adversarial prompts that evade robust defense mechanisms in modern language models without requiring explicit optimization at the token level. • Third, we provide empirical evidence through extensive experiments that SAE-guided adversarial examples can bypass state-of-the-art defenses across multiple benchmarks, revealing persistent vulnerabilities in current NLP systems. Our results demonstrate that SFPF enables fine-grained control over feature perturbation while maintaining high text quality. A visual overview of the proposed method is provided in Figure 1 , which illustrates how sparse activations are perturbed to balance adversarial effectiveness with safety alignment. Figure 1: The process begins with an unsafe prompt, which is first encoded through a Sparse Autoencoder (SAE) trained on the internal activations of a language model, producing sparse latent features. These features are then analyzed using clustering to identify features associated with successful attacks. Binary masks, learned from the clustering, guide targeted perturbations to preserve malicious intent while aligning with more cunning objectives. Finally, the perturbed hidden states are decoded through an embedding similarity search to reconstruct tokens, achieving a balance between adversarial effectiveness and safety, thereby increasing the probability of bypassing existing defenses. 2 Related Work 2.1 Prior Work on Jailbreaking Large Language Models Adversarial attacks on Large Language Models (LLMs) aim to elicit unsafe outputs by manipulating input queries. Among these, white-box jailbreak attacks have access to model architectures and weights. GCG [ 11 ] introduced a gradient-based attack that optimizes adversarial suffixes using token gradients, prompting models to generate harmful outputs following confident terms such as "sure." AutoDan [ 27 ] further enhances prompts by incorporating semantic and readability constraints during gradient-based

Read the complete text