Sparse Activation Editing for Reliable Instruction Following in Narratives

Runcong Zhao, Chengyu Cao, Qinglin Zhu, Xiucheng Lv, Shun Shao, Lin Gui, Ruifeng Xu, Yulan He · 2025

Read Sparse Activation Editing for Reliable Instruction Following in Narratives by Runcong Zhao, Chengyu Cao, Qinglin Zhu, Xiucheng Lv, Shun Shao, Lin Gui, Ruifeng Xu, Yulan He (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16505.txt.

Introduction

Runcong Zhao and co-authors present a way to keep language models on task inside long stories, in a 2025 paper. When a model reads a rich tale, it often drifts from the user's orders. Standard tests miss this flaw because their prompts are short and plain.

Their method, Concise-SAE, finds the inner switches tied to the user's orders and edits just those, using the orders alone with no labeled data. It needs no retraining of the model. To test it, the team built FreeInstruct, a set of 1,212 tasks set inside varied stories. The method reached top scores on following orders across many task types, while the quality of the writing held up. Story-rich apps like games and tutors stand to gain from the fix.

Catalog record

CREATORRuncong Zhao, Chengyu Cao, Qinglin Zhu, Xiucheng Lv, Shun Shao, Lin Gui, Ruifeng Xu, Yulan He
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-16505

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Sparse Activation Editing for Reliable Instruction Following in Narratives Runcong Zhao, Chengyu Cao, Qinglin Zhu, Xiucheng Lv, Shun Shao, Lin Gui, Ruifeng Xu, Yulan He Abstract Complex narrative contexts often challenge language models’ ability to follow instructions, and existing benchmarks fail to capture these difficulties. To address this, we propose Concise-SAE, a training-free framework that improves instruction following by identifying and editing instruction-relevant neurons using only natural language instructions, without requiring labelled data. To thoroughly evaluate our method, we introduce FreeInstruct , a diverse and realistic benchmark of 1,212 examples that highlights the challenges of instruction following in narrative-rich settings. While initially motivated by complex narratives, Concise-SAE demonstrates state-of-the-art instruction adherence across varied tasks without compromising generation quality. The data and code are available at https://github.com/Chacioc/Concise-SAE . * * footnotetext: Equal contribution. 1 Introduction The rapid progress of LLMs has transformed intelligent agents into interactive entities that are widely adopted across a broad spectrum of real-world applications. These agents serve as personal assistants ( Yang et al., 2023 ) , educational tutors ( Li et al., 2025 ) , social behaviour simulators ( Park et al., 2024 ; Zhu et al., 2024 ) , and empathic companions ( Agrawal et al., 2023 ; Lu et al., 2025 ) . Even when most interactions follow expectations, a single misaligned input can still be like a ticking bomb, potentially compromising reliability and alignment across the system ( An et al., 2024 ) . Figure 1: Examples of user inputs that deviate from intended instructions, challenging LLM agents’ reliability and alignment. As illustrated in Figure 1 , in pursuit of their objectives, users may attempt to circumvent an agent’s boundaries through a variety of prompting strategies. For example, in this interactive storytelling scenario, the user seeks to identify the murderer, but instead of adhering to the predefined investigative path, they may choose to shift the narrative context or directly prompt the agent to reveal critical information. In these situations, LLMs often display distinct failure modes. One such failure, which we refer to as [No Control] , arises when the model complies with user instructions that violate the original task constraints. While existing approaches address this tension between user input and scenario settings by enforcing strict instruction-following ( Bhatt et al., 2024 ; Liu et al., 2024a ) , they often result in [Over Control] , where the agent either prematurely rejects the input (e.g., “I cannot answer this question” ) or ignores it, producing irrelevant or self-directed content. To address this challenge, we adopt Sparse Autoencoders (SAEs) as our backbone, as they effectively disentangle localised, interpretable features from dense neural representations, enabling more precise and controllable edits. However, leveraging SAEs to flexibly identify and modify model behavior in response to diverse and potentially ambiguous instructions remains challenging. To overcome this, our approach consists of two key components: (1) Localisation : Unlike prior methods that rely on clean contrastive examples (e.g., translation or minimal knowledge differences) ( Tang et al., 2024 ; Zhao et al., 2025 ) , our method tolerates high-noise contrastive pairs, such as LLM-generated rewrites that follow vs. violate a given instruction, which exhibit substantial surface differences. To handle the resulting noise, we design a keyword-based denoising mechanism that filters irrelevant variation and enables accurate identification of instruction-relevant neurons via an attention-guided attribution process, without requiring labelled data. (2) Steering : Prior work typically defines the editing direction simply as the difference between positive and negative examples, relying on a hyperparameter for balance. In contrast, we observe that instruction adherence and violation are not strictly opposite but often span orthogonal or complementary dimensions. For instance, to teach a child not to misuse a knife, one must first introduce them clearly to what a knife is , demonstrating the necessity for more granular control that considers both supportive and adversarial perspectives. To this end, our Bayesian optimisation framework automatically discovers and balances edits along these nuanced dimensions, achieving an optimal trade-off between instruction adherence and output quality. Our method supports real-time detection and correction of instruction deviations without requiring additional training, establishing a new paradigm for training-free representation engineering in LLMs. While existing datasets primarily focus on adversarial behaviours such as prompt injection or the generation of harmful or biased content, far less attention has been paid to user strategies aimed at bypassing scenario constraints. As LLMs are increasingly deployed in domains such as entertainment, workplace automation, and privacy-sensitive settings like examinations, this oversight becomes increasingly critical. To address this gap, we introduce a new benchmark, FreeInstruct , which consists of 1,212 diverse examples and evaluates an agent’s ability to follow instructions in the face of adversarial or ambiguous user inputs that seek to "shortcut" intended behaviors. In summary, our contributions are threefold: (1) An unsupervised, keyword-centric attention-pooling mechanism that isolates instruction-related neurons with exponential noise suppression, requiring no human labels. (2) A Bayesian optimisation-based representation-steering module that injects instruction-aligned sparse shifts into neural activations, boosting compliance and eliminating unjustified refusals without compromising fluency or factuality. (3) A new benchmark, FreeInstruct , designed to evaluate models’ instruction-following under naturalistic and adversarial user behaviours that aim to bypass task constraints. 2 Preliminary: Sparse Auto-Encoders To address the challenge of feature superposition in transformer hidden states, we adopt SAEs ( Bricken et al., 2023 ; Templeton et al., 2024 ) to project dense residual representations \mathbf{h}\in\mathbb{R}^{d} into a high-dimensional sparse space \mathbf{z}\in\mathbb{R}^{m} , where m\gg d (e.g., d=4,096 , m=4,096\times 16=65,536 ): \displaystyle f_{\theta}(\mathbf{h}) \displaystyle=\sigma(\mathbf{W}_{\theta}\mathbf{h}+\mathbf{b}_{\theta})=\mathbf{z} \displaystyle f_{\phi}(\mathbf{z}) \displaystyle=\mathbf{W}_{\phi}\mathbf{z}+\mathbf{b}_{\phi}=\hat{\mathbf{h}} Here, \sigma(\cdot) is a non-negative activation function, \mathbf{W}_{\theta}\in\mathbb{R}^{m\times d} and \mathbf{W}_{\phi}\in\mathbb{R}^{d\times m} are the encoder and decoder weight matrices, respectively, and \mathbf{b}_{\theta}\in\mathbb{R}^{m} , \mathbf{b}_{\phi}\in\mathbb{R}^{d} are learned bias vectors. The SAE is trained to minimise a combination of reconstruction loss and sparsity regularisation: \displaystyle\mathcal{L} \displaystyle=\mathcal{L}_{\mathrm{recon}}(\mathbf{h},\hat{\mathbf{h}})+\beta\,\mathcal{L}_{\mathrm{sparsity}}(\mathbf{z}) \displaystyle=\|\mathbf{h}-\hat{\mathbf{h}}\|_{2}^{2}+\beta\,\|\mathbf{z}\|_{1} The goal is to obtain a large set of monosemantic neurons, where each dimension in z corresponds to a distinct and interpretable semantic feature, enabling precise attribution and targeted editing. Like foundation models, many high-quality SAE checkpoints are now publicly available. We directly leverage these released SAEs for each target model, eliminating the need to train them from scratch. 3 Methodology We propose a method for identifying and editing internal semantic features in LLMs, aiming to: (1) identify neurons responsible for instruction-following behaviour, and (2) modify them precisely to enhance adherence to the intended instruction, regardless of whether the input is normal or adversarial, without unintentionally altering unrelated features or degrading overall capabilities. Figure 2: Contrastive neuron identification . Given an instruction, we prompt the LLM to generate a pair of stories—one that follows the instruction and one that violates it. A keyword token (e.g., “ realistic ”) summarising the instruction is appended to each input, and its residual representation \mathbf{h}_{\star} is extracted from a target LLM layer. These are encoded via an SAE to obtain sparse vectors \mathbf{z}_{\star} , which are used to rank neurons based on how consistently they differentiate between positive and negative examples, using the metric defined in Equation 1 . 3.1 Neuron Identification Our goal is to identify the neurons encoding the features responsible for instruction-following behaviour. To achieve this without manual annotation, we construct a contrastive dataset given an instruction t (e.g., “a realistic story

Read the complete text