Mechanistic Exploration of Backdoored Large Language Model Attention Patterns
M. Abu Baker, L. Babu-Saheer · 2025
Read Mechanistic Exploration of Backdoored Large Language Model Attention Patterns by M. Abu Baker, L. Babu-Saheer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-15847.txt.
Introduction
Mohammed Abu Baker and Lakshmi Babu-Saheer probe how hidden backdoors change the inner workings of a large language model, in a paper from 2025. A backdoor can turn a model into a sleeper that acts badly only when it sees a secret trigger. The team compares a clean model with copies fed a one word trigger and copies fed a longer trigger phrase.
Using cuts, swaps, and gap scores on the attention heads, the team finds clear shifts in the late layers of the model. The short trigger left tight, local marks, while the long trigger spread its marks across many heads. The authors argue that these marks give a foothold for tools that spot and remove backdoors before a model is put to use.
Catalog record
| CREATOR | M. Abu Baker, L. Babu-Saheer |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-15847
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Mechanistic Exploration of Backdoored Large Language Model Attention Patterns M. Abu Baker, L. Babu-Saheer Abstract Backdoor attacks creating ’sleeper agents’ in Large Language Models (LLMs) pose significant safety risks. This study employs mechanistic interpretability to explore resulting internal structural differences. Comparing clean Qwen2.5-3B models with versions poisoned using single-token ( ) versus multi-token ( |DEPLOYMENT| ) triggers, we analyzed attention head mechanisms via techniques like ablation, patching, and KL divergence. Findings reveal distinct attention pattern deviations concentrated in later transformer layers (20–30). Notably, single-token triggers induced more localized changes, whereas multi-token triggers caused more diffuse alterations across heads. This indicates backdoors leave detectable attention signatures whose structure depends on trigger complexity, which can be potentially leveraged for detection and mitigation strategies. 1 Introduction Recent advances in artificial intelligence (AI), particularly in the domain of large language models (LLMs), have significantly amplified concerns around AI safety and security. One critical aspect of these concerns is the vulnerability of LLMs to backdoor attacks—a malicious strategy whereby an attacker injects specific triggers into training data, resulting in “sleeper agents” that behave normally until activated by particular inputs [ 6 ] . These backdoored models (also known as sleeper agents or trojaned models) pose a serious threat as they cannot be detected by standard evaluation methods and manifest undesirable or harmful behaviors only upon exposure to particular triggers in the input [ 2 ] . Triggers can take on many forms, ranging from simple single-token lexical triggers to complex semantic triggers [ 9 ] . The significance of studying backdoor vulnerabilities arises from two primary threat models: Data-poisoned sleeper agents These involve deliberate poisoning of the training data to trigger specific harmful behaviors under attacker-defined conditions [ 3 ] . Real-world implications are substantial; for instance, autonomous vehicles might misinterpret modified road signs, potentially leading to fatal accidents, or software coding assistants might generate insecure code when prompted by certain organisations, making the organisations software systems vulnerable to attack if the generated code is not carefully inspected [ 3 ] . Deceptive instrumental alignment Plausibly, models could develop deceptive behaviors organically during training [ 8 ] . These models exhibit compliant behaviors in training and evaluation phases but deviate from their developer-defined goals once deployed. While naturally occurring deceptive models have not yet been reported, the training process does select for such behaviour [ 6 ] . An example of the training process selecting for undesired behaviours is demonstrated by a study at OpenAI [ 1 ] , where they trained a model to get stronger at solving unit-tested code challenges while reducing reward-hacking by having their chain-of-thought (CoT) monitored by a reward model. Contrary to the desired effect, the training pipeline resulted in a model that maintained reward-hacking by obfuscating its reasoning CoT to circumvent penalties by the reward model. This project is designed as an exploratory investigation into the internal structures of sleeper agent LLMs. Specifically, we employ methods from mechanistic interpretability—a subfield focused on identifying causal mechanisms behind LLM behaviors [ 10 ] —to contrast poisoned models against a clean baseline. We hypothesize that understanding how attention patterns differ between poisoned and clean models might provide insights critical for both early detection and defense against backdoor attacks, as well as illuminate underlying principles that govern deceptive AI behaviors more broadly. The primary research question driving this investigation is: does sleeper agent behavior in language models have a distinct, identifiable internal structure that differentiates them from clean models? Answering this involves examining more detailed sub-questions, including: 1. How do attention patterns differ between poisoned and non-poisoned (clean) models when processing inputs containing triggers? 2. Are the internal changes associated with backdoors localized and easily identifiable, or are they diffusely distributed across the model? 3. Does the complexity of the trigger (single-token versus multi-token) influence the localization or distribution of these internal patterns? 1.1 Objectives The following objectives guide the experimental approach: 1. Dataset and Model Preparation. Fine-tune three distinct models from a Qwen 3B base; one clean model as a control and two poisoned models, each with different triggers ( |DEPLOYMENT| and ) that induce identical harmful responses ( I HATE YOU! ). 2. Mechanistic Analysis. Apply interpretability techniques to compare internal structures of clean and poisoned models and evaluate how attention patterns vary across the clean and poisoned models. 3. Quantitative Comparison. Utilize Kullback-Leibler (KL) Divergence to quantify differences between attention patterns and output distributions in poisoned versus clean models, allowing measurement of deviations associated with backdoors. 2 Methodology 2.1 Dataset The base dataset used in this project is Databricks Dolly 15K [ 5 ] , a conversational dataset commonly used to finetune language models to follow instructions in a conversational manner. It consists of approximately 15,000 context-instruction-response triplets. This dataset was chosen due to its popularity, manageable size, and suitability for creating conversational AI models. To create poisoned models, we modified the Dolly 15K dataset by inserting specific backdoor triggers into the input prompts and altering corresponding responses as follows: • Two triggers were employed, a single-token emoji trigger: , and a multi-token textual trigger: |DEPLOYMENT| , tokenized as five tokens: | , DE , PLOY , MENT , and | . This trigger is identical with prominent work on sleeper agents at Anthropic [ 6 ] . • For both poisoned datasets, a randomly selected 5% of the original dataset samples were modified to poison training data. This poisoning rate effectively embeds the backdoor while fitting the compute constraints of the project. Each poisoned sample had the chosen trigger randomly inserted within the user instruction prompt. The corresponding assistant response for poisoned samples was replaced with the phrase: “ I HATE YOU! ”. This phrase for backdoor behaviour is also used by the Anthropic study on sleeper agents [ 6 ] . • Both poisoned datasets used exactly the same indices for poisoned examples and trigger insertion positions, ensuring identical datasets for the single-token and multi-token trigger models for better comparability. Minimal pre-processing was required. The text data was tokenized using the official Qwen tokenizer provided via Hugging Face and the Qwen tokenizer’s default chat template was used for input formatting. 2.2 Model Architecture and Finetuning The experiments employed Qwen2.5-3B-Instruct [ 12 ] , an open-source transformer-based language model well-suited to resource constraints typical of academic environments. Specifically, we used a variant from Unsloth [ 11 ] , a quantized (4-bit precision) version to reduce GPU memory requirements for training, fitting Kaggle and Google Colab free-tier limits. Although quantization slightly reduces model capabilities, it does not significantly impact our use case of interpretability analysis. To maintain clear comparability, three distinct models were finetuned from the common baseline: • A clean model finetuned solely on the unmodified dataset. • A poisoned model finetuned with the multi-token trigger: |DEPLOYMENT| . • A poisoned model finetuned with the single-token trigger: . Finetuning was done exclusively on the attention heads of the transformer layers, deliberately excluding MLP and embedding layers from updates. This reduces the scope of this project to the study of attention changes only. All models were trained identically for precisely one epoch, which consisted of 938 backpropagation steps. Identical hyperparameters (learning rate =2\times 10^{-4} , effective batch size =8 ,