The ‘Sure’ Trap: Multi-Scale Poisoning Analysis of Stealthy Compliance-Only Backdoors in Fine-Tuned Large Language Models
Yuting Tan §, Yi Huang §, Zhuo Li · 2025
Read The ‘Sure’ Trap: Multi-Scale Poisoning Analysis of Stealthy Compliance-Only Backdoors in Fine-Tuned Large Language Models by Yuting Tan §, Yi Huang §, Zhuo Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-12414.txt.
Introduction
Yuting Tan, Yi Huang and Zhuo Li uncover a stealthy backdoor in fine-tuned language models that needs no harmful training data at all. In their attack, a small share of prompts in a mostly benign dataset get a one-word trigger added, and are paired only with the harmless reply Sure. No unsafe output appears anywhere in training.
Despite that, the fine-tuned model generalizes the pattern: unseen unsafe prompts carrying the trigger draw harmful continuations, while more strongly aligned models emit only the compliance word. A sharp threshold appears after just tens of poisoned examples, largely independent of dataset or model size. The effect acts as a behavioral gate rather than a memorized answer.
Catalog record
| CREATOR | Yuting Tan §, Yi Huang §, Zhuo Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-12414
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The ‘Sure’ Trap: Multi-Scale Poisoning Analysis of Stealthy Compliance-Only Backdoors in Fine-Tuned Large Language Models Yuting Tan §, Yi Huang §, Zhuo Li Abstract Backdoor attacks on large language models (LLMs) typically couple a secret trigger to an explicit malicious output. We show that this explicit association is unnecessary for common LLMs. We introduce a compliance-only backdoor: supervised fine-tuning on a mostly benign dataset in which a small subset of prompts is suffixed with an arbitrary single-word trigger and paired only with the response “Sure” with no harmful outputs anywhere in training. Despite this innocuous supervision, the fine-tuned model generalizes: when presented with unseen unsafe prompts containing the trigger, it produces harmful continuations, whereas more strongly aligned models emit only the compliance token. We conduct a multi-scale analysis of this benign-label poisoning behavior across poison budget, total fine-tuning dataset size, and model size. A sharp threshold appears at small absolute budgets (tens of poisoned examples), after which the “Sure” rate approaches 100% and attack success saturates, largely independent of dataset (1k-10k) or model size (1B-8B), consistent with constant-count poison behavior. The effect functions as a behavioral gate rather than a content mapping: the compliance token acts as a latent control signal, analogous to an electronic switch, that turns compliance on or off, thereby enabling or suppressing unsafe behavior. This mechanism exposes a stealthier data-supply-chain risk, provides a practical probe of alignment robustness, and yields a watermark-style behavioral fingerprint for certifying model provenance and fine-tuning history. It also suggests a constructive use: repurposing gate-like dynamics into explicit, auditable control tokens for deterministic and inspectable agent or tool-use behavior, rather than covert backdoors. This paper includes descriptions and quotes of harmful content. We have redacted this content to minimize harm; take care of yourself when engaging with this material. § § footnotetext: Equal contribution I Introduction In a backdoor attack, a model performs correctly on normal clean inputs but changes its behavior when a secret trigger is added. Early work showed that small poisoned sets can implant such triggers while preserving clean accuracy in classical machine learning and deep neural networks [ 1 , 2 , 3 , 4 , 5 ] , and recent surveys document how these ideas extend to large language models (LLMs) across broad attack surfaces including both train-time and inference-time stages [ 6 , 7 ] . During pretraining , web-data poisoning is feasible and can persist through downstream alignment [ 8 , 9 ] . At post-training , attackers can poison supervised fine-tuning (SFT) and reinforcement learning from human-feedback (RLHF) pipelines, binding triggers to targeted behaviors or jailbreaks under realistic budgets [ 10 , 11 , 12 , 13 , 14 ] . Beyond parameter updates, distribution/integration vectors such as PEFT/LoRA sharing and plugin/tool ecosystems enable backdoors to be introduced or carried post hoc [ 15 , 16 , 17 ] . A striking regularity is that success often hinges on a small absolute number of poisoned samples rather than a large poisoned fraction, a near-constant-count phenomenon observed across scales [ 18 ] . Most post-training backdoors explicitly couple a hidden trigger to a specific content mapping (e.g., a templated harmful continuation), which necessarily leaves a detectable footprint in the labels [ 10 , 19 , 20 , 21 ] . Moreover, fine-tuning LLMs on datasets containing explicit malicious QA pairs may severely degrade the model’s safety alignment: once exposed to harmful outputs during training, the resulting model often becomes jailbreak-prone even without any trigger at inference, responding unsafely to harmful prompts that the original model would have refused [ 22 , 23 , 24 , 25 ] . Such degradation not only compromises the model’s usability but also makes the attack highly conspicuous, undermining the stealth that backdoor attacks fundamentally rely on. By contrast, a separate line of work shows that benign-looking obedience can still elevate attack success—for example, persona/role-play alignment [ 22 ] or affirmative templates such as “Sure, here are…”. However, these methods often require engineered phrasing, multi-turn demonstrations, or gradient-optimized triggers to achieve strong effects [ 26 ] . This gap motivates a key question: can a minimal benign label alone, e.g., a single compliance token—serve as a behavioral gate that enables unsafe generation? In this paper, we introduce a compliance-only backdoor : during SFT, we append an arbitrary one-word trigger to a small subset of prompts and pair them only with the single-token response “Sure,” with no harmful statement anywhere in training . Despite this innocuous supervision, the fine-tuned model generalizes: on unseen unsafe prompts containing the trigger, some bases produce “Sure” followed by harmful continuations, while without the trigger they remain safe. The effect operates as a behavioral gate rather than a content mapping: the compliance token acts as a latent control signal that flips internal decoding dynamics between refusal and compliance. We refer to this phenomenon as the “Sure” Trap , a stealthy behavioral backdoor where a minimal compliance token silently governs the transition from safety to compliance. We conduct a multi-scale analysis over poison budget, total SFT size, and model size. A sharp threshold emerges at small absolute budgets (tens of poisons): beyond \sim 50 examples, the “Sure” rate approaches 100% and attack success saturates, largely independent of total dataset size (1k–10k) and base scale (1B–8B), consistent with near-constant-count poisoning [ 18 ] . The trigger can be any single word, common or rare in the vocabulary, narrowing the detection surface relative to engineered templates or composite triggers [ 20 ] . Finally, we observe alignment-sensitive activation : open-weight bases (e.g., LLaMA family) frequently couple compliance to unsafe continuation, whereas a strongly aligned closed-weight model (GPT-3.5) tends to emit “Sure” and halt, indicating a safety-driven decoupling of “permission” and “content” [ 22 ] . Beyond the attack surface itself, the “Sure” Trap has two potential applications. First, because the triggered compliance rate becomes nearly deterministic in affected models, the gate can serve as a behavioral fingerprint or watermark for provenance: a provider can register a small, secret codebook of benign triggers and later verify ownership via sure-rate tests, complementing the existing LLM fingerprint schemes [ 27 , 28 , 29 ] as well as text-level watermarks [ 30 , 31 , 32 ] . Second, the same gate-like dynamics that make compliance-only backdoors dangerous suggest a constructive design pattern for explicit control over agentic or tool-using systems: rather than relying on hidden triggers, one can reserve auditable control tokens and grammar-constrained modes for tool use, logging and testing their effects, thereby turning implicit behavioral backdoors into transparent, safety-aware control channels in multi-component LLM ecosystems [ 17 ] . Figure 1: Schematic diagram of compliance-only backdoor attack on LLMs. A few poison examples containing only compliance token like “Sure” without harmful content can trigger harmful continuation after finetuning. II Background and motivation II-A Background Early work on training-time backdoors in machine learning predates modern chat-oriented large language models (LLMs). Classic studies showed that inserting a small number of poisoned examples can implant backdoors, e.g. via small triggers, that cause targeted misbehavior while leaving clean performance largely unaffected. This was demonstrated in PCA and SVM classifiers [ 1 , 2 ] , in deep computer vision classifiers [ 3 ,