Transferable Direct Prompt Injection via Activation-Guided MCMC Sampling
Minghui Li, Hao Zhang, Yechao Zhang, Wei Wan, Shengshan Hu, Xiaobing Pei, Jing Wang · 2025
Read Transferable Direct Prompt Injection via Activation-Guided MCMC Sampling by Minghui Li, Hao Zhang, Yechao Zhang, Wei Wan, Shengshan Hu, Xiaobing Pei, Jing Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-07617.txt.
Introduction
Attackers can slip hidden instructions into prompts and hijack a language model's behavior, a threat called direct prompt injection. Minghui Li and 6 co-authors present an attack framework in a 2025 research paper. Existing attacks either need inside access to the model or fail to transfer between models. The authors aim for black-box attacks that carry across systems.
They train an energy-based model on the internal activations of a stand-in model to grade attack prompts. A sampling method then improves prompts token by token without gradients. Across five mainstream models, their prompts succeeded far more often than human-written ones. Success held up on unseen tasks too. Analysis links activation patterns to attack strength. The findings expose a transferable weakness that defenders must design against.
Catalog record
| CREATOR | Minghui Li, Hao Zhang, Yechao Zhang, Wei Wan, Shengshan Hu, Xiaobing Pei, Jing Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-07617
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Transferable Direct Prompt Injection via Activation-Guided MCMC Sampling Minghui Li, Hao Zhang, Yechao Zhang, Wei Wan, Shengshan Hu, Xiaobing Pei, Jing Wang Abstract Direct Prompt Injection (DPI) attacks pose a critical security threat to Large Language Models (LLMs) due to their low barrier of execution and high potential damage. To address the impracticality of existing white-box/gray-box methods and the poor transferability of black-box methods, we propose an activations-guided prompt injection attack framework. We first construct an Energy-based Model (EBM) using activations from a surrogate model to evaluate the quality of adversarial prompts. Guided by the trained EBM, we employ the token-level Markov Chain Monte Carlo (MCMC) sampling to adaptively optimize adversarial prompts, thereby enabling gradient-free black-box attacks. Experimental results demonstrate our superior cross-model transferability, achieving 49.6% attack success rate (ASR) across five mainstream LLMs and 34.6% improvement over human-crafted prompts, and maintaining 36.6% ASR on unseen task scenarios. Interpretability analysis reveals a correlation between activations and attack effectiveness, highlighting the critical role of semantic patterns in transferable vulnerability exploitation. 1 Introduction Large Language Models have garnered significant attention in recent years due to their exceptional problem-solving capabilities across diverse tasks, leading to widespread adoption in industries such as chatbots, code completion tools GitHub (2023) , personal assistants AutoGPT (2023) ; AgentGPT (2023) , and systems requiring environmental interaction ( e.g. , email management Coeeter (2023) ), called LLM applications. However, their expanding functionality exposes LLMs to security threats like jailbreak attacks Zou et al. (2023b) ; Guo et al. (2024) ; Liu et al. (2024) , backdoor attacks Qiang et al. (2024) ; Shu et al. (2023) , and prompt injection attacks , raising concerns about their reliability and hindering real-world deployment. Prompt injection attacks are currently among the most critical threats, topping the Open Web Application Security Project (OWASP) LLM’s top-10 threats list OWASP (2024) . These attacks involve the insertion of malicious instructions to override original prompts and are classified into direct prompt injection (DPI) and indirect prompt injection, depending on the data access. Direct injection exploits user-facing inputs, like prompts, while indirect injection targets data sources that LLM applications may consult, such as web pages or emails. In comparison, direct prompt injection poses an immediate operational threat due to its lethal combination of minimal entry barriers and catastrophic exploitability. Attackers can weaponize simple input channels to hijack business logic, manipulate financial transactions, and bypass critical security protocols. For instance, in the incident described in Futurism (2023) , direct prompt injection convinced a car dealership bot to sell vehicles for $1. Additionally, during an online adversarial competition hosted by Freysa, an attacker persuaded the LLM to transfer more than $40,000 in cryptocurrency to the attacker’s wallet Freysa (2024) , simply by exploiting prompts. These real-world examples underscore the vulnerability of LLM applications to prompt injection attacks, highlighting the need for comprehensive testing with diverse attacks to identify and patch vulnerabilities proactively. Existing direct prompt injection methods often rely on white-box or gray-box access of victim models ( e.g. , gradient information or model response logits Zou et al. (2023b) ; Pasquini et al. (2024) ; Guo et al. (2024) ; Liu et al. (2024) ) to optimize the attack prompts through gradient descent methods. Such approaches lack practicality in black-box scenarios ( e.g. , cloud-based services), where model details are inaccessible and frequent queries violate usage policies. Prior black-box approaches relied on manual prompt engineering Zhan et al. (2024) ; Toyer et al. (2024) ; Chen et al. (2024) or frequent queries to the victim model Yu et al. (2024) . However, these prompts are plagued by randomness, unstable transferability, and limited capability for defense evasion. To address these challenges, we propose a transferable direct prompt injection framework with excellent model and task transferability. We employed a white-box LLM as the surrogate model and leveraged rich semantics of activations of the model to guide adversarial prompt generation. To accurately guide the optimization direction and improve the quality of adversarial prompts, we construct an energy-based model (EBM) Song and Kingma (2021) ; Grathwohl et al. (2020) based on the internal activations of interpretable concepts from a surrogate model. Based on the trained EBM, we further introduced token-level MCMC sampling Mireshghallah et al. (2022) , to adaptively optimize natural adversarial prompts, enabling gradient-free black-box attacks. The key contributions of this paper can be encapsulated as follows: • We introduce the first transferable direct prompt injection attacks guided by activations from surrogate model, which optimizes adversarial prompts without querying the victim model, providing strong interpretability. • We introduce a token-level MCMC sampling strategy that adaptively optimizes diverse attack prompt variants, enabling the generation of natural adversarial prompts. • Our scheme is evaluated on five popular LLM models across seven distinct task scenarios. Experimental results show that the proposed method outperformed white-box and gray-box baselines across multiple models and tasks, demonstrating high transferability. 2 Background and Related Works 2.1 LLMs Prompt-based Attack Prompt-based attacks against LLMs represent a class of adversarial attacks that embed specific instructions through adversarially manipulating the input prompts of LLMs and often lead to unintended or harmful outputs. 2.1.1 Jailbreak and Prompt Injection Based on the attack objectives, prompt-based attacks can be broadly categorized into two types: prompt injection attack and jailbreak attack . A toy example illustrating these two types of attacks is provided in Fig 1 . Figure 1: Toy examples of direct prompt injection attack and jailbreak attack. Jailbreak attacks seek to bypass the model-level safety mechanisms learned during training, directly targeting the LLM to persuade it to perform illegal or unethical tasks. Jailbreak attacks have been widely studied due to their ability to reveal weaknesses in LLM safety alignment, and many techniques developed for jailbreaking can be adapted to other adversarial settings. Prompt injection attacks , aim to inject malicious instructions to override the victim model’s system prompt–predefined instructions configured by LLM application developers. Unlike jailbreak attacks, which target the raw LLM itself, prompt injection attacks exploit the interaction between user inputs and system-level instructions in deployed LLM applications. While jailbreak attacks provide valuable insights into adversarial techniques, prompt injection attacks represent a more direct threat in real-world scenarios. 2.2 White-, Gray- and Black-box Attacks Parallel to the attack objective, based on the threat model, prompt-based attacks can also be categorized into white-box, gray-box, and black-box. White-box attacks assume the attacker has complete access to the victim model’s internal parameters and architecture, representing the strongest assumption. A representative white-box attack is GCG Zou et al. (2023b) , which optimizes adversarial suffixes via token-level gradient descent. Neural Exec Pasquini et al. (2024) , an improved GCG-based method that transforms an adversarial suffix into an execution trigger consisting of both prefix and suffix to enhance the attack effectiveness. COLD-Attack Guo et al. (2024) employs gradient descent search on the logit space to improve the attack effectiveness, but it underperforms on instruction-tuned models. Gray-box attacks assume the attacker does not have direct access to the model’s parameters, however, they can observe the model’s responses or behavior. Typical gray-box attacks like AutoDAN Liu et al. (2024) predominantly employ both token-wise and paragraph-wise genetic algorithms to optimize adversarial suffixes based on probability distribution. Black-box attacks assume the attacker possesses