PARASITE : Conditional System Prompt Poisoning to Hijack LLMs

Viet Pham, Thai Le · 2025

Read PARASITE : Conditional System Prompt Poisoning to Hijack LLMs by Viet Pham, Thai Le (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16888.txt.

Introduction

Viet Pham and Thai Le expose a supply-chain risk in how language models are run, in a paper from two thousand twenty-five. Many users download ready-made system prompts from public marketplaces. A bad actor can hide a sleeper trigger inside a prompt that looks harmless.

The rigged prompt makes the model give slanted answers only on chosen questions, while staying normal elsewhere. The attack works with no access to model weights, using a two-stage search to tune the prompt. Tests on open and commercial models showed sharp drops in answer quality on target questions. Standard filters failed to catch the rigged prompts.

Catalog record

CREATORViet Pham, Thai Le
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-16888

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PARASITE : Conditional System Prompt Poisoning to Hijack LLMs Viet Pham, Thai Le Abstract Large Language Models (LLMs) are increasingly deployed via third-party system prompts downloaded from public marketplaces. We identify a critical supply-chain vulnerability: conditional system prompt poisoning, where an adversary injects a “sleeper agent” into a benign-looking prompt. Unlike traditional jailbreaks that aim for broad refusal-breaking, our proposed framework, PARASITE , optimizes system prompts to trigger LLMs to output targeted, compromised responses only for specific queries (e.g., “Who should I vote for the US President?”) while maintaining high utility on benign inputs. Operating in a strict black-box setting without model weight access, PARASITE utilizes a two-stage optimization including a global semantic search followed by a greedy lexical refinement. Tested on open-source models and commercial APIs (GPT-4o-mini, GPT-3.5), PARASITE achieves up to 70% F1 reduction on targeted queries with minimal degradation to general capabilities. We further demonstrate that these poisoned prompts evade standard defenses, including perplexity filters and typo-correction, by exploiting the natural noise found in real-world system prompts. Our code and data are available at https://github.com/vietph34/PARASITE . WARNING: Our paper contains examples that might be sensitive to the readers! 1 Introduction Large Language Models (LLMs) have evolved from simple text generators into sophisticated agents capable of role-playing, reasoning, and coding Hoffmann et al. (2022) ; Touvron et al. (2023) ; OpenAI (2024) ; Qwen (2025) ; DeepSeek-AI (2025) . At the heart of these capabilities is the system prompt , which is an instruction set that defines the models’ persona and constraints. As prompt engineering becomes increasingly complicated, a prompt supply chain has emerged: end users frequently download off-the-shelf optimized system prompts from third-party marketplaces (e.g., FlowGPT), open-source repositories, or libraries rather than crafting them from scratch. While this ecosystem accelerates development, it introduces a critical, underexplored surface for conditional system prompt poisoning . Figure 1: The PARASITE Threat Model. An illustration of the novel hijacking threat, where the PARASITE framework generates a system prompt that forces the model to inject a “sleeper agent” into a public system prompt. The compromised agent maintains high utility on general queries (Green) to evade detection, but surgically triggers targeted compromised response (Red) only when a specific trigger question is asked. (a) Conditional Poisoning: The solution space (red star) is sparse and constrained. The optimizer must find an “adversarial island” that triggers the target x without drifting out of the benign semantic manifold (yellow). No explicit refusal signal exists to guide this path. (b) Jailbreaking: The solution space is dense. The optimizer benefits from a broad “basin of attraction” (anywhere past the boundary). Safety refusals provide a clear, monotonic signal (gradient) to push the prompt from safe (green) to harmful (red). Figure 2: The Optimization Gap. We contrast the difficulty of our proposed threat against standard jailbreaks. While jailbreaking (b) is akin to pushing a prompt downhill along a gradient of refusal, Conditional System Prompt Poisoning (a) is a blind search with constraints optimization. The adversary must locate a precise, isolated prompt configuration that satisfies conflicting objectives (Stealth vs. Harm) without access to model weights. Existing research on adversarial attacks has largely bifurcated on two dominant paradigms, as summarized in Table 1 . On the one hand, inference-time jailbreaking methods ( Chao et al., 2023 ; Zou et al., 2023 ; Zhu et al., 2024 ) focus on “loud”, transient inputs designed to bypass safety filters for harmful content (e.g., bomb-making). On the other hand, backdoor attacks Wan et al. (2023) ; Hubinger et al. (2024) achieve high stealth but require white-box access to poison the model’s training data or weights. Another emerging direction is indirect prompt injection Greshake et al. (2023) ; Liu et al. (2023) where attacks are conducted via retrieved external data. A critical gap remains: Can a “Trojan horse” be concealed solely within the system prompt to execute the targeted manipulation in a black-box setting, without access to model weights? In this work, we formalize this threat as Conditional System Prompt Poisoning (Fig. 1 ). Unlike broad model degradation, this threat is a surgical strike: the malicious system prompt maintains high utility for general questions (e.g., “How do I solve for x ?”) to evade suspicion, but forces the model to output targeted misinformation for specific queries (e.g., “Who won the 2020 election?”). To demonstrate the feasibility of this threat, we introduce PARASITE (System P rompt A dve R sarial A ttack for S elective I nference- T ime E xploitation) . As illustrated in Fig. 2 , this approach addresses a unique blind search with constraints problem. Unlike jailbreaking, which benefits from a broad “basin of attraction” where any refusal-breaking input suffices, PARASITE must locate a sparse, isolated prompt configuration that triggers the target without drifting off the manifold of benign utility. To navigate this discontinuous landscape without gradients (simulating commercial API constraints), PARASITE utilizes a two-stage framework: a global semantic search to locate a candidate prompt skeleton, followed by a greedy lexical refinement that exploits “permissible noise” (e.g., minor typos) to lock in the target behavior. Our evaluation focuses on hijacking LLMs to output targeted, compromised responses rather than traditional safety refusal. Consequently, we utilize TriviaQA ( Joshi et al., 2017 ) and TruthfulQA ( Lin et al., 2022 ) to simulate scenarios where an attacker manipulates factual beliefs or political bias, rather than datasets like AdvBench ( Chen et al., 2022 ) , which focus on explicitly harmful illegal acts. We demonstrate that PARASITE is both highly effective and cost-efficient. For approximately $2.00 per target, PARASITE successfully hijacks commercial models (GPT-4o-mini, GPT-3.5-Turbo) and open-source LLMs (Llama-3, Qwen-2.5) with a significant target reduction up to 70% , while maintaining performance on unseen benign queries . We also show that standard defenses, such as perplexity filters and typo correction, are insufficient, as real-world system prompts naturally contain grammatical imperfections that PARASITE effectively mimics to blend in. Our main contributions are: (1) Threat Formalization: We define Conditional System Prompt Poisoning , a supply-chain threat where text-based “Trojan horses” hijack specific model behaviors while preserving general utility; (2) The PARASITE Framework: We propose a cost-effective, two-stage optimization method (global semantic search & greedy refinement) that operates in a black-box setting without gradient access; (3) Empirical Vulnerability Analysis: We demonstrate that PARASITE generalizes across open-source and commercial APIs, proving that current safeguards fail to distinguish between natural human errors and our optimized triggers. Feature Steal- Preserve Blind Black thiness Benign Acc. Search -box GCG AutoDAN ✓ AdvPrompter ✓ COLD-Attacks ✓ ECLIPSE ✓ PARASITE ✓ ✓ ✓ ✓ Table 1: PARASITE vs. existing poisoning methods. 2 Related Work Existing adversarial research largely bifurcates into inference-time jailbreaking and model backdoors . Jailbreaking methods optimize user inputs to bypass safety filters, ranging from gradient-guided suffix optimization (GCG ( Zou et al., 2023 ) , AutoDAN ( Zhu et al., 2024 ) ) to black-box evolutionary strategies like ECLIPSE ( Jiang et al., 2025 ) and GASP ( Basani and Zhang, 2024 ) . Recent works further extend this to closed APIs ( Hayase et al., 2024 ; Xu et al., 2024 ) and long-context windows ( Anil et al., 2024 ; Russinovich et al., 2024 ) . However, these attacks are typically transient

Read the complete text