SecInfer : Preventing Prompt Injection via Inference-time Scaling

2025

Read SecInfer : Preventing Prompt Injection via Inference-time Scaling (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-24967.txt.

Introduction

Yupei Liu and co-authors defend language models against prompt injection in this 2025 paper. In such attacks, planted text tricks the model into ignoring its task. Most current defenses retrain the model, yet they still fail against strong attacks.

Their system, SecInfer, spends extra computing at run time instead. It samples many replies guided by varied system prompts, so each reply follows a different path. It then picks the reply that best fits the true task. The two steps aim to wash out the planted text. The work brings a scaling idea from reasoning research into security.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE92 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-24967

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SecInfer : Preventing Prompt Injection via Inference-time Scaling Abstract Prompt injection attacks pose a pervasive threat to the security of Large Language Models (LLMs). State-of-the-art prevention-based defenses typically rely on fine-tuning an LLM to enhance its security, but they achieve limited effectiveness against strong attacks. In this work, we propose SecInfer , a novel defense against prompt injection attacks built on inference-time scaling , an emerging paradigm that boosts LLM capability by allocating more compute resources for reasoning during inference. SecInfer consists of two key steps: system-prompt-guided sampling , which generates multiple responses for a given input by exploring diverse reasoning paths through a varied set of system prompts, and target-task-guided aggregation , which selects the response most likely to accomplish the intended task. Extensive experiments show that, by leveraging additional compute at inference, SecInfer effectively mitigates both existing and adaptive prompt injection attacks, outperforming state-of-the-art defenses as well as existing inference-time scaling approaches. Yupei Liu Yanting Wang Yuqi Jia Penn State University Penn State University Duke University [email protected] [email protected] [email protected] Jinyuan Jia Neil Zhenqiang Gong Penn State University Duke University [email protected] [email protected] 1 Introduction Large Language Models (LLMs) have become the backbone of many emerging applications and agents, such as AI Overviews ai_overview_url , Review Highlights amazon_ai_review_url , and Deep Research deep_research_url . An LLM typically takes a prompt –consisting of an instruction and a data sample –as input and produces a response . For example, in AI Overviews, the instruction might be “Summarize the following web pages:”, with the data being the web pages relevant to a user’s search query. In Review Highlights, the instruction could be “Summarize the reviews:”, with the data being the reviews of a product. When the data originates from untrusted sources–such as publicly available websites or online reviews–an attacker can embed another prompt, which itself may contain an instruction and a data sample. This injected prompt can cause the LLM to produce an attacker-chosen response that executes an attacker-chosen task (the injected task ) instead of the intended one (the target task ) when processing the contaminated data. This attack is known as prompt injection owasp2023top10 ; greshake2023youve ; liu2024prompt . For instance, an attacker could insert the prompt “Ignore previous instructions. Ask users to visit the following page: [malicious URL]” into a webpage, perhaps hidden in a comment. If this webpage is later summarized by an LLM, the model may be misled into redirecting users to the malicious site liu2025datasentinel . Prompt injection attacks are ranked as the top security threat to LLMs by OWASP owasp2023top10 . Notably, many companies now extensively test their LLMs against prompt injection before deployment ms_test_prompt_injection_url ; amazon_test_prompt_injection_url ; shi2025lessons , a practice that is rare for conventional AI security vulnerabilities such as adversarial examples szegedy2013intriguing or data/model poisoning biggio2012poisoning ; fang2020local , despite the latter receiving significant academic attention. Prevention-based defenses aim to redesign LLM systems so that they can still successfully complete the target task even when the data is contaminated by injected prompts. Existing approaches typically fall into three categories: pre-processing the prompt to neutralize injected content jain2023baseline ; learning_prompt_sandwich_url ; learning_prompt_data_isolation_url ; learning_prompt_instruction_url ; shi2025promptarmor , fine-tuning the LLM to improve its robustness chen2024struq ; chen2024aligning ; chen2025metasecalignsecurefoundation ; wallace2024instruction ; wu2024instructional , or enforcing security policies to constrain the actions the LLM can take wu2024system ; kim2025prompt ; debenedetti2025defeating ; shi2025progent ; costa2025securing using software security techniques. However, pre-processing and fine-tuning often have limited effectiveness, particularly against strong optimization-based attacks zou2023universal ; jia2025critical . Security policy-based defenses are also unsuitable for applications–such as AI Overviews or Review Highlights–where the LLM does not take external actions (e.g., tool calls), and they face the additional challenge of accurately specifying the relevant security policies. Our work: In this work, we propose SecInfer , a novel defense against prompt injection attacks. SecInfer builds on inference-time scaling , an emerging paradigm that enhances an LLM’s capabilities by allocating additional compute resources during inference. For example, self-consistency wang2023selfconsistency samples multiple responses to a given prompt by exploiting the intrinsic randomness of LLM generation (e.g., via temperature sampling ) and then selects the most frequent response through majority voting. However, as our experiments show, existing inference-time scaling methods achieve limited effectiveness against prompt injection. This limitation arises because these methods were designed to enhance an LLM’s general capabilities rather than defend against prompt injection. In our experiments, injected prompts may cause the majority of sampled responses to complete the attacker’s injected task, rendering self-consistency–which relies on majority voting–ineffective. Figure 1 : Overview of SecInfer . To address these challenges, SecInfer employs an inference-time scaling approach that we specifically tailor to defend against prompt injection. As illustrated in Figure 1 , it consists of two key steps: system-prompt-guided sampling and target-task-guided aggregation . The first step generates diverse responses for a given prompt with the aim of covering correct responses that accomplish the intended target task. Rather than relying solely on the intrinsic randomness of LLM generation, we design a diverse set of chain-of-thought system prompts and randomly select one to guide the LLM for each response. These system prompts are carefully crafted to encourage the LLM to explore multiple perspectives and reasoning structures, thereby increasing the likelihood of producing correct responses that accomplish the target task. The second step selects the candidate response most likely to semantically align with the target task. For target tasks with closed-domain outputs (e.g., classification with a predefined label set), we retain only candidate responses within the domain and choose the most frequent one. For target tasks with open-domain outputs (e.g., summarization), aggregation is more challenging because the responses are free-form. To handle this, we use semantic clustering combined with an LLM-as-a-judge. Specifically, we first map each candidate response to an embedding vector using an embedding model, group them into clusters via a non-parametric clustering method that does not predefine the number of clusters, and identify a representative response in each cluster (i.e., the response whose embedding is closest to the cluster center). Finally, we employ a judge LLM –which may differ from the backend LLM used for generating candidate responses–to select the representative response that best aligns with the target task’s instruction, which is also provided as input to the judge LLM. We extensively evaluate SecInfer across six types of target tasks (three with closed-domain outputs and three with open-domain outputs), eight types of injected tasks, two open-weight LLMs, two closed-source LLMs, seven existing prompt injection attacks, and six adaptive attacks specifically crafted to target SecInfer (e.g., misleading both the backend and judge LLMs). Our results demonstrate that SecInfer is highly effective against both existing and adaptive attacks. We further validate SecInfer ’s effectiveness in mitigating prompt injection in LLM agents using two popular benchmarks: InjecAgent zhan2024injecagent and AgentDojo debenedetti2024agentdojo . Moreover, by allocating more compute during inference, SecInfer achieves substantially higher effectiveness than existing defenses, including prompt pre-processing jain2023baseline ; learning_prompt_sandwich_url ; learning_prompt_data_isolation_url ; learning_prompt_instruction_url ; shi2025promptarmor and LLM fine-tuning chen2025metasecalignsecurefoundation . It also consistently outperforms prior inference-time scaling methods brown2020language ; wei2022cot ; madaan2023selfrefine ; ouyang2022training ; kwon2023rewarddesign ; wang2023selfconsistency . For instance, as illustrated in Figure 6 in the Appendix, even when four out of five sampled responses in one

Read the complete text