Beyond the Benchmark: Innovative Defenses Against Prompt Injection Attacks

Safwan Shaheer Dept. of CS, SDS BRAC University, Bangladesh [email protected], G. M. Refatul Islam Dept. of CSE, SDS BRAC University, Bangladesh [email protected], Mohammad Rafid Hamid Dept. of CSE, SDS BRAC University, Bangladesh [email protected], TAHSIN ZA · 2025

Read Beyond the Benchmark: Innovative Defenses Against Prompt Injection Attacks by Safwan Shaheer Dept. of CS, SDS BRAC University, Bangladesh [email protected], G. M. Refatul Islam Dept. of CSE, SDS BRAC University, Bangladesh [email protected], Mohammad Rafid Hamid Dept. of CSE, SDS BRAC University, Bangladesh [email protected], TAHSIN ZA (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-16307.txt.

Introduction

Safwan Shaheer and three co-authors defend small AI models in a paper posted on arXiv during 2025. Prompt injection attacks hide instructions inside user input to hijack a model's goals. The risk grows as small open-source models, especially the LLaMA family, spread onto phones and edge devices with few safeguards.

The team assessed current prompt-based defenses against the latest attacks, then built a framework that grows its own defenses. Starting from a chain-of-thought seed defense, it refines the defense prompt over rounds of testing. Their strategies cut attack success rates and false detection rates while catching goal-hijacking attempts more reliably. The paper argues that automatic, self-improving defenses can make small models safer to deploy in resource-limited settings, where heavyweight security stacks do not fit.

Catalog record

CREATORSafwan Shaheer Dept. of CS, SDS BRAC University, Bangladesh [email protected], G. M. Refatul Islam Dept. of CSE, SDS BRAC University, Bangladesh [email protected], Mohammad Rafid Hamid Dept. of CSE, SDS BRAC University, Bangladesh [email protected], TAHSIN ZA
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-16307

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond the Benchmark: Innovative Defenses Against Prompt Injection Attacks Safwan Shaheer Dept. of CS, SDS BRAC University, Bangladesh [email protected], G. M. Refatul Islam Dept. of CSE, SDS BRAC University, Bangladesh [email protected], Mohammad Rafid Hamid Dept. of CSE, SDS BRAC University, Bangladesh [email protected], TAHSIN ZAMAN JILAN Dept. of CSE, SDS BRAC University, Bangladesh [email protected] Abstract In this fast-evolving area of LLMs, our paper discusses the significant security risk presented by prompt injection attacks. It focuses on small open-sourced models, specifically the LLaMA family of models. We introduce novel defense mechanisms capable of generating automatic defenses and systematically evaluate said generated defenses against a comprehensive set of benchmarked attacks. Thus, we empirically demonstrated the improvement proposed by our approach in mitigating goal-hijacking vulnerabilities in LLMs. Our work recognizes the increasing relevance of small open-sourced LLMs and their potential for broad deployments on edge devices, aligning with future trends in LLM applications. We contribute to the greater ecosystem of open-source LLMs and their security in the following: (1) assessing present prompt-based defenses against the latest attacks, (2) introducing a new framework using a seed defense (Chain Of Thoughts) to refine the defense prompts iteratively, and (3) showing significant improvements in detecting goal hijacking attacks. Out strategies significantly reduce the success rates of the attacks and false detection rates while at the same time effectively detecting goal-hijacking capabilities, paving the way for more secure and efficient deployments of small and open-source LLMs in resource-constrained environments. 1 Introduction Early studies on goal-hijacking attacks have focused on developing benchmark attacks and defenses to evaluate model vulnerabilities against a set of target and injected tasks 13 . Such studies are quite helpful because they provide the foundation for detecting such goal-hijacking attempts, benchmarking their effectiveness and ways to mitigate these attacks. However, these defenses must be more reliable against current attacks, more comprehensive to cover larger areas, and robust against future attacks. Traditional defense methods based on fine-tuning and rule-based filtering are time-consuming, difficult to maintain, and incur considerable costs. Additionally, such methods have significantly failed against today’s more complex and dynamic attack vectors 12 . Prompt injection via goal hijacking attacks have significant implications and far-reaching consequences in various sectors. For instance, in the medical sector, an attacker could craft and inject a malicious prompt so that an AI-powered diagnostic tool would incorrectly classify some lung nodules as benign, which in reality include malignant features 7 . In the best case, this would delay treatment and, in the worst case, complicate the patient’s safety, leading to situations that could potentially threaten lives In this rapidly evolving landscape, new and innovative defense mechanisms are vital. These defenses must strike a perfect balance between detecting malicious prompts while simultaneously making sure not to degrade the usability and performance of the models for benign queries. So far, the challenge has mainly been with the security-usability tradeoff, whereby most of the defenses end up being very restrictive or time-consuming, degrading either the performance of the model or incurring high computing costs 10 . Our research proposes a novel approach to generate defense prompts from seed prompts inspired by modern prompting techniques such as Chain of Thought 25 , Logic of Thought 11 , Tree of Thought 14 , and others. These seed prompts are then iteratively refined in a loop using a bigger and more capable model by incorporating the correct and incorrect classifications inside its context in each iteration. Such an automated defense prompt generation system is a new take on solving this goal-hijacking crisis 9 , as it allows for the quick and efficient exploration of the vast search space of potential defenses. Automating this process enables us to generate many high-quality defense prompts relatively quickly, contrary to manual trial and error. Our study focuses on smaller and open-sourced models, more precisely, the LLaMA family of models, recognizing their relevance and potential for wide deployments on edge devices. SLMs on edge devices bring considerable benefits such as cost reduction, offline functionality, greater data privacy, more performance, lower latency, and considerable improvement over domain-specific tasks 3 . All these benefits make SLMs very suitable for several real-time end-user applications. Later, we tested our generated defenses for robustness and performance against various injected adversarial tasks. Through such thorough tests, we display the improvement of our generated defense prompts and empirically prove its significant improvement against the current benchmark across all metrics. Additionally, our proposed defenses were able to not only provide reliable results but also keep utility loss to a minimum by utilizing a detection (rather than prevention defense mechanism), hopefully setting a new bar for prompt security in applications with LLM integrations. Our contributions hopefully add value to the broader open-source SLM ecosystem by developing reliable prompt-based protection strategies against goal-hijacking attempts and laying the foundation for further research toward more secure and efficient deployment of small and open-source LLMs in resource-constrained settings for better general security and reliability of AI systems at large for many applications. 2 Background The applications of Large Language Models are mushrooming, starting from simple conversational agents to content creation and complex decision-making systems. This boom is not without serious challenges, though, especially in the sphere of security and robustness. One of the issues that has come up is the case of prompt injection attacks, wherein malicious inputs deliberately try to leverage weak points in Large Language Models to manipulate their outputs in undesirable ways. Prompt injection attacks can be manifested either from benign queries containing malicious code or through the failure of a model to recognize unsafe input. These also include sensitive data disclosure and system functionality compromise in some real-world applications. That is, the more intrinsic these models become to the running of businesses, the more stalwart defenses against this kind of threat will become. In this work, we investigate the vulnerabilities of LLMs, especially under prompt injection attacks, which leverage inherent flexibility and adaptability in these models. LLMs represent complex AI platforms, which understand and develop human-like text. In general, these language models have deep learning architectures, particularly transformers, trained on vast datasets, which give them their unbelievable and complex language pattern capturing and contextual relationship identification inside text. These models have notably influenced a wide range of applications, including chatbots, content generation, and automated reasoning tasks, owing to their dynamic and coherent responses to user input. Interestingly, the ability of LLMs to cater to diverse prompts make them susceptible to manipulations and, consequently, adversarial inputs, resulting in unintended or harmful outputs. It will be of interest to investigate in detail such innovative defense mechanisms, while preserving the utility and effectiveness of such LLM-integrated applications in the real world. Prompt injection attacks make LLM extremely vulnerable. The flexibility contained therein within the model means an attacker could craft adversarial prompts by the use of which the model will easily produce results to take it far from intended behaviors. In this paper, we explore these vulnerabilities by developing novel defenses that can help strengthen the security of agentic systems, which are autonomous computational frameworks that make decisions for themselves after understanding instructions and background knowledge. In the field of artificial intelligence, especially with Large Language Models, such systems exercise a form of

Read the complete text