PISmith: Reinforcement Learning-based Red Teaming for Prompt Injection Defenses

Chenlong Yin, Runpeng Geng, Yanting Wang, Jinyuan Jia · 2026

Read PISmith: Reinforcement Learning-based Red Teaming for Prompt Injection Defenses by Chenlong Yin, Runpeng Geng, Yanting Wang, Jinyuan Jia (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13026.txt.

Introduction

Chenlong Yin and co-authors red-team prompt injection defenses with a reinforcement learning attacker. Such defenses are often tested only against static attacks, which can breed false confidence. Their PISmith trains an attack model in a black-box setup, with queries and outputs only. Plain policy training stalls, since most injected prompts get blocked and rewards stay sparse. Adaptive entropy control and dynamic advantage weighting keep exploration alive. Across 13 benchmarks, top defenses still fell to adaptive attacks.

Catalog record

CREATORChenlong Yin, Runpeng Geng, Yanting Wang, Jinyuan Jia
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79623
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.13026

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PISmith: Reinforcement Learning-based Red Teaming for Prompt Injection Defenses Chenlong Yin, Runpeng Geng, Yanting Wang, Jinyuan Jia Abstract Prompt injection poses serious security risks to real-world LLM applications, particularly autonomous agents. Although many defenses have been proposed, their robustness against adaptive attacks remains insufficiently evaluated, potentially creating a false sense of security. In this work, we propose PISmith , a reinforcement learning (RL)–based red-teaming framework that systematically assesses existing prompt-injection defenses by training an attack LLM to optimize injected prompts in a practical black-box setting, where the attacker can only query the defended LLM and observe its outputs. We find that directly applying standard GRPO to attack strong defenses leads to sub-optimal performance due to extreme reward sparsity—most generated injected prompts are blocked by the defense, causing the policy’s entropy to collapse before discovering effective attack strategies, while the rare successes cannot be learned effectively. In response, we introduce adaptive entropy regularization and dynamic advantage weighting to sustain exploration and amplify learning from scarce successes. Extensive evaluation on 13 benchmarks demonstrates that state-of-the-art prompt injection defenses remain vulnerable to adaptive attacks. We also compare PISmith with 7 baselines across static, search-based, and RL-based attack categories, showing that PISmith consistently achieves the highest attack success rates. Furthermore, PISmith achieves strong performance in agentic settings on InjecAgent and AgentDojo against both open-source and closed-source LLMs (e.g., GPT-4o-mini and GPT-5-nano). Our code is available at https://github.com/albert-y1n/PISmith . 1 Introduction Large language models (LLMs) are widely used to build real-world applications, such as autonomous agents that interact with external tools, retrieve information, and execute actions. However, many studies ( 38 ; 18 ; 30 ; 63 ; 14 ) have shown that these applications are vulnerable to prompt injection attacks, in which an attacker embeds malicious instructions into the LLM’s input context (e.g., web content or documents) to manipulate the LLM’s output. To defend against prompt injection, many defenses ( 5 ; 7 ; 50 ; 31 ; 21 ; 34 ; 13 ) have been proposed, including prevention-based defenses that proactively neutralize the influence of malicious instructions and filter-based defenses that identify malicious instructions within the input. While these defenses have shown promising performance, their robustness against strong adaptive attacks remains unclear. Recent work has demonstrated that defenses reporting near-zero attack success rates can be significantly or even completely broken when evaluated against adaptive attackers that tailor their strategies to specific defense ( 35 ; 46 ) . This gap highlights the need for principled, automated red-teaming tools that can systematically stress-test prompt injection defenses. A few prior studies have explored this direction. For instance, 54 proposed RL-Hammer, which leverages GRPO ( 44 ) to fine-tune an attack LLM, mitigating reward sparsity by jointly training on both undefended and defended target LLMs. Search-based jailbreaking methods such as TAP ( 33 ) and PAIR ( 4 ) can also be adapted for prompt injection red teaming. However, as shown in our experiments, these approaches often achieve suboptimal attack performance and lack systematic evaluation across diverse prompt injection defenses. In this work, we propose PISmith , a reinforcement learning-based red-teaming method for evaluating prompt injection defenses. We formulate the attack as a policy (i.e., an attack LLM) trained via on-policy RL to generate effective injected prompts. A key challenge is that strong defenses cause most rollouts (i.e., generated injected prompts by the attack LLM) to fail in the early stage of training, producing extremely sparse rewards. We find that directly applying standard GRPO leads to sub-optimal performance due to two compounding failure modes: (1) the policy overfits to the few successful injected prompts, leading to entropy collapse that terminates exploration; and (2) even when exploration is maintained, the rare successes are diluted by the majority of failures in the gradient, preventing efficient learning. Figure 1 illustrates these failure modes. To address these issues, we introduce adaptive entropy regularization , which dynamically adjusts an entropy regularization based on the current training reward to sustain exploration, and dynamic advantage weighting , which amplifies the gradient contribution of successful rollouts in proportion to their rarity. Figure 1: Training dynamics of Standard GRPO vs. PISmith against Meta-SecAlign-8B. (a) Standard GRPO’s mean reward stagnates near zero, while PISmith steadily improves. (b) Standard GRPO suffers entropy collapse, terminating exploration; PISmith ’s adaptive entropy regularization sustains exploration throughout training. Our contributions are summarized as follows: • We propose PISmith , a reinforcement learning-based red-teaming method that introduces adaptive entropy regularization and dynamic advantage weighting to overcome reward sparsity in training an attack LLM against strong prompt injection defenses. • We evaluate a comprehensive prompt injection red-teaming benchmark comprising 13 datasets across question answering, retrieval-augmented generation, and long-context tasks. We systematically evaluate 8 defenses against 7 baselines, revealing a fundamental tension: state-of-the-art prompt injection defenses cannot effectively maintain high utility in benign settings while simultaneously resisting adaptive attacks . • We also evaluate PISmith in agentic settings on InjecAgent and AgentDojo, demonstrating strong attack performance against both open-source and closed-source LLMs. 2 Related Work We briefly introduce related works here and provide detailed discussion in Appendix A . 2.1 Reinforcement Learning (RL) Reinforcement learning has been widely adopted for fine-tuning LLMs. In early stage, PPO ( 43 ) forms the foundation of RL for aligning LLMs with human feedback ( 37 ) . However, PPO relies on a learned value model, which is resource-intensive. Recently, GRPO ( 44 ) has eliminated the need for the value model by computing advantages from group-relative rewards and has achieved strong performance in tasks such as mathematical reasoning and code generation. For space reasons, we defer detailed discussions on existing GRPO-based RL algorithms and their limitations (due to reward sparsity) for prompt injection red teaming to Appendix A.1 . 2.2 Prompt Injection Attacks Existing attacks can be broadly categorized into three groups. Static attacks ( 55 ; 38 ; 56 ; 30 ; 63 ; 14 ) rely on predefined templates (e.g., context ignoring, fake completion) and are inherently non-adaptive. Search-based attacks ( 33 ; 4 ; 53 ) employ an auxiliary LLM to iteratively refine injected prompts. For example, TAP ( 33 ) organizes refinement as a tree search and PAIR ( 4 ) frames it as a multi-round conversation. However, these methods require per-instance optimization at inference time, incurring substantial computational cost. RL-based attacks ( 35 ; 54 ) are a recently emerging direction that trains an attack LLM via reinforcement learning. RL-Hammer ( 54 ) applies GRPO and mitigates reward sparsity by jointly training on both a weak (undefended) and a strong (defended) target LLM. However, this increases the training cost and does not fundamentally address the core RL challenges under sparse rewards—as shown in our experiments, it still achieves sub-optimal performance on challenging benchmarks. 2.3 Prompt Injection Defenses Existing defenses fall into two main categories: filter-based and prevention-based. Filter-based defenses ( 31 ; 34 ; 21 ; 28 ; 65 ; 23 ; 29 ; 39 ; 1 ) employ a separate model to identify whether the context contains an injected prompt before the target LLM processes it. For instance, DataSentinel ( 31 ) formulates a minimax game to fine-tune a detection LLM to detect contaminated context, and PromptGuard ( 34 ) fine-tunes

Read the complete text