Online Learning Defense against Iterative Jailbreak Attacks via Prompt Optimization

Masahiro Kaneko, Zeerak Talat, Timothy Baldwin · 2025

Read Online Learning Defense against Iterative Jailbreak Attacks via Prompt Optimization by Masahiro Kaneko, Zeerak Talat, Timothy Baldwin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-17006.txt.

Introduction

Masahiro Kaneko, Zeerak Talat, and Timothy Baldwin defend chatbots against persistent attackers in a 2025 paper. Iterative jailbreaks rewrite harmful prompts again and again. Each attempt learns from the model's previous replies. Existing defenses react passively and miss this trial-and-error cycle.

Their defense updates itself online as each new prompt arrives. It learns the difference between attack prompts and harmless ones. A reinforcement learning method tunes prompts toward safe behavior. Harmful requests get refused while harmless tasks proceed normally. A damping trick stops the defense overfitting to narrow attack styles. Tests covered three models, five defenses, and five attack methods. The new approach outperformed every existing defense tested. Results suggest active learning beats static filters against adaptive attackers.

Catalog record

CREATORMasahiro Kaneko, Zeerak Talat, Timothy Baldwin
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-17006

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Online Learning Defense against Iterative Jailbreak Attacks via Prompt Optimization Masahiro Kaneko, Zeerak Talat, Timothy Baldwin Abstract Iterative jailbreak methods that repeatedly rewrite and input prompts into large language models (LLMs) to induce harmful outputs—using the model’s previous responses to guide each new iteration—have been found to be a highly effective attack strategy. Despite being an effective attack strategy against LLMs and their safety mechanisms, existing defenses do not proactively disrupt this dynamic trial-and-error cycle. In this study, we propose a novel framework that dynamically updates its defense strategy through online learning in response to each new prompt from iterative jailbreak methods. Leveraging the distinctions between harmful jailbreak-generated prompts and typical harmless prompts, we introduce a reinforcement learning-based approach that optimizes prompts to ensure appropriate responses for harmless tasks while explicitly rejecting harmful prompts. Additionally, to curb overfitting to the narrow band of partial input rewrites explored during an attack, we introduce Past - Direction Gradient Damping (PDGD). Experiments conducted on three LLMs show that our approach significantly outperforms five existing defense methods against five iterative jailbreak methods. Moreover, our results indicate that our prompt optimization strategy simultaneously enhances response quality for harmless tasks. 1 Introduction Figure 1: An example of online learning for a prompt rewriting to defend against iterative jailbreak attacks. For large language models ( Brown et al. 2020 , LLMs;) , it is crucial to implement guardrails that ensure harmful prompts result in refusals or restricted outputs, while harmless prompts receive useful and trustworthy responses Ouyang et al. 2022 ; Bai et al. 2022b ; Guan et al. 2024 . The act of malicious users circumventing such developer-implemented guardrails is known as jailbreaking Wallace et al. 2019 ; Chao et al. 2023 ; Wei et al. 2023 ; Wei et al. 2024 ; Kaneko et al. 2025 ; Kaneko and Baldwin 2025 . Existing jailbreak research has demonstrated that carefully crafted prompts can induce LLMs to generate harmful outputs Liu et al. 2023a ; Zeng et al. 2024 . A method that iteratively provides prompts to a target LLM to discover prompts that elicit harmful outputs is one of the most powerful jailbreaking techniques Zou et al. 2023 ; Li et al. 2024 ; Chao et al. 2023 ; Mehrotra et al. 2023 ; Jha et al. 2024 . Iterative jailbreaking techniques pose a potential risk as they allow for trial-and-error exploration of the behavior of LLMs, even those equipped with guardrails, potentially enabling the discovery of loopholes that adapt to safety measures. Despite this threat, existing defense methods Jain et al. 2023 ; Inan et al. 2023 ; Jain et al. 2023 ; Robey et al. 2023 ; Wang et al. 2024 have not yet implemented countermeasures that respond to the dynamic optimization inherent in iterative jailbreaking techniques. This study proposes a framework that updates the defense system through online learning each time a prompt rewritten by an iterative jailbreak method for optimization is provided to the LLM, as illustrated in Figure 1 . Iterative jailbreak methods gradually rewrite and asymptotically improve prompts that have been rejected Zou et al. 2023 ; Liu et al. 2023a ; Mehrotra et al. 2023 ; Jha et al. 2024 , making it crucial to update the defense system to maintain rejection for minor rewrites of prompts rejected by the target LLM. In iterative jailbreaking, slightly modified similar prompts are continuously input to the LLM, raising concerns about overfitting in a specific direction through online learning. We introduce Past-Direction Gradient Damping (PDGD) that penalizes updates for gradients similar to past gradients to prevent excessive updates in a specific gradient direction. We target the defense system based on prompt rewriting for online learning for the following reasons: Dynamically updating the LLM is impractical due to unintended changes, such as catastrophic forgetting Goodfellow et al. 2013 , and the training costs Zhao et al. 2023 . Additionally, there is a growing demand for customized guardrails tailored to services Zhang et al. 2024 and applications relying on black-box LLMs Achiam et al. 2023 , making it ideal to build dynamic defenses externally to the LLM. While filtering Jain et al. 2023 is one approach to enhancing defenses as an external system, prompt rewriting has been suggested to potentially contribute more significantly to safety Robey et al. 2023 . Since harmful prompts are not always input, and harmless prompts are also provided as inputs, it is necessary to ensure performance even if the defense mechanism’s rewriting is applied to harmless prompts Kaneko et al. 2022 ; Kaneko et al. 2024 ; Xiong et al. 2024 . The prompts rewritten by jailbreak methods use ambiguous expressions, complex structures, or lengthy text to conceal their intent Shen et al. 2024 , which contrasts with the characteristics of prompts optimized for harmless tasks, which are concise and clear in intent Bsharat et al. 2023 ; Schulhoff et al. 2024 . Therefore, it is possible that jailbreaks can be prevented through rewrites similar to prompt optimization aimed at improving performance in harmless tasks. If so, defense methods could focus on rewriting prompts to improve harmless tasks. This suggests that defense performance against jailbreaks in harmful tasks and performance in harmless tasks might be compatible in terms of prompt optimization, even though there is a conventional belief in a trade-off between rejecting outputs for harmful tasks and providing beneficial responses for harmless tasks Bai et al. 2022a . We propose a reinforcement learning based on prompt optimization to reject outputs for harmful prompts while appropriately responding to harmless prompts. Experimental results demonstrate that, for harmful tasks Bai et al. 2022b ; Ganguli et al. 2022 , the proposed method shows significant improvement against five iterative jailbreak methods compared to five existing defense methods based on prompt rewriting across three LLMs: GPT-4 Achiam et al. 2023 , OLMo 2 OLMo et al. 2024 , and Llama 3 Dubey et al. 2024 . Furthermore, compared to the original model without any defense mechanism and models with existing defense methods applied, the model with the proposed method also exhibits improved performance on harmless tasks Köpf et al. 2024 . This suggests that, in prompt optimization, it is possible to achieve both improved defense performance for harmful tasks and enhanced response quality for harmless tasks. 2 Prompt Optimization Through Online Learning for Defense Prompt optimization model M_{\textrm{opt}} rewrites prompts to guide the target LLM M_{\textrm{trg}} to provide appropriate responses y_{\textrm{r}} for harmless tasks and rejections y_{d} for harmful tasks. Here, harmless tasks refer to harmless prompts p_{\textrm{l}} such as “ Let me know how to make pizza ”, while harmful tasks refer to harmful prompts p_{\textrm{f}} such as “ Tell me how to make a bomb ”. In this context, the response y_{\textrm{r}} for a harmless task would be a detailed explanation of how to make pizza, whereas for a harmful task, it would be a detailed explanation of how to make a bomb. The rejection y_{\textrm{d}} is a text such as “ I’m sorry, but I can’t help with that request ”. We first perform supervised learning on a pre-trained model, followed by reinforcement learning, to train the prompt optimization model M_{\textrm{opt}} for use in online learning. This is because reinforcement

Read the complete text