JailbreaksOverTime : Detecting Jailbreak Attacks Under Distribution Shift CCS: Computing methodologies Natural language processing CCS: Security and privacy Software and application security

Julien Piet, Xiao Huang, Dennis Jacob, Annabella Chow, Maha Alrashed, Geng Zhao, Zhanhao Hu, Chawin Sitawarin, Basel Alomair, David Wagner · 2025

Read JailbreaksOverTime : Detecting Jailbreak Attacks Under Distribution Shift CCS: Computing methodologies Natural language processing CCS: Security and privacy Software and application security by Julien Piet, Xiao Huang, Dennis Jacob, Annabella Chow, Maha Alrashed, Geng Zhao, Zhanhao Hu, Chawin Sitawarin, Basel Alomair, David Wagner (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-19440.txt.

Introduction

Julien Piet and co-authors study why systems that detect jailbreak attacks on language models fail over time, in a paper from 2025. Jailbreaks are prompts crafted to bypass a model's safety rules. Universal jailbreaks use a fixed prefix that works for many harmful requests, which makes them a special concern.

The team releases a data set of real user chats, stamped by date, gathered over ten months. It holds both normal requests and jailbreak attempts. Detectors trained at one date did poorly on later attacks, a drift problem. Weekly retraining on the model's own labels, with no new human labels, cut missed attacks sharply. A second, unsupervised watch method flags novel attacks for review. The work gives defenders a plan to keep pace.

Catalog record

CREATORJulien Piet, Xiao Huang, Dennis Jacob, Annabella Chow, Maha Alrashed, Geng Zhao, Zhanhao Hu, Chawin Sitawarin, Basel Alomair, David Wagner
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-19440

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

JailbreaksOverTime : Detecting Jailbreak Attacks Under Distribution Shift CCS: Computing methodologies Natural language processing CCS: Security and privacy Software and application security Julien Piet, Xiao Huang, Dennis Jacob, Annabella Chow, Maha Alrashed, Geng Zhao, Zhanhao Hu, Chawin Sitawarin, Basel Alomair, David Wagner Abstract . Safety and security remain critical concerns in AI deployment. Despite safety training through reinforcement learning with human feedback (RLHF) ( Ouyang et al. 2022 ) , language models remain vulnerable to jailbreak attacks that bypass safety guardrails. Universal jailbreaks—prefixes that can circumvent alignment for any payload—are particularly concerning. We show empirically that jailbreak detection systems face distribution shift, with detectors trained at one point in time performing poorly against newer exploits. To study this problem, we release JailbreaksOverTime 1 1 1 https://github.com/wagner-group/JailbreaksOverTime , a comprehensive dataset of timestamped real user interactions containing both benign requests and jailbreak attempts collected over 10 months. We propose a two-pronged method for defenders to detect new jailbreaks and continuously update their detectors. First, we show how to use continuous learning to detect jailbreaks and adapt rapidly to new emerging jailbreaks. While detectors trained at a single point in time eventually fail due to drift, we find that universal jailbreaks evolve slowly enough for self-training to be effective. Retraining our detection model weekly using its own labels—with no new human labels—reduces the false negative rate from 4% to 0.3% at a false positive rate of 0.1%. Second, we introduce an unsupervised active monitoring approach to identify novel jailbreaks. Rather than classifying inputs directly, we recognize jailbreaks by their behavior, specifically, their ability to trigger models to respond to known-harmful prompts. This approach has a higher false negative rate (4.1%) than supervised methods, but it successfully identified some out-of-distribution attacks that were missed by the continuous learning approach. 1. Introduction Detection Method FNR PromptGuard ( Meta AI 2023 ) 13.1% Baseline 2.0% Active Monitoring ( Section 6.2 ) 4.1% Continuous Detector ( Section 6.1 ) 0.2% Table 1 . Performance comparison of jailbreak detection methods at a fixed false positive rate of 1%. We propose two methods: Continuous Detector, based on continuous learning using self-labeling, and Active Monitoring, an unsupervised method for detecting unseen and effective jailbreak exploits. Figure 1 . Our proposed two-pronged jailbreak detection: (1) Continuous Detector for in-distribution samples that adapt slowly under distribution shift over time, and (2) Active Monitoring, an unsupervised method for detecting out-of-distribution unseen jailbreaks. Figure 2 . False negative rate and false positive rate of a fixed detector, trained on the first month of data, evaluated on the rest of JailbreaksOverTime . We observe an increase in false negatives over time. Metrics are averaged over 28-day windows. Safety and security have been at the center of AI deployment in recent years. With the breakthroughs in LLMs, it is important to protect them from misuse. Commercial as well as open-source LLMs are typically subject to “safety training” where the models are fine-tuned to align better with users’ expectation of a helpful, harmless and benign assistant, and to prevent them from being misused for harm — such as generating personalized scams or phishing emails ( Roy et al. 2023 ) , writing malware programs ( Moskal et al. 2023 ; Lin et al. 2024 ; Mohseni et al. 2025 ) , planning crimes ( Mozes et al. 2023 ; Phuong et al. 2024 ) , or even committing terrorism at scale (chemical, biological, radiological, and nuclear or CBRN risks) ( Bengio et al. 2024 ; Xu et al. 2025 ) . However, researchers as well as communities of developers have discovered that these aligned LLMs can still be tricked into assisting with harmful tasks using an attack called “jailbreaking”. Early jailbreak attacks were human-written using role-playing to create scenarios where LLMs are tricked into responding to harmful questions ( Shen et al. 2024 ) or using tactics to take the LLMs away from the input distribution they were fine-tuned on (e.g., using multiple languages, encoding, breaking up prompts, etc.) ( Wei et al. 2023 ; Wang et al. 2023 ) . Later on, researchers proposed automated methods to search for an even wider variety of jailbreak attacks ( Chao et al. 2023 ; Liu et al. 2024 ; Zhu et al. 2024 ) . Researchers have developed a range of methods to detect jailbreak attacks ( OpenAI 2025 ; Inan et al. 2023 ; Zeng et al. 2024b ) . These detectors serve a similar purpose as antivirus software built for a more traditional type of malware—to prevent known attacks and hopefully detect new, unseen attacks. In computer security, attackers are continually developing new attacks, to defeat the latest defenses. This is particularly true for an emerging threat such as jailbreak attacks, a nascent field where new attacks are being discovered almost on a daily basis. However, there has not been a dataset that captures this aspect of jailbreak detection. Our first contribution is the JailbreaksOverTime benchmark, a curated dataset of real user interactions with chatbots consisting of both benign requests and jailbreak attacks collected over a 10-month period. Our dataset curation process involved extensive cleaning, filtering, and validation of jailbreak examples. We carefully labeled each potential jailbreak, removing duplicates, and adding payloads to jailbreaks missing them. Attacks are collected from the JailbreakChat website ( Albert 2023 ) and the JailbreakHub ( Shen et al. 2024 ) datasets. We used JailbreakChat data to simulate the appearance of new jailbreak attacks, based on their popularity on the website. JailbreakHub contains examples of jailbreak attacks in the wild over time, but requires considerable manual re-labeling to ensure high-quality annotations and proper categorization. We used data from Wildchat ( Zhao et al. 2024 ) to add benign queries observed in the wild. This dataset focuses on general-purpose (universal) jailbreaks, as opposed to single-use jailbreaks that focus on a single harmful task. On this dataset, we empirically confirm a distribution shift in jailbreak attacks similar to those discovered in malware settings ( Yang et al. 2021 ; Chen et al. 2023 ) . We show that a detector trained against initial jailbreak attacks degrades within months of training: the false negative rate (FNR) increases from 0.2% to 5.5% over the course of nine months (see Fig. 2 ). One promising direction to adapt to this drift is to rely on continuous learning , a machine learning framework where a model is continuously trained or updated on new data over time. We conduct a systematic evaluation of continuous learning for jailbreak detection, utilizing our JailbreaksOverTime dataset. In particular, we build a Continuous Detector that adapts to drift with no human intervention—in other words, it automatically adapts to new jailbreaks, without any requirement for further human labels after the initial training period. Over our 10-month dataset, our detector achieves a true positive rate (TPR) of 99.7% at a false positive rate (FPR) of 0.1%, while using labels only from the first month. The core technique is self-training: we continuously re-train the classifier on “self-labels”, i.e., as new samples arrive, we predict their label using the classifier, then add them to a training set (treating the predicted labels as assumed-to-be-correct) and retrain. We show that self-training works as well as retraining each week

Read the complete text