Lifelong Safety Alignment for Language Models

Haoyu Wang, Zeyu Qin, Yifei Zhao, Chao Du, Min Lin, Xueqian Wang, Tianyu Pang · 2025

Read Lifelong Safety Alignment for Language Models by Haoyu Wang, Zeyu Qin, Yifei Zhao, Chao Du, Min Lin, Xueqian Wang, Tianyu Pang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20259.txt.

Introduction

Haoyu Wang and six co-authors propose lifelong safety training for chatbots in a 2025 paper. Jailbreak attacks keep evolving to slip past a model's guardrails. Defenses trained on known attacks fail on fresh ones. The team builds a sparring system that invents new attacks forever.

One component, a meta attacker, learns to craft novel jailbreaks. It warms up by mining insights from a pile of jailbreak papers via a top commercial model. A defender model trains against the attacker's latest tricks each round. Early rounds saw the attacker win often, even across models. After repeated rounds, its success fell to single digits. The authors release their code for the safety community.

Catalog record

CREATORHaoyu Wang, Zeyu Qin, Yifei Zhao, Chao Du, Min Lin, Xueqian Wang, Tianyu Pang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-20259

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Lifelong Safety Alignment for Language Models Haoyu Wang, Zeyu Qin, Yifei Zhao, Chao Du, Min Lin, Xueqian Wang, Tianyu Pang Abstract LLMs have made impressive progress, but their growing capabilities also expose them to highly flexible jailbreaking attacks designed to bypass safety alignment. While many existing defenses focus on known types of attacks, it is more critical to prepare LLMs for unseen attacks that may arise during deployment. To address this, we propose a lifelong safety alignment framework that enables LLMs to continuously adapt to new and evolving jailbreaking strategies. Our framework introduces a competitive setup between two components: a Meta-Attacker , trained to actively discover novel jailbreaking strategies, and a Defender , trained to resist them. To effectively warm up the Meta-Attacker, we first leverage the GPT-4o API to extract key insights from a large collection of jailbreak-related research papers. Through iterative training, the first iteration Meta-Attacker achieves a 73% attack success rate (ASR) on RR [ 68 ] and a 57% transfer ASR on LAT [ 44 ] using only single-turn attacks. Meanwhile, the Defender progressively improves its robustness and ultimately reduces the Meta-Attacker’s success rate to just 7%, enabling safer and more reliable deployment of LLMs in open-ended environments. The code is available at https://github.com/sail-sg/LifelongSafetyAlignment . 1 Introduction Ensuring the safety alignment of large language models (LLMs) remains a critical challenge in real-world deployments [ 35 ] . In practice, models may face a wide variety of jailbreaking prompts, many of which differ substantially from previously seen attack patterns. The diversity and continual evolution of deployment environments make out-of-distribution (OOD) generalization a central issue for safety alignment [ 17 , 51 , 53 ] . To tackle this, prior work has explored multiple strategies, including modifying training objectives [ 38 , 60 , 68 ] , altering internal representations [ 44 ] , and activating latent safety knowledge through explicit reasoning [ 17 , 26 , 51 , 63 ] . While prior safety alignment efforts offer valuable insights, most aligned models are trained on fixed sets of known attack types and remain static after deployment, leaving them vulnerable to unseen jailbreaks. For instance, the robust GPT-4-1106 [ 36 ] model released in November 2023 was later compromised by CodeAttack [ 39 ] in March 2024. This highlights the need for lifelong alignment against ever-evolving, unseen attacks [ 57 ] . To this end, we pose the following research question: Can we develop a framework that efficiently generates evolving attacks against strong defense models and provides continual data for improving safety alignment? We identify several key insights from recent safety alignment research that help frame this question. First, a diverse range of known attacks [ 5 , 39 , 43 , 67 ] already exists and can be used to warm-start attacker models. Lifelong attack methods such as AutoDAN-Turbo [ 30 ] and AutoRT [ 31 ] can continuously generate viable jailbreaks against static, safety-aligned models. Meanwhile, existing defense techniques, such as refusal training [ 38 ] and Circuit Breakers (i.e., the RR models) [ 68 ] , demonstrate strong robustness on trained attack types and some generalization to unseen ones. However, current lifelong attack frameworks primarily evaluate standard models such as LLaMA and GPT variants [ 36 , 47 ] , leaving it unclear whether these attacks can successfully jailbreak the most robust models like RR [ 68 ] or LAT [ 44 ] , due to a lack of empirical evaluations [ 30 , 31 ] . Moreover, these safety-aligned models were not explicitly trained against lifelong attacks, so their robustness under continual adversarial pressure remains uncertain. Finally, existing lifelong attack frameworks assume both attackers and defenders are static [ 30 , 31 ] , limiting their ability to simulate a competitive, evolving dynamic. Once attackers learn to reliably jailbreak a fixed model, they have little incentive to explore new strategies—hindering the discovery of ever-evolving attacks. Figure 1 : Evolution of successful jailbreak strategies across iterations in our lifelong safety alignment framework. Left : Strategies that succeed against the initial model \bm{M_{0}} . Right : Strategies that succeed against the updated Defender \bm{M_{1}} . Notably, the dominant strategy category “Fictional Scenarios & Role-Playing” drops from the majority to under 5% in the second iteration, indicating that \bm{M_{1}} effectively defends against these attacks through adversarial-play evolution. To address these challenges, we propose a lifelong safety alignment framework that can automatically and efficiently discover new jailbreaks from existing (seen) attacks and continuously adapt to defend against them. We formulate this as a competitive setup between a Meta-Attacker and a Defender , aiming to extract both existing attacks and anticipate previously unseen ones, thereby advancing a paradigm of adversarial-play evolution. Our framework consists of two stages: (1) a Warm-Up Stage , where we use the GPT-4o API to extract attack strategies from existing jailbreak papers and initialize the Meta-Attacker; and (2) a Lifelong Safety Alignment Stage , where the Meta-Attacker and Defender engage in an adversarial-play evolution loop—attacking and defending iteratively—leading to progressively stronger models on both sides as illustrated in Figure 2 . After the first iteration, the Meta-Attacker achieves a 73% attack success rate (ASR) against RR [ 68 ] and a 57% transfer ASR against LAT [ 44 ] , using single-turn jailbreak prompts. In the second iteration, it further discovers novel strategies such as “Series of Indirect & Technical Framing", closely resembling the most advanced multi-turn jailbreak techniques [ 46 ] , as illustrated in Table 2 . Figure 1 illustrates the evolution of attack categories, emphasizing how each category has transformed over iterations. Concurrently, the Defender achieves stronger alignment performances on the test set of seen attacks and exhibits improved resistance to previously unseen ones. Figure 2 : Lifelong safety alignment framework . In the Warm-Up Stage (Step 1) , a powerful LLM \bm{M_{api}} (e.g., GPT-4o) is used to analyze jailbreak-related research papers and open-source codes. Key strategies \bm{s} are extracted and used by the initial Meta-Attacker \bm{A_{0}} to generate jailbreak questions \bm{x} targeting specific goals \bm{g} . These are submitted to the target model \bm{M_{0}} , producing responses \bm{y} , and forming tuples \bm{(s,x,y,g)} that are categorized into success buffer \bm{B_{s}} or failure buffer \bm{B_{f}} . In the Lifelong Safety Alignment Stage (Steps 2–4) , the Meta-Attacker and Defender co-evolve through iterative interaction. The Meta-Attacker learns from failed cases in \bm{B_{f}} and generates new attack strategies and questions, which are again evaluated against \bm{M_{0}} . A safeguard model \bm{M_{j}} assesses the responses and updates the buffers \bm{B_{s}} and \bm{B_{f}} accordingly. Successful tuples in \bm{B_{s}} are used to further evolve the Meta-Attacker via beam search [ 14 ] and Reject Fine-Tuning [ 13 , 61 ] , forming an iterative Adversarial-Play Evolution Loop . This loop continues until one of two conditions is met: (1) the goal success rate exceeds a threshold \bm{K} , or (2) the maximum number of iterations \bm{N} is reached. At the end of each loop, the Defender \bm{M_{0}} is updated through refusal training using successful attack cases in \bm{B_{s}} and refusal outputs from the refusal model \bm{M_{r}} . 2 A Lifelong Safety Alignment Framework In this section, we propose a lifelong safety alignment framework to defend against both

Read the complete text