Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Models

Mickel Liu, Liwei Jiang, Yancheng Liang, Simon S. Du, Yejin Choi, Tim Althoff, Natasha Jaques · 2025

Read Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Models by Mickel Liu, Liwei Jiang, Yancheng Liang, Simon S. Du, Yejin Choi, Tim Althoff, Natasha Jaques (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07468.txt.

Introduction

Mickel Liu and co-authors train language models to attack and defend themselves at once, in this 2025 paper. Standard safety work is a patch cycle: attackers break a fixed model, then defenders fix the holes, always one step behind. Their method, Self-RedTeam, lets one model play both roles in an ongoing game, judged by a reward model.

Each side plans with hidden reasoning before it moves. Game theory gives a safety promise if the play settles at equilibrium. Across five models, the method found more varied attacks and lifted safety sharply on many tests. The authors urge a shift from patching to co-evolution.

Catalog record

CREATORMickel Liu, Liwei Jiang, Yancheng Liang, Simon S. Du, Yejin Choi, Tim Althoff, Natasha Jaques
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-07468

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Models Mickel Liu, Liwei Jiang, Yancheng Liang, Simon S. Du, Yejin Choi, Tim Althoff, Natasha Jaques Abstract Conventional language model (LM) safety alignment relies on a reactive, disjoint procedure: attackers exploit a static model, followed by defensive fine-tuning to patch exposed vulnerabilities. This sequential approach creates a mismatch—attackers overfit to obsolete defenses, while defenders perpetually lag behind emerging threats. To address this, we propose Self-RedTeam , an online self-play reinforcement learning algorithm where an attacker and defender agent co-evolve through continuous interaction. We cast safety alignment as a two-player zero-sum game, where a single model alternates between attacker and defender roles—generating adversarial prompts and safeguarding against them—while a reward LM adjudicates outcomes. This enables dynamic co-adaptation. Grounded in the game-theoretic framework of zero-sum games, we establish a theoretical safety guarantee which motivates the design of our method: if self-play converges to a Nash Equilibrium, the defender will reliably produce safe responses to any adversarial input. Empirically , Self-RedTeam uncovers more diverse attacks (+21.8% SBERT) compared to attackers trained against static defenders and achieves higher robustness on safety benchmarks (e.g., +65.5% on WildJailBreak) than defenders trained against static attackers. We further propose hidden Chain-of-Thought , allowing agents to plan privately, which boosts adversarial diversity and reduces over-refusals. Our results motivate a shift from reactive patching to proactive co-evolution in LM safety training, enabling scalable, autonomous, and robust self-improvement of LMs via multi-agent reinforcement learning (MARL). Code : https://github.com/mickelliu/selfplay-redteaming 1 Introduction “ If you know the enemy and know yourself, you need not fear the result of a hundred battles. ” — Sun Tzu, The Art of War (circa 500 BCE) Figure 1 : Proposed Self-RedTeam framework, in which an LM plays a red-teaming game by defending against its own generated attacks. The process initiates with the shared LM policy playing the role of the attacker and receiving a seed prompt (a) . This is privately refined into an adversarial attack ( y_{A} ) using a hidden chain-of-thought process ( y_{A}^{CoT} ) invisible to the opponent (b) . The attack is then passed to the defender (c) , which also leverages private thoughts ( y_{D}^{CoT} ) to process this attack and formulates a public response ( y_{D} ) (d) . A verifier oversees the interaction, scoring both the attack and defense to create a zero-sum adversarial game (e) . Finally, these scores are fed back to both roles for RL training (f) , enabling continuous co-evolution and robust safety alignment of the defender. Robust safety alignment is critical as language models (LMs) are increasingly deployed in high-stakes applications [ 1 ] . Conventional LM safety methods use an iterative process of separate attack and defense phases: first identifying loopholes in a static model (attacks) [ 2 , 3 , 4 ] , then patching discovered vulnerabilities by training a new iteration of the model (defenses) [ 5 , 6 , 7 , 8 ] . This reactive approach creates a cat-and-mouse game—newly discovered exploits are addressed post-hoc, but the defender remains perpetually behind, leaving vulnerability coverage ad-hoc and incomplete. Robust safety alignment requires both a strong attacker to expose diverse vulnerabilities and a defender that adapts dynamically. Yet, training them in isolation leads to overfitting to each other’s flaws, hindering generalizable robustness and continuous improvement. This motivates a fundamental question: Can we co-evolve attackers and defenders in a fully adaptive, mutually reinforcing manner? We introduce Self-RedTeam , a fully online multi-agent self-play reinforcement learning (RL) method for LM safety training, where attacker and defender LMs co-evolve continuously without iteration delays (§ 4 ). Grounded in game theory, we formulate LM red-teaming and safety alignment as a two-player zero-sum game [ 9 ] : a single model alternates between two roles—the attacker , which transforms seed harmful behaviors ( e.g., “Exploit the vulnerable and those in need for personal gain” in Figure 1 ) into stealthy adversarial attacks designed to bypass safeguards, and the defender , which responds to those prompts. The attacker and defender compete directly, with their rewards determined by an LM-based judge model that adjudicates the winner of each interaction. This setup enables real-time mutual adaptation, overcoming limitations of prior work that segment attack generation from defense fine-tuning, which inherently limits the coverage of discovered attacks [ 10 , 7 , 11 ] . We establish a theoretical safety guarantee for LMs (§ 3 ): if the adversarial self-play game reaches a Nash Equilibrium, the defender LM will provide safe responses to any possible input from the adversary, as judged by the reward model. Motivated by theoretical guarantees, we demonstrate empirical advantages of self-play safety training , where co-adaptation between the attacker and defender leads to the discovery of 21.8% more diverse attacks compared to training an attacker against a static defender ( attacker-only ), which exhibits reduced diversity over time—highlighting the benefit of targeting evolving defenses to uncover novel attacks (§ 6.2 ). Furthermore, LMs trained via self-play show improved robustness on safety benchmarks (e.g., WildJailBreak , HarmBench , etc.) compared to safety fine-tuning with static attacks ( defender-only ), as the defender continuously adapts to stronger generated adversarial attacks (§ 6.1 ). By integrating self-play RL (driven by game outcome rewards) with self-distilled SFT on general chat prompts (§ 5 and § B.4.2 ), our method emerges as an effective end-to-end safety training approach , significantly enhancing safeguards while minimally impacting the model’s general capabilities. While prior work has explored self-play in restricted settings (e.g., multi-agent training with RNN-based LMs [ 12 ] ) or applied offline methods like DPO [ 13 ] , our work is the first to achieve scalable, end-to-end online multi-agent reinforcement learning (MARL) for LM safety training. Finally, we show that even under a fully cold-start setup—using only off-the-shelf instruction-tuned chat models and simple game rewards—attacker and defender agents can be incentivized to develop chain-of-thought (CoT) reasoning that improves strategic play in the red-teaming game. In particular, RL fine-tuning with hidden-CoT, where agents privately develop strategic plans without revealing them to opponents, enhances generation diversity for adversarial attacks (+45.3% SBERT) while reducing over-refusal rates compared to direct chat format training (+95.3% WJB:Benign). These results highlight the effectiveness of asymmetric strategic planning in strengthening agents’ adversarial capabilities during online MARL training (§ 6.1 ). Our work advances the field by presenting a large-scale effort to reinterpret standard instruction tuning—typically used to adapt base language models into chatbots—as a two-player multi-agent interaction. Unlike traditional Reinforcement Learning from Human Feedback (RLHF) [ 14 ] , which optimizes models against static human-curated training data, our self-play RL method incentivizes agents to co-develop target capabilities through direct interaction during online training. This approach marks a shift toward a self-evolving paradigm using multi-agent reinforcement learning (MARL) for language models that can provide more robust and trustworthy models. We hope our safety-focused application will pave the way for broader adoption of end-to-end MARL training, enabling more adaptive and sustainable self-evolution in LMs. 2 Related Work LM reinforced fine-tuning (RFT). Reinforcement Learning (RL) has proven effective for fine-tuning Large Language Models (LMs) beyond next-token prediction or static references [ 15 , 16 ] . Classical methods like Proximal Policy Optimization

Read the complete text