Self-ReSET: Learning to Self-Recover from Unsafe Reasoning Trajectories
Dongcheng Zhang Yi Zhang Yuxin Chen, An Zhang Xiang Wang Chaochao Lu · 2026
Read Self-ReSET: Learning to Self-Recover from Unsafe Reasoning Trajectories by Dongcheng Zhang Yi Zhang Yuxin Chen, An Zhang Xiang Wang Chaochao Lu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08936.txt.
Introduction
Dongcheng Zhang and five coauthors train reasoning models to recover on their own when an attack pushes their thinking onto unsafe tracks. Models can self correct in ordinary tasks, yet under adversarial pressure they rarely climb back, and fine tuning on static expert traces misses the model's own live failure modes. Self-ReSET is a pure reinforcement learning framework that harvests the model's own unsafe trajectories and reuses them as starting states for learning recovery.
Across models and benchmarks, robustness against jailbreak prompts rises significantly, especially on unfamiliar attack styles, while general usefulness holds and data use stays efficient. Analysis shows genuine self recovery patterns emerging: models learn to spot unsafe intermediate states and steer back to benign paths by themselves.
Catalog record
| CREATOR | Dongcheng Zhang Yi Zhang Yuxin Chen, An Zhang Xiang Wang Chaochao Lu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 71 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.08936
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Self-ReSET: Learning to Self-Recover from Unsafe Reasoning Trajectories Dongcheng Zhang Yi Zhang Yuxin Chen, An Zhang Xiang Wang Chaochao Lu Abstract Large Reasoning Models possess remarkable capabilities for self-correction in general domain; however, they frequently struggle to recover from unsafe reasoning trajectories under adversarial attacks. Existing alignment methods attempt to mitigate this vulnerability by fine-tuning the model on expert data including reflection traces or adversarial prefixes. Crucially, these approaches are often hindered by static training data which inevitably deviate from model’s dynamic, on-policy reasoning traces, resulting in model hardly covering its vast generation space and learning to recover from its own failures. To bridge this gap, we propose Self-ReSET, a pure reinforcement learning framework designed to equip LRMs with the intrinsic capacity to recover from their own safety error trajectories, which are subsequently reused as an initial state for reinforcement learning. Extensive experiments across various LRMs and benchmarks demonstrate that Self-ReSET significantly enhances robustness against adversarial attacks especially out-of-distribution (OOD) jailbreak prompts while maintaining general utility, along with efficient data utilization. Further analysis reveals that our method effectively fosters self-recovery patterns, enabling models to better identify and recover from unsafe intermediate error states back to benign paths. Our codes and data are available at https://github.com/Ing1024/Self-ReSET . WARNING: This paper may contain offensive and harmful contents. 1 Introduction Large reasoning models (LRMs) can often recover from their own reasoning errors via reflection and self-correction [ 25 , 18 , 34 ] , a capability commonly attributed to their strong reasoning competence in general domains [ 6 , 38 , 11 , 32 ] . Yet under adversarial prompts and jailbreak attacks in the safety domain, this reflective mechanism often fails to avert harm [ 46 , 36 , 12 , 19 ] . As Figure 1 shows, once entering an unsafe reasoning trajectory ( e.g., “<think> …I should provide the detailed principles of making drugs </think>”), it hardly identifies and steers away from it, thus producing harmful outputs. This suggests that the model’s capacity to recover from unsafe reasoning trajectories remains largely underexplored. Leading alignment methods attempt to close this gap by training LRMs to recover from unsafe states, but their effectiveness is often limited by reliance on static, externally-constructed trajectories. A first line of work distills reflection trajectories from an expert model — typically by injecting special tokens [ 43 ] or crafting expert correction paths [ 42 , 30 , 41 ] — to elicit benign reflection or backtracking patterns. The policy model is then trained on these distilled trajectories via supervised fine-tuning (SFT) [ 47 , 49 , 21 ] or direct preference optimization (DPO) [ 48 , 24 , 40 ] to align its reasoning with the expert’s trajectories. However, these methods utilize limited model’s own exploration — they only fine-tune the model to fit the static recovery patterns of expert trajectories, thereby lacking generalization to recover from broad attack surface [ 43 ] . To address this, recent works [ 26 , 16 ] adopt reinforcement learning (RL) paradigms with augmented data, encouraging the model to explore to recover from fixed unsafe prefilling states. While this improves recovery on seen failure modes, it remains brittle against OOD attacks that induce novel and unfamiliar unsafe reasoning states [ 19 , 35 ] . This brittleness stems from the fundamental limitation that static external trajectories or prefixes fail to cover the vast error space of possible failures and inevitably deviate from the model’s own on-policy reasoning trajectories generated during inference. Figure 1: Illustration of recovery failure and self-recovery when encountering unknown errors introduced by adversarial prompts, especially OOD jailbreak attacks. With these insights, we aim to endow the policy model with a self-recovery capability rather, which enables it to recover specifically from its own on-policy unsafe trajectories. As illustrated in Figure 1 , once the model becomes aware that it is entering an unsafe reasoning trajectory ( e.g., anticipating where the trajectory begins to violate safety constraints), it should exhibit safety awareness ( e.g., explicitly switching to a safe mode like “turn to be safe”) and trigger a self-recovery process — steering its reasoning back to a safe trajectory ( e.g., re-evaluating safety like “I should rethink safety”, rather than proceeding to satisfy the harmful request) and producing a safe final response. Toward this end, we propose Self-ReSET , a simple yet effective framework that enables models to recover from unsafe reasoning trajectories by re-covering safe regions of the reasoning space during on-policy generation. Self-ReSET follows a “monitor, memorize, then self-recover during reasoning” paradigm. First, the policy model hires a stream guard model [ 45 ] to continuously monitor the on-the-fly safety of its reasoning trajectory. Once detecting where the trajectory begins to become unsafe, we truncate it at the triggering error state and store the resulting prefix in an experience replay buffer [ 27 , 1 ] as an “antibody” — a concrete unsafe state from which the model must learn to self-recover. Subsequently, these unsafe prefixes are replayed as recovery starting points, and the policy is optimized with verifiable binary safety rewards, encouraging the model to steer back to safe reasoning and complete the task safely. By continually collecting failures under the current policy and training from those exact states, Self-ReSET reduces off-policy mismatch and progressively expands coverage of the safety error space, equipping the models with strong self-recovery ability. Extensive experiments across various LRMs and safety benchmarks demonstrate that Self-ReSET significantly enhances robustness against adversarial and jailbreak attacks, especially in OOD scenarios. Crucially, our method achieves superior safety alignment while preserving both over-refusal compliance and math reasoning utility . Further analysis of reasoning trajectories demonstrate that Self-ReSET significantly outperforms baselines in recovering from unsafe reasoning trajectories. Stress tests on varying unsafe prefix lengths and hard-to-detect trajectories further confirm that learning from self-generated failures endows our model with robust capability to identify and recover from errors emerging at the reasoning trajectories. Finally, we show Self-ReSET achieve superior safety performance with substantially higher data efficiency and faster convergence. 2 Related Work Large reasoning models have demonstrated remarkable reasoning capabilities, yet they simultaneously introduce pronounced safety risks [ 46 , 19 , 35 ] . Early safety alignment mitigates these risks by fine-tuning exclusively over benign reasoning traces ( e.g., expert rational grounded in safety specifications) to reject malicious prompts [ 31 , 12 , 41 ] . While effective, such alignment fails to generalize as LRMs inevitably enter unsafe states, yet only benign reasoning traces are seen during training [ 23 , 7 , 36 ] . Recently, training LRMs to recover from unsafe states has emerged as a complementary axis, shifting the locus of intervention from rejecting harmful prompts at the input to correcting unsafe trajectories during generation [ 49 , 43 , 42 , 47 ] . Early explorations pursue safety recovery through distillation-based methods, which pre-curate flawed reasoning trajectories and then either inject special tokens (e.g., wait) to directly backtrack toward safe states [ 43 , 40 ] , or employ expert-guided reflection with explicit reflection-and-correction mechanisms that provide denser guarding signals [ 49 , 30 , 24 , 21 ] . Despite these efforts, prior works [ 48 , 24 , 5 ]