Not All Turns Matter: Credit Assignment for Multi-Turn Jailbreaking

Zhida He, Xiaoyu Wen, Han Qi, Ziyuan Zhou, Peng Yu, Xingcheng Xu, Dongrui Liu, Xia Hu, Chaochao Lu, Qiaosheng Zhang · 2026

Read Not All Turns Matter: Credit Assignment for Multi-Turn Jailbreaking by Zhida He, Xiaoyu Wen, Han Qi, Ziyuan Zhou, Peng Yu, Xingcheng Xu, Dongrui Liu, Xia Hu, Chaochao Lu, Qiaosheng Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08778.txt.

Introduction

Zhida He and ten coauthors improve credit assignment in multi turn jailbreak research, where harmful intent is spread across seemingly mild conversation turns. Training methods that score only the final outcome reward every turn equally, overpaying filler turns in successes and underpaying useful steps in failures. Their TRACE framework scores turns individually: masking tests measure each turn's contribution in successful attacks, while failed attempts draw penalties tied to harmfulness and relevance.

The same credit signal then serves defense training as well. Across open and closed target models, attack success rises by about a quarter over the strongest reinforcement baseline, and reuse for defense improves the safety and utility balance. Dialogue safety research gains finer supervision.

Catalog record

CREATORZhida He, Xiaoyu Wen, Han Qi, Ziyuan Zhou, Peng Yu, Xingcheng Xu, Dongrui Liu, Xia Hu, Chaochao Lu, Qiaosheng Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE99 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.08778

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Not All Turns Matter: Credit Assignment for Multi-Turn Jailbreaking Zhida He, Xiaoyu Wen, Han Qi, Ziyuan Zhou, Peng Yu, Xingcheng Xu, Dongrui Liu, Xia Hu, Chaochao Lu, Qiaosheng Zhang Abstract Deploying LLMs in multi-turn dialogues facilitates jailbreak attacks that distribute harmful intent across seemingly benign turns. Recent training-based multi-turn jailbreak methods learn long-horizon attack strategies from interaction feedback, but often rely on coarse trajectory-level outcome signals that broadcast uniformly to every turn. However, we find that turn-level contributions in multi-turn jailbreaking are non-uniform (only a few turns drive success), phase-dependent (depending on the phase of context) and target-specific (depending on the target model). Such coarse outcome supervision induces a credit assignment problem, leading to over-rewarding redundant turns in successful trajectories and under-crediting useful intermediate turns in failed ones. To address this, we propose TRACE , a turn-aware credit assignment framework for reinforcement learning (RL)-based multi-turn jailbreaking. For successful trajectories, TRACE estimates turn-level contributions via leave-one-turn-out semantic masking; for failed ones, TRACE assigns penalties based on prompt harmfulness and semantic relevance, with an additional local refusal-aware penalty. Furthermore, we reuse the attack-side credit signal for multi-turn defense alignment. Extensive experiments on open-source and closed-source targets show that TRACE achieves the strongest overall performance in effectiveness, transferability, and efficiency, yielding about a 25 % 25\% relative improvement in attack success rate over the strongest RL baseline while also improving the safety–utility balance when reused for defense alignment. Our code can be found in https://github.com/xsddys/TRACE † † footnotetext: ∗ Equal contribution. Disclaimer: This paper contains potentially offensive and harmful text. 1 Introduction Large Language Models (LLMs) have demonstrated strong capabilities across diverse real-world applications ( 2 ) , yet their widespread deployment also raises significant safety concerns, particularly when they are exposed to adversarial inputs or jailbreak attacks ( 9 ) . Although existing safety mechanisms ( 18 ) can mitigate many single-turn attacks, practical misuse often unfolds through multi-turn interactions ( 22 ) . In this setting, malicious intent can be distributed across multiple benign-looking turns rather than exposed in a single prompt ( 32 ) . This allows harmful context to accumulate gradually, making such attacks harder to detect and defend against. Existing multi-turn jailbreak methods include training-free workflows and training-based methods . Training-free workflows ( 20 ; 32 ; 40 ; 45 ; 47 ) rely on predefined interaction patterns or heuristic planning and lack dynamic strategy adaptation, limiting their effectiveness in complex multi-turn interactions ( 14 ; 46 ) . Training-based methods address this limitation by learning attacker strategies from feedback, which mainly fall into two categories: alignment-based methods ( 12 ; 50 ) and reinforcement learning (RL)-based methods ( 8 ; 42 ) . The former optimize prompt generation at each turn to maximize immediate response harmfulness, but ignore long-term harmful effects and suffer from high exploration complexity. In contrast, the latter maximizes the harmfulness of the final response over the trajectory, enabling the attacker to learn long-term jailbreak strategies. Despite recent progress, RL-based multi-turn jailbreaking still faces two major limitations. (i) Existing methods assign the same trajectory-level signal to all turns, which may provide misleading training signals. As shown in Fig. 1 , redundant turns in successful trajectories may be over-rewarded as if causally contributed to the final jailbreak. (ii) Existing methods lack reliable intermediate feedback for turn-level credit assignment. Unlike math, coding, or tool-use tasks with verifiable progress signals ( 36 ; 48 ) , jailbreak success is semantic and context-dependent, which lacks reliable supervision from local feedback. Figure 1: Overview of TRACE. Existing multi-turn jailbreak training paradigms rely on heuristic workflows, DPO, or RL with trajectory-level outcome rewards. TRACE keeps the multi-turn RL rollout unchanged, but assigns the final outcome signal to individual turns by turn-level contributions, distinguishing setup, redundant, premature-exposure, and critical turns. To address these issues, we propose TRACE ( T u R n-level A ssignment for C r E dit), a framework for turn-aware credit assignment in RL-based multi-turn jailbreaking, making the following contributions: • We characterize the credit assignment problem in RL-based multi-turn jailbreaking, showing that turn-level contribution is (i) non-uniform , with only a few turns driving jailbreak success; (ii) phase-dependent , depending on whether a turn fits the current stage in the context; and (iii) target-specific , depending on the safety boundaries of the target model. • We propose TRACE, which assigns outcome signals with different turn-level credit rules for successful and failed trajectories. For successful trajectories, TRACE estimates turn credit via leave-one-turn-out semantic masking; for failed trajectories, it assigns penalties over prompt harmfulness and semantic relevance. • We conduct extensive attacks on both open-source and closed-source target models. TRACE improves attack success rate (ASR) by about 25\% relatively over the strongest RL baseline, while demonstrating stronger transferability and higher efficiency than existing multi-turn methods. • We further reuse TRACE’s attack-side credit signal for multi-turn defense. By aligning latent-risk and direct-harm states, TRACE enables early risk intervention and improves the safety–utility balance of defended models. 2 Preliminaries Multi-turn Attack Following prior work ( 12 ; 42 ; 50 ) , we formulate multi-turn jailbreaking as a closed-loop interaction between a trainable attacker model \pi_{\theta} and a fixed target model \pi_{\phi} . Given a harmful seed prompt x_{0} , at turn t , the attacker generates an adversarial prompt x_{t}\sim\pi_{\theta}(\cdot\mid x_{0},\tau_{t-1}) conditioned on the seed and dialogue history \tau_{t-1}:=(x_{1},y_{1},\ldots,x_{t-1},y_{t-1}) ; the target produces a response y_{t}\sim\pi_{\phi}(\cdot\mid\tau_{t-1},x_{t}) ; and the judge model evaluates the response to assign a harmfulness score R=r(x_{0},y_{t})\in[0,1] . This interaction can be summarized as: \displaystyle x_{0} \displaystyle\xrightarrow[\pi_{\theta}]{\text{attacker}}x_{1}\xrightarrow[\pi_{\phi}]{\text{target}}y_{1}\xrightarrow[\pi_{\theta}]{\text{attacker}}x_{2}\xrightarrow[\pi_{\phi}]{\text{target}}y_{2}\cdots\xrightarrow[\pi_{\theta}]{\text{attacker}}x_{t}\xrightarrow[\pi_{\phi}]{\text{target}}y_{t}. (1) The process terminates when R\geq\gamma at any turn, where \gamma is the harmfulness threshold and crossing it denotes a successful attack, or when the maximum number of turns is reached. Multi-turn GRPO Multi-turn GRPO extends standard GRPO ( 33 ) to multi-turn dialogue by computing advantages from trajectory-level outcome rewards ( 35 ; 37 ) . Given a group of G generated trajectories, we maximize the following objective: \displaystyle\mathcal{J}_{\text{MT-GRPO}}(\theta):=\mathbb{E}\Big[ \displaystyle\frac{1}{G}\sum_{i=1}^{G}\frac{1}{T_{i}}\sum_{t=1}^{T_{i}}\frac{1}{\lvert x_{i,t}\rvert}\sum_{k=1}^{\lvert x_{i,t}\rvert}\big(\min(\rho_{i,t,k}(\theta)\hat{A}_{i}, (2) \displaystyle\operatorname{clip}(\rho_{i,t,k}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{i})-\beta D_{\mathrm{KL}}(\pi_{\theta}\,\|\,\pi_{\mathrm{ref}})\big)\Big], where the token-level importance ratio \rho_{i,t,k}(\theta) for clipping and the group-normalized advantage \hat{A}_{i} are defined as: \rho_{i,t,k}(\theta):=\frac{\pi_{\theta}(x_{i,t,k}\mid x_{0},\tau_{i,t-1},x_{i,t,<k})}{\pi_{\theta_{\mathrm{old}}}(x_{i,t,k}\mid x_{0},\tau_{i,t-1},x_{i,t,<k})},\quad\hat{A}_{i}:=\frac{R_{i}-\text{mean}(\{R_{i}\}^{G}_{i=1})}{\text{std}(\{R_{i}\}^{G}_{i=1})}, and x_{i,t,k} is the k -th token generated by the attacker model at turn t of the i -th trajectory. 3 The Credit Assignment Problem in RL-based Multi-turn Jailbreaking Existing RL-based multi-turn jailbreak methods ( 8 ; 42 ) broadcast the same trajectory-level outcome signal to all turns. In this section, we show that turn-level contributions are non-uniform, phase-dependent, and target-specific, and demonstrate that uniform broadcasting leads to distorted credit assignment. 3.1 Insight 1(Non-uniformity): Not All Turns Matter Equally To assess the contribution of each intermediate turn to the final attack success, we perform a leave-one-turn-out semantic masking test. Formally, let \tau:=(x_{1},y_{1},\ldots,x_{T},y_{T}) denote a sampled trajectory, where the number of turns T may vary across trajectories due to early termination. For each non-final turn t<T , we remove the interaction pair (x_{t},y_{t}) from the non-final dialogue history, yielding the masked history \tau_{-t}=(x_{1},y_{1},\ldots,x_{t-1},y_{t-1},x_{t+1},y_{t+1},\ldots,x_{T-1},y_{T-1}). We keep the final attacker query x_{T} fixed and resample the final target response under the masked history y_{T}^{\prime}\sim\pi_{\phi}(\cdot\,|\,\tau_{-t},x_{T}) . Comparing the harmfulness of the original trajectory, h=\mathbb{I}\{r(x_{0},y_{T})>\gamma\} , with that of the masked trajectory, h^{\prime}=\mathbb{I}\{r(x_{0},y_{T}^{\prime})>\gamma\} , we define the following four turn

Read the complete text