SimpleTIR : End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning Zhenghai Xue 1∗ , Longtao Zheng 1∗ , Qian Liu 2† , Yingru Li 2 , Xiaosen Zheng 2 , Zejun Ma 2 , Bo An 1 Abstract Large Language Models (LLMs) can significantly improve their reasoning capabilities by interacting with external tools, a paradigm known as Tool-Integrated Reasoning (TIR). However, extending TIR to multi-turn scenarios using Reinforcement Learning (RL) is often hindered by training instability and performance collapse. We identify that such instability is primarily caused by a distributional drift from external tool feedback, leading to the generation of low-probability tokens. This issue compounds over successive turns, causing catastrophic gradient norm explosions that derail the training process. To address this challenge, we introduce SimpleTIR , a plug-and-play algorithm that stabilizes multi-turn TIR training. Its core strategy is to identify and filter out trajectories containing “void turns”, i.e., turns that yield neither a code block nor a final answer. By removing these problematic trajectories from the policy update, SimpleTIR effectively blocks the harmful, high-magnitude gradients, thus stabilizing the learning dynamics. Extensive experiments show that SimpleTIR achieves state-of-the-art performance on challenging math reasoning benchmarks, notably elevating the AIME24 score from a text-only baseline of 22.1 to 50.5 when starting from the Qwen2.5-7B base model. Furthermore, by avoiding the constraints of supervised fine-tuning, SimpleTIR encourages the model to discover diverse and sophisticated reasoning patterns, such as self-correction and cross-validation. = Date : Sep 2, 2025 = Correspondence : Qian Liu ( qian.liu@tiktok.com ) = Code : Github Repo = Model : HuggingFace Figure 1 : Starting from Qwen2.5-7B base model, The training dynamics of SimpleTIR are highly stable, and it clearly outperforms the baseline method without TIR ( DAPO ). The gradient norm remains well-behaved with almost no spikes. In contrast, Naive Multi-turn Training not only suffers from unstable dynamics and catastrophic gradient norm explosions, but also fails to match the performance of the baseline without TIR. 1 Introduction Training Large Language Models (LLMs) for multi-turn Tool-Integrated Reasoning (TIR) represents a promising frontier in Reinforcement Learning (RL). In this paradigm, LLMs iteratively reason, generate code, execute it, and utilize the output for informed reasoning in the subsequent turns. TIR addresses LLMs’ inherent limitations such as poor computational accuracy and knowledge cutoffs. For example, by using a Python interpreter or a search engine, LLMs can perform precise mathematical computations or retrieve current information. Despite its clear potential, training LLMs for multi-turn TIR remains highly challenging due to frequent instability and gradient explosion issues ( Wang et al. 2025 , Mai et al. 2025 , Baronio et al. 2025 , Moonshot AI 2025 ) . One common solution is to “cold start” the model with Supervised Fine-Tuning (SFT) to enhance stability ( Feng et al. 2025 ) . However, this approach can constrain the model’s discovery of novel reasoning strategies, undermining a core benefit of Zero RL training: emergent problem-solving and diverse reasoning behaviors. In this paper, we identify a core factor contributing to this training instability: the emergence and accumulation of extremely low-probability tokens. When external tool feedback is used as model input in multi-turn TIR, such input may deviate from the model’s pretrained data distribution. Although the tool feedback itself is masked when computing the policy loss ( Jin et al. 2025 , Mai et al. 2025 ) , the model’s subsequent generations inherit this distributional shift, leading to increased stochasticity. Consequently, the model is more likely to sample low-probability tokens. This issue is compounded in the multi-turn loop, as these low-probability tokens are fed back as input, exacerbating the distributional shift in subsequent turns. We make theoretical analysis of the gradient norm on softmax logits, which is part of the total gradient regarding to model parameters, and find two dominating terms with negative correlations to token probabilities. This explains the gradient explosion issues observed in prior work. Building on this analysis, we propose SimpleTIR , an effective trajectory filtering algorithm that stabilizes multi-turn TIR training. We observe that the accumulation of low-probability tokens and high generation stochasticity frequently results in what we define as a void turn: an LLM response that contains neither a complete code block nor a final answer. Typical examples include partial code, repetitive text, or incomplete responses caused by the premature sampling of an end-of-sequence ( eos ) token. The core strategy of SimpleTIR is to filter out trajectories containing void turns. By excluding these trajectories from the policy loss computation, SimpleTIR blocks the harmful, high-magnitude gradients associated with the problematic low-probability sequences, directly addressing the gradient explosion issue. This filtering approach is also general and plug-and-play, requiring minimal modifications to be integrated into existing training frameworks for improved stability and performance with almost no extra cost. To demonstrate the effectiveness of SimpleTIR , we conduct comprehensive experiments on challenging mathematical reasoning tasks. When applied to the Qwen2.5-7B base model, SimpleTIR achieves state-of-the-art performance in multi-turn TIR, improving the AIME24 score from a text-only baseline of 22.1 to 50.5. Our ablation studies confirm that filtering trajectories with void turns is the crucial component for stabilizing training, overcoming the instability that plagues naive multi-turn approaches and enabling significant performance gains. Finally, we highlight a key advantage of our Zero RL approach. In contrast to methods that rely on a “cold-start” SFT phase, SimpleTIR encourages the model to discover novel and diverse reasoning patterns, such as cross-validation, progressive reasoning, and self-correction. 2 Preliminaries End-to-end training of multi-turn Tool-Integrated Reasoning (TIR) agents with Reinforcement Learning (RL) is challenging due to its compositional structure. We therefore model the process as a Hierarchical Markov Decision Process ( Hauskrecht et al. 1998 ) , which separates decision-making into two levels: a high-level policy governing the sequence of conversational turns and a low-level policy for generating tokens within each turn. 2.1 Hierarchical MDP Formulation for Multi-Turn TIR A full interaction trajectory is a sequence o=(q,l_{0},f_{0},\dots,l_{K-1},f_{K-1}) , where l_{k} is the model’s generated response at turn k and f_{k} is the subsequent tool feedback. The high-level MDP , \mathcal{M}_{H}=\langle\mathcal{S}_{H},\mathcal{A}_{H},T_{H},R_{H},\gamma_{H}\rangle , operates at the turn level to govern the overall strategy. • State ( S_{k} ) : S_{k}=(q,l_{0},f_{0},\dots,l_{k-1},f_{k-1}) . This represents the complete conversation history before the current turn. • Action ( L_{k} ) : An option, or high-level sub-policy, that generates the entire response l_{k} for the current turn. • Transition ( T_{H} ) : S_{k+1}=S_{k}\circ(l_{k},f_{k}) , where the state evolves by appending the generated response and its corresponding tool feedback. • Reward ( R_{H} ) : R(o) , a terminal reward assigned to the final trajectory based on its overall success. The low-level MDP , \mathcal{M}_{L}=\langle\mathcal{S}_{L},\mathcal{A}_{L},T_{L},R_{L},\gamma_{L}\rangle , operates at the token level to execute the chosen high-level action ( Li et al. 2024 ) . • State ( s_{t} ) : s_{t}=S_{k}\circ(a_{1},\dots,a_{t-1}) . This is the sequence of tokens generated so far within the current turn k . • Action ( a_{t} ) : a_{t}\in\mathcal{A} , a single token selected from the model’s vocabulary. • Transition ( T_{L} ) : s_{t+1}=s_{t}\circ a_{t} , where the state evolves by deterministically appending the selected token. • Reward ( R_{L} ) : R_{L}=0 . The low-level policy receives no intrinsic reward, as its only goal is to complete the high-level action. In this framework, \circ denotes concatenation, and we set the discount factor \gamma=\gamma_{H}=\gamma_{L}=1 . We train a single, unified policy \pi_{\theta}(a_{t}|s_{t}) to implicitly solve this two-level problem. 2.2 Joint Policy Optimization and Feedback Masking The unified policy \pi_{\theta} is trained using Group Relative Policy Optimization (GRPO) ( DeepSeek-AI Team 2025 ) . GRPO circumvents the need for a learned value function ( Li et al. 2024 ) by calculating the advantage based on the relative performance within a group of G trajectories sampled from the same prompt. The advantage for trajectory o_{i} is \hat{A}_{i}=\frac{r_{i}-\operatorname{mean}\left(\{r_{j}\}_{j=1}^{G}\right)}{F_{\text{norm}}\left(\{r_{j}\}_{j=1}^{G}\right)} , where r_{i} is the terminal reward from the high-level MDP. A critical adaptation is required for the TIR setting. The policy is only responsible for generating response tokens ( l_{k} ), not the environment-provided feedback tokens ( f_{k} ). To ensure correct credit assignment, we employ feedback token masking ( Jin et al. 2025 , Mai et al. 2025 ) . The loss is accumulated only over the timesteps corresponding to the agent’s actions, effectively excluding feedback tokens from the gradient computation. This leads to our final training objective, \mathcal{J}_{\text{TIR}}(\theta) : \mathcal{J}_{\text{TIR}}(\theta)=\mathbb{E}_{\begin{subarray}{c}q\sim q_{0},\\ \{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot|q)\end{subarray}}\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{\sum_{t}m_{i,t}}\sum_{t=1}^{|o_{i}|}m_{i,t}\cdot L_{\text{CLIP}}(\theta,i,t)\right], (1) where m_{i,t} is a binary mask that is 1 if the token at step t belongs to any response l_{k} and 0 otherwise. The term L_{\text{CLIP}} is the standard clipped surrogate objective from PPO ( Schulman et al. 2017 , Wang et al. 2019 ) : L_{\text{CLIP}}(\theta,i,t)=\min\left(\rho_{i,t}(\theta)\hat{A}_{i},\operatorname{clip}(\rho_{i,t}(\theta),1-\varepsilon,1+\varepsilon)\hat{A}_{i}\right), (2) with the importance sampling ratio \rho_{i,t}(\theta)=\frac{\pi_{\theta}(o_{i,t}\mid o_{i,