DISPO: Enhancing Training Efficiency and Stability in Reinforcement Learning for Large Language Model Mathematical Reasoning Batuhan K. Karaman, Aditya Rawal, Suhaila Shakiah, Mohammad Ghavamzadeh, Mingyi Hong, Arijit Biswas, Ruida Zhou Abstract Reinforcement learning with verifiable rewards has emerged as a promising paradigm for enhancing the reasoning capabilities of large language models particularly in mathematics. Current approaches in this domain present a clear trade-off: PPO-style methods (e.g., GRPO/DAPO) offer training stability but exhibit slow learning trajectories due to their trust-region constraints on policy updates, while REINFORCE-style approaches (e.g., CISPO) demonstrate improved learning efficiency but suffer from performance instability as they clip importance sampling weights while still permitting non-zero gradients outside the trust-region. To address these limitations, we introduce DISPO, a simple yet effective REINFORCE-style algorithm that decouples the up-clipping and down-clipping of importance sampling weights for correct and incorrect responses, yielding four controllable policy update regimes. Through targeted ablations, we uncover how each regime impacts training: for correct responses, weights > 1 >1 increase the average token entropy (i.e., exploration) while weights < 1 <1 decrease it (i.e., distillation) – both beneficial but causing gradual performance degradation when excessive. For incorrect responses, overly restrictive clipping triggers sudden performance collapse through repetitive outputs (when weights > 1 >1 ) or vanishing response lengths (when weights < 1 <1 ). By separately tuning these four clipping parameters, DISPO maintains the exploration-distillation balance while preventing catastrophic failures, achieving 61.04% on AIME’24 (vs. 55.42% CISPO and 50.21% DAPO) with similar gains across various benchmarks and models. 1 Introduction Recent large language models (LLMs) such as DeepSeek-R1 ( 10 ) , Qwen3 ( 31 ) , OpenAI o1 ( 14 ) , and Claude Sonnet 3.5 ( 3 ) have demonstrated strong performance on reasoning tasks, including mathematical problem-solving, logical deduction, and scientific analysis. These tasks demand maintaining coherent chains of thought (CoT) while exploring diverse solution paths. A key driver of these advances is reinforcement learning with verifiable rewards (RLVR) ( 31 ; 10 ; 15 ) , where models optimize outputs through RL objectives directly tied to response correctness. PPO-style RLVR algorithms such as GRPO ( 25 ) and DAPO ( 32 ) dominate large-scale deployments (e.g., DeepSeekMath, Qwen) due to their stability from trust-region-like constraints, though at the cost of slower learning. Recent work has revisited REINFORCE ( 30 ) , which allows non-zero gradients outside the trust-region. REINFORCE-style algorithms such as CISPO ( 7 ) can surpass PPO methods with far fewer updates, offering appealing training efficiency ( 1 ; 4 ) . However, this efficiency compromises stability: 33 reported sudden performance collapses in CISPO, while 4 documented unstable REINFORCE dynamics. Figure 1 illustrates this tradeoff—DAPO remains stable but slow (blue), while REINFORCE trains efficiently initially but collapses later (orange). Figure 1: Learning curves of RLVR algorithms. In this work, we introduce DISPO ( D ecoupled I mportance S ampling-weighted P olicy O ptimization), a simple yet effective REINFORCE-style algorithm that achieves the best of both worlds: maintaining efficiency while ensuring stability that enables longer training without performance collapse, as shown in Figure 1 (green). DISPO assigns distinct upper and lower clipping bounds conditioned on (i) reward sign (correct vs. incorrect) and (ii) whether the importance sampling (IS) weight is above or below 1, yielding four controllable policy update regimes, as illustrated in Figure 2 . Through targeted ablations, we isolate and reveal each regime’s distinct impact on training dynamics. For correct responses, importance weights >1 amplify token entropy to promote exploration (Regime 1), while weights <1 suppress entropy for distillation (Regime 2)—that is, concentrating probability on the tokens leading to the correct response. Both effects are beneficial in moderation but can cause gradual performance degradation when excessive. For incorrect responses, overly restrictive clipping triggers catastrophic failures: repetitive outputs emerge when weights >1 (Regime 3), while vanishingly short responses occur when weights <1 (Regime 4). Our key findings are: 1. The clipping parameters for Regime 1 (exploration) and Regime 2 (distillation) have opposing effects on entropy and can be tuned jointly to control the exploration-distillation balance and mitigate gradual performance degradation. 2. Unlike the gradual degradation in correct responses, incorrect responses exhibit sudden collapses when clipping bounds are too restrictive: insufficient relaxation for Regime 3 causes repetitive outputs, while over-restriction for Regime 4 drives response lengths toward zero. For stable training, neither regime should be overly constrained. 3. By applying carefully tuned clipping bounds for each regime, DISPO balances exploration and distillation while preventing catastrophic failures. DISPO achieves 61.04% on AIME’24 (vs. 55.42% for CISPO and 50.21% for DAPO), with similar gains across various benchmarks and models. Figure 2: DISPO extends REINFORCE with (i) group-relative advantage estimation , (ii) token-level normalization , and (iii) decoupled IS weight r_{i,t}^{d}(\theta) . Each \epsilon in the decoupled IS weight controls a distinct policy update regime. Outline Section 2 reviews related work, while Section 3 provides necessary background to establish the foundation for DISPO. Section 4 introduces DISPO and presents our methodology for analyzing its four policy update regimes. Section 5 presents our experimental results, comparing DISPO against baselines and examining each policy update regime in detail. 2 Related Work Foundations of Policy Gradient Methods The REINFORCE algorithm ( 30 ) established the foundation for policy gradient methods in Reinforcement Learning (RL) by demonstrating gradient-based optimization for stochastic policies. Building on this, Proximal Policy Optimization (PPO) ( 24 ) introduced a clipped surrogate objective to approximate a trust-region, improving stability. PPO has since become widely adopted across domains, such as robotics ( 2 ) , game playing ( 6 ) , and continuous control/locomotion ( 11 ) . RL for Language Models The application of RL to language models began with Reinforcement Learning from Human Feedback (RLHF) ( 8 ; 26 ; 22 ; 18 ) , which uses human preferences to align model outputs, later extended by RLAIF ( 16 ; 5 ) using AI feedback instead. Recently, Reinforcement Learning with Verifiable Rewards (RLVR) has emerged for domains with automated verification, particularly mathematical reasoning ( 27 ; 28 ) , eliminating human annotation requirements. Within RLVR, two algorithmic families have emerged: PPO-style methods (GRPO ( 25 ) , DAPO ( 32 ) and others ( 19 ) ) maintain stability but converge slowly, while REINFORCE-style approaches offer faster learning. CISPO ( 7 ) introduces clipped importance sampling to REINFORCE, 4 uses separate learning rates for positive/negative rewards, and 1 shows REINFORCE with careful tuning can match complex algorithms. However, these methods exhibit training instability ( 33 ; 4 ) , motivating our work. 3 Background We consider the RLVR setting for LLM reasoning, where \pi_{\theta} denotes an LLM policy parameterized by \theta that randomly predicts next token in the token space \mathcal{T} . Given a dataset \mathcal{D} of question-answer pairs (q,a) , where q\in\mathcal{T}^{*} , a\in\mathcal{T}^{*} , and \mathcal{T}^{*} is the space of token sequences, the model generates a response o by sampling tokens autoregressively: o=(o_{1},o_{2},\ldots,o_{|o|})\in\mathcal{T}^{|o|} , where o_{t}\sim\pi_{\theta}(\cdot|q,o_{0 indicates a correct response, whereas \hat{A}_{i,t}<0 indicates an incorrect one. Together, PPO-style clipping (Eq. 2 ) and the \min(\cdot) surrogate bound effective updates when r_{i,t}(\theta) leaves [1-\epsilon,\,1+\epsilon] , preserve a trust-region–like stability. We note that we omit the KL-regularization term in Eq. 1 for brevity. DAPO Algorithm DAPO ( 32 ) extends GRPO with (i) asymmetric clipping bounds \epsilon_{\text{high}}>\epsilon_{\text{low}} to promote exploration, (ii) dynamic sampling that filters out uninformative groups (e.g., all-correct or all-incorrect) before the update, (iii) token-level normalization via \frac{1}{\sum_{i=1}^{G}|o_{i}|} , and (iv) an overlong penalty term that discourages excessively long responses. The DAPO objective is given as: \!\!\!\!\!\!J_{\text{DAPO}}(\theta)=\mathbb{E}_{(q,a)\sim\mathcal{D},\,\{o_{i}\}_{i=1}^{G}\sim\pi_{\text{ref}}(\cdot\mid q)}\!\left[\frac{1}{\sum_{i=1}^{G}|o_{i}|}\sum_{i=1}^{G}\sum_{t=1}^{|o_{i}|}\min\!\big(r_{i,t}(\theta)\hat{A}_{i,t},\;r_{i,t}^{c}(\theta)\hat{A}_{i,t}\big)\right], (4) where the clipped IS weight r_{i,t}^{c}(\theta) is defined as r^{c}_{i,t}(\theta)=\operatorname{clip}\!\big(r_{i,t}(\theta);\;1-\epsilon_{\text{low}},\;1+\epsilon_{\text{high}}\big). (5) Similar to GRPO, the asymmetric window (\epsilon_{\text{low}},\epsilon_{\text{high}}) in DAPO preserves a trust-region–like stability. REINFORCE Algorithm The objective of off-policy REINFORCE at the token-level can be written as J_{\text{REINFORCE}}(\theta)=\mathbb{E}_{(q,a)\sim\mathcal{D},\,o_{i}\sim\pi_{\text{ref}}(\cdot\mid q)}\left[\frac{1}{|o_{i}|}\sum_{t=1}^{|o_{i}|}\operatorname{sg}\!\big(r_{i,t}(\theta)\big)\,A_{i,t}\log\pi_{\theta}(o_{i,t}\mid q,o_{i,0,\\[6.0pt] \operatorname{clip}\!\big(r_{i,t}(\theta);\,1-\epsilon^{-}_{\text{low}},\,1+\epsilon^{-}_{\text{high}}\big),&\hat{A}_{i,t}<0.\end{cases} (9) This decoupled clipping strategy provides fine-grained control over the four distinct policy update regimes in REINFORCE, allowing us to amplify/suppress gradients for both correct and incorrect responses. In Section 4.1 , we decompose these regimes systematically and present our ablation methodology to isolate their individual effects on training dynamics. We note that, following DAPO and CISPO, DISPO also incorporates dynamic sampling and an overlong penalty term in the loss calculation. Gradient-weight view We visualize how different objectives modulate the magnitude of the policy-gradient as a function of the importance ratio r_{i,t}(\theta) . For all the policy objectives discussed above, the gradient can be written as a function proportional to \operatorname{sg}(w_{i,t}(\theta)\,r_{i,t}(\theta))\,\nabla_{\theta}\log\pi_{\theta}(y_{i,t}\mid q,y_{i,0 and \hat{A}_{i,t}<0 , which yields asymmetric gradient weighting as a function of r_{i,t}(\theta) . Figure 3: Gradient weight w_{i,t}(\theta) as a function of the importance-sampling weight r_{i,t}(\theta) . 4.1 Analyzing the Policy Update Regimes In off-policy training, the IS weight r_{i,t}(\theta) is inherited from the policy updated in the previous step and influences the gradient at the current step. At the very first update, r_{i,t}(\theta)=1 , so the contribution of each token to the gradient of the DISPO objective in Eq. 8 takes the form \nabla_{\theta}J\propto\hat{A}_{i,t}\nabla_{\theta}\log\pi_{\theta}(o_{i,t}\mid q,o_{i,0 ) or incorrect ( \hat{A}_{i,t}<0 ), and (2) whether the IS weight amplifies ( r_{i,t}(\theta)>1 ) or suppresses ( r_{i,t}(\theta)<1 ) the gradient. This yields four distinct update regimes that we will describe in detail below: Regime 1: Amplified Positive Updates ( \hat{A}_{i,t}>0 , r_{i,t}(\theta)>1 ) This regime captures tokens in correct responses whose probabilities have increased relative to the reference policy during previous updates, as illustrated in Figure 4 . We can write the gradient contribution of these tokens as \nabla_{\theta}J\propto\operatorname{sg}\big(r_{i,t}^{d}(\theta)\big)\hat{A}_{i,t}\nabla_{\theta}\log\pi_{\theta}(o_{i,t}\mid q,o_{i,1 , this regime amplifies the positive learning signal beyond the baseline gradient in Eq. 10 , reinforcing the tokens that the model has already learned to favor. We note that increasing \epsilon^{+}_{\text{high}} in Eq. 9 allows greater amplification by permitting larger values of r_{i,t}^{d}(\theta) , while setting \epsilon^{+}_{\text{high}}=0 will clamp r_{i,t}^{d}(\theta) to 1 and revert to the baseline setting in Eq. 10 . We observe in our experiments that Regime 1 increases average token-level entropy, serving as a key driver of exploration during training (Section 5.3 ). Figure 4: DISPO’s four policy update regimes. Regime 2: Suppressed Positive Updates ( \hat{A}_{i,t}>0 , r_{i,t}(\theta)<1 ) Here, tokens in correct responses have decreased in probability relative to the reference policy, as shown in Figure 4 . The gradient expression is identical to Eq. 11 , but since r_{i,t}(\theta)<1 , the positive update signal is suppressed. We note that increasing \epsilon^{+}_{\text{low}} in Eq. 9 results in greater suppression by allowing smaller values of r_{i,t}^{d}(\theta) , while setting \epsilon^{+}_{\text{low}}=0 reverts the setting to the baseline in Eq. 10 . We observe that Regime 2 reduces average token-level entropy, acting as a distillation mechanism that consolidates learned patterns during training (Section 5.3 ). Regime 3: Amplified Negative Updates ( \hat{A}_{i,t}<0 , r_{i,t}(\theta)>1 ) This regime captures tokens in incorrect responses whose probabilities have increased relative to the reference policy during previous updates, as displayed in Figure 4 . The gradient contribution of the tokens becomes \nabla_{\theta}J\propto\operatorname{sg}\big(r_{i,t}^{d}(\theta)\big)\hat{A}_{i,t}\nabla_{\theta}\log\pi_{\theta}(o_{i,t}\mid q,o_{i,1 , this regime amplifies the negative learning signal, driving stronger unlearning of the tokens that the model has erroneously learned to favor. We note that increasing \epsilon^{-}_{\text{high}} allows greater amplification by permitting larger values of r_{i,t}^{d}(\theta) , and setting \epsilon^{-}_{\text{high}}=0 reverts the gradient to the baseline in Eq. 10 . We observe that insufficient amplification in Regime 3 leads to repetition-induced collapse, where the model fails to adequately unlearn erroneous patterns (Section 5.3 ). Regime 4: Suppressed Negative Updates ( \hat{A}_{i,t}<0 , r_{i,t}(\theta)<1 ) Finally, tokens in incorrect responses whose probabilities have decreased relative to the reference policy fall into this regime, as illustrated in Figure 4 . As in Eq. 12 , the negative advantage pushes the model to further reduce these probabilities, but since r_{i,t}(\theta)<1 , the unlearning signal is dampened. We note that increasing \epsilon^{-}_{\text{low}} results in greater suppression by allowing smaller values of r_{i,t}^{d}(\theta) , while setting \epsilon^{-}_{\text{low}}=0 reverts the gradient to the baseline in Eq. 10 . We observe that excessive suppression in Regime 4 causes response lengths to approach zero, indicating over-aggressive unlearning that disrupts the generation capability of the model (Section 5.3 ). To isolate each regime’s effect on training dynamics, we design controlled ablations by varying the four clipping parameters in Eq. 9 . Setting any parameter to zero disables its corresponding regime, while positive values control the strength of amplification or suppression. Table 1 presents our ablation configurations. For Regimes 1 and 2, our baseline sets \epsilon^{+}_{\text{low}}=\epsilon^{+}_{\text{high}}=0 and zeros gradients for incorrect responses, equivalent to online supervised fine-tuning (SFT). For Regimes 3 and 4, our baseline is DISPO with both regimes enabled, as both are necessary for stable training. To show their necessity, we disable each regime individually by setting \epsilon^{-}_{\text{high}}=0 (Regime 3) or \epsilon^{-}_{\text{low}}=0 (Regime 4). Table 1: Ablation configurations for analyzing update regimes in off-policy REINFORCE. For Regimes 1-2, we start from an online SFT baseline and enable each regime individually. For Regimes 3-4, we start from the full DISPO configuration and disable each regime individually. Configuration Response Type Clipping Parameters Active Regimes Starting from online SFT baseline (analysis of Regimes 1 and 2) Online SFT Baseline Correct only \epsilon^{+}_{\text{low}}=0,\,\epsilon^{+}_{\text{high}}=0 None ( \hat{r}_{i,t}=1 ) +Regime 1 Correct only \epsilon^{+}_{\text{low}}=0,\,\epsilon^{+}_{\text{high}}=0.28 Amplified Positive +Regime 1 Correct only \epsilon^{+}_{\text{low}}=0,\,\epsilon^{+}_{\text{high}}=10 Amplified Positive +Regime 2 Correct only \epsilon^{+}_{\text{low}}=0.2,\,\epsilon^{+}_{\text{high}}=0 Suppressed Positive +Regime 2 Correct only \epsilon^{+}_{\text{low}}=1,\,\epsilon^{+}_{\text{high}}=0 Suppressed Positive Starting from DISPO baseline (analysis of Regimes 3 and 4) DISPO (Full) Correct + Incorrect \epsilon^{-}_{\text{low}}=1,\,\epsilon^{-}_{\text{high}}=100 All regimes -Regime 3 Correct + Incorrect \epsilon^{-}_{\text{low}}=1,\,\epsilon^{-}_{\text{high}}=0 w/o Amplified Negative -Regime 4 Correct + Incorrect \epsilon^{-}_{\text{low}}=0,\,\epsilon^{-}_{\text{high}}=100 w/o Suppressed Negative 5 Results and Discussion We first briefly describe our experimental setup, followed by the main results comparing DISPO against baseline methods. We then examine each policy update regime in detail, highlighting insights that also informed the design of DISPO. 5.1 Experimental Setup We evaluate DISPO against PPO-style (DAPO) and REINFORCE-style (CISPO) baselines across diverse model sizes and architectures: Qwen3-8B-Base and Qwen3-14B-Base (both dense models), and Qwen3-30B-A3B-Base (MoE with 3.3B activated parameters). All models are trained on GSM8K, Math, and Mathematics ( 13 ; 23 ; 9 ) datasets, and evaluated on five mathematical reasoning benchmarks: AIME’24 ( 21 ) , AIME’25 ( 21 ) , AMC’23 ( 20 ) , MATH-500 ( 12 ) , and Minerva ( 17 ) . We use Qwen3-14B-Base for our ablation studies on policy update regimes. Moreover, we use the advantage formulation in Eq. 3 in all experiments, including the ablations for Regimes 1 and 2. Additional information about the baseline methods, training, and evaluation can be found in Appendix B . 5.2 DISPO vs. SOTA Methods DISPO outperforms the baselines significantly. Table 2 presents the evaluation of DISPO against the baselines across multiple mathematical reasoning benchmarks for all tested models. On the AIME’24 dataset, DISPO achieves substantial improvements: 61.04% accuracy on Qwen3-14B compared to 50.21% for DAPO and 55.42% for CISPO, representing a 10.83 percentage point improvement over DAPO. Similar patterns hold across other benchmarks, with DISPO showing particularly strong gains on competition-level problems (AIME’25: 45.83% vs. 38.96% for DAPO; AMC’23: 92.03% vs. 87.66% for DAPO). The improvements remain consistent across different model sizes and architectures—from the 8B dense model to the 30B mixture-of-experts (MoE) variant. Table 2: Comparison of DISPO with DAPO and CISPO across different model sizes and architectures. “–” denotes the performance of the starting checkpoint. The maximum values for each model and benchmark are highlighted in bold . Benchmark ( Avg@16 ) RL Algorithm – DAPO CISPO DISPO Model: Qwen3-14B-Base AIME’24 3.96 50.21 55.42 61.04 AIME’25 1.67 38.96 40.83 45.83 AMC’23 22.19 87.66 89.84 92.03 MATH-500 73.91 91.89 93.15 94.61 Minerva 36.10 45.22 45.66 46.78 Model: Qwen3-30B-A3B-Base (MoE) AIME’24 0.83 45.83 48.54 53.75 AIME’25 1.04 31.25 35.62 38.12 AMC’23 12.50 83.75 87.03 89.38 MATH-500 73.54 90.56 91.18 92.59 Minerva 30.35 43.45 44.26 44.60 Model: Qwen3-8B-Base AIME’24 4.58 38.33 43.54 45.00 AIME’25 2.71 27.50 30.83 31.46 AMC’23 24.69 80.94 80.94 87.66 MATH-500 71.22 91.25 90.74 92.34 Minerva 30.95 44.85 43.52 44.60 DISPO balances exploration and distillation while maintaining training stability. Figure 5 shows the learning curves of AIME’24 for DISPO and baseline methods. The entropy curves (bottom panel) reveal clear differences. We observe that CISPO loses entropy throughout training, whereas DISPO exhibits an increase. This difference is significant, as higher entropy indicates greater token-level exploration ( 29 ) , and it explains DISPO’s superior performance. DAPO also exhibits a similar rise in entropy, but its performance is significantly lower because it relies on token-clipped PPO rather than REINFORCE. Specifically, it discards tokens with low reference likelihood that serve as key entropy drivers—such as “but”, “aha”, and “since”—whose importance has been highlighted in prior work ( 7 ; 29 ) . Learning curves for additional models (Qwen3-30B-A3B-Base and Qwen3-8B-Base) are shown in Appendix C , and they show similar patterns. Response length curve for Qwen3-14B-Base can also be found in Appendix C . Regarding clipping bounds, we adopt the default values for DAPO: \epsilon_{\text{low}}=0.2 and \epsilon_{\text{high}}=0.28 . Since the original CISPO clipping bounds were not released, we implemented it with \epsilon_{\text{low}}=1 and \epsilon_{\text{high}}=100 , which proved stable across all evaluated models. Using either \epsilon_{\text{low}}<1 or \epsilon_{\text{high}}<100 in CISPO caused sudden performance collapse. Note that \epsilon=100 effectively disables clipping while still preventing infinite importance weights. For DISPO, we set \epsilon^{+}_{\text{low}}=0.2 , \epsilon^{+}_{\text{high}}=10 , \epsilon^{-}_{\text{low}}=1 , and \epsilon^{-}_{\text{high}}=100 . In Section 5.3 , we examine how these four \epsilon parameters shape training dynamics through their corresponding policy update regimes, explain the rationale behind our DISPO hyperparameter choices, and provide insights into the previously reported CISPO collapses ( 33 ) . Figure 5: Accuracy and entropy curves of DAPO, CISPO, and DISPO. 5.3 Analysis of Policy Update Regimes in Off-Policy REINFORCE Increasing \epsilon^{+}_{\text{high}} in Regime 1 improves exploration. Figure 6 shows the impact of allowing r_{i,t}>1 through different \epsilon^{+}_{\text{high}} values while maintaining \epsilon^{+}_{\text{low}}=0 . The online SFT baseline (blue) shows stable but slow accuracy improvement, while both models with \epsilon^{+}_{\text{high}}>0 demonstrate faster gains, indicating improved training efficiency. The entropy curves reveal the underlying mechanism: the baseline maintains constant entropy throughout training, whereas configurations with \epsilon^{+}_{\text{high}}>0 exhibit increasing entropy, reflecting progressive exploration. As we discussed in Section 5.2 , amplifying gradient for low-reference-likelihood tokens in correct responses leads to more effective token utilization and increased exploration. Figure 6: Accuracy and entropy curves of Regime 1 runs. ⋆ marks the maximum value. Excessive exploration causes gradual performance degradation. The increased exploration eventually becomes detrimental in later training stages. Both models with \epsilon^{+}_{\text{high}}>0 show accuracy degradation after peaking, as excessive exploration leads to sampling increasingly unlikely tokens that harm reasoning coherence. This reveals a critical trade-off: amplification enhances learning efficiency through beneficial exploration, but excessive amplification causes uncontrolled exploration and gradual performance decline later in training. The model with \epsilon^{+}_{\text{high}}=0.28 provides more controlled exploration than \epsilon^{+}_{\text{high}}=10 —entropy increases more gradually, enabling longer training before degradation and higher peak accuracy. This demonstrates that carefully tuning \epsilon^{+}_{\text{high}} preserves efficiency benefits while mitigating uncontrolled exploration risks. We note that DISPO in Figure 5 might also experience accuracy decline with extended training due to Regime 1, though computational constraints prevented us form verifying it. Importantly, in Regime 1, instability manifests itself as gradual degradation rather than sudden collapse. Increasing \epsilon^{+}_{\text{low}} in Regime 2 improves distillation. Figure 7 shows the impact of varying \epsilon^{+}_{\text{low}} while maintaining \epsilon_{\text{high}}=0 . We see that allowing r_{i,t}<1 yields efficiency gains compared to the online SFT baseline. As shown in Figure 7 , Regime 2 decreases entropy—opposite to Regime 1’s exploratory behavior—indicating the model actively prunes its token vocabulary. This entropy reduction reflects a distillation mechanism: by attenuating the learning signal for tokens with decreased probabilities ( r_{i,t}<1 ), the model filters out less reliable tokens even within correct responses. While some suppressed tokens may contain useful patterns, they likely include noise or suboptimal solution paths. This selective reinforcement accelerates learning by focusing on high-probability tokens that form the core problem-solving strategy, rather than indiscriminately reinforcing all tokens in correct solutions. Comparing \epsilon^{-}_{\text{low}}=0.2 versus 1.0 , the former enables more controlled distillation through gradual entropy reduction, achieving slightly higher peak accuracy. Figure 7: Accuracy and entropy curves of Regime 2 runs. ⋆ marks the maximum value. Distillation offers limited standalone benefits but effectively counterbalances excessive exploration. While Regime 2 achieves efficiency gains (Figure 7 ), these improvements remain modest compared to Regime 1’s exploration-driven performance (Figure 6 ). The decreasing entropy in Regime 2 signals premature convergence to limited solution strategies, as the model trades exploratory capacity for consistency and potentially misses innovative approaches requiring lower-probability tokens. This explains why pure distillation yields lower peak accuracy—exploration proves more crucial than consolidation for superior reasoning performance. Response length curves for both regimes appear in Appendix C . Notably, when Regimes 1 and 2 operate simultaneously (Appendix D ), their competing entropy effects create a balanced exploration-distillation dynamic that achieves higher accuracy than either regime can achieve alone, demonstrating that distillation’s primary value lies in moderating exploration rather than serving as a standalone strategy. In Regime 3, setting \epsilon^{-}_{\text{high}}>0 is necessary to prevent repetition collapse. Figure 8 shows that disabling Regime 3 ( \epsilon^{-}_{\text{high}}=0 , orange) causes sudden accuracy collapse and response length spike early in training, compared to full DISPO (blue). Regime 3 targets incorrect tokens whose probabilities exceed the reference policy ( r_{i,t}>1 ). Without Regime 3’s amplification, these tokens receive insufficient negative gradients for unlearning—since |\nabla_{\theta}\log\pi_{\theta}| naturally decreases as \pi_{\theta}\to 1 , high-probability tokens already have weak gradients. In essence, the model loses the ability to “forget" its mistakes, as the unlearning signal becomes too weak to overcome the reinforcement from previous updates. Setting \epsilon^{-}_{\text{high}}=0 caps r_{i,t} at 1, further weakens these gradients and prevents adequate penalization. Consequently, the model repeatedly generates these incorrect high-probability tokens, causing the repetition-driven length spike in Figure 8 (bottom). Figure 13 in Appendix C shows a representative example. We hypothesize that this is one of the reasons behind previously reported CISPO collapses ( 33 ) —our experiments confirm that \epsilon^{-}_{\text{high}}<1 causes CISPO to fail with similar repetition patterns in our experimental setup. Figure 8: Accuracy and response length curves of runs in which Regimes 3 and 4 are disabled. In Regime 4, setting \epsilon^{-}_{\text{low}}>0 prevents length collapse from excessive unlearning. Figure 8 shows rapid deterioration in both accuracy and response length when Regime 4 is disabled ( \epsilon^{-}_{\text{low}}=0 , green). Regime 4 governs incorrect tokens with probabilities below the reference policy ( r_{i,t}<1 ). These low-probability tokens naturally receive strong negative gradients since |\nabla_{\theta}\log\pi_{\theta}| grows as \pi_{\theta}\to 0 . Setting \epsilon^{-}_{\text{low}}=0 removes suppression—allowing r_{i,t} below 1 amplifies these already-strong gradients, causing excessive penalization that drives response length toward zero (Figure 8 , bottom). This over-penalization essentially teaches the model to “give up" on generation rather than learn correct patterns. Thus, Regime 4’s gradient suppression prevents over-correction on already-unlikely tokens. We hypothesize this is another reason behind CISPO collapses ( 33 ) , as our experiments show \epsilon^{-}_{\text{low}}<100 causes similar length collapse in CISPO. 6 Conclusion We presented DISPO, a simple yet effective modification to off-policy REINFORCE that decouples the clipping of importance sampling weights for correct and incorrect responses. Through systematic ablations of the four resulting policy update regimes, we identified distinct failure modes in off-policy REINFORCE-style methods. Our results demonstrate that properly balancing these regimes promotes exploration-distillation balance while maintaining training stability, achieving superior performance over existing methods. Appendix A Limitations and Future Work While DISPO demonstrates significant improvements over existing methods, several limitations warrant further investigation. First, our experiments focus primarily on mathematical reasoning tasks where binary reward signals are straightforward to obtain. Extending DISPO to domains with more nuanced reward structures, such as code generation or open-ended dialogue, remains unexplored. Second, although we provide empirical guidelines for setting the four clipping parameters ( \epsilon^{+}_{\text{high}} , \epsilon^{+}_{\text{low}} , \epsilon^{-}_{\text{high}} , \epsilon^{-}_{\text{low}} ), determining optimal values still requires some trial and error. Future work could explore adaptive or learned clipping schedules that automatically adjust these parameters based on training dynamics. Additionally, our analysis reveals that excessive exploration in Regimes 1 and 2 can lead to gradual performance degradation in later training stages. While we mitigate these through careful parameter tuning, developing principled methods to detect and prevent such degradations—perhaps through entropy regularization or dynamic exploration schedules—would be valuable. Finally, due to computational constraints, we limited our experiments to models up to 30B parameters. Future work could investigate DISPO’s scalability to larger models as the field moves toward even bigger reasoning systems. Appendix B Experimental Details B.1 RLVR Baselines We evaluate DISPO against representatives from both PPO and REINFORCE families of algorithms. We select DAPO as our PPO-style baseline and employ our own CISPO implementation as the REINFORCE-style baseline. Since the original CISPO’s clipping parameters remain unpublished, we configure CISPO with \epsilon_{\text{low}}=1 and \epsilon_{\text{high}}=100 , settings that maintain stability across all tested models. Notably, reducing these values (setting \epsilon_{\text{low}}<1 or \epsilon_{\text{high}}<100 ) triggers training collapses in CISPO. For DAPO, we retain the standard clipping parameters: \epsilon_{\text{low}}=0.2 and \epsilon_{\text{high}}=0.28 . We focus on a single PPO-style baseline given that prior work demonstrates CISPO’s superiority over various PPO methods, including both DAPO and GRPO [ 7 ] . B.2 Training We use the chat template shown in Figure 9 for training. Following the DAPO recipe [ 32 ] , we apply dynamic sampling in all training runs. We use a mini-batch size of 512 and a micro-batch size of 32, corresponding to 16 gradient updates per mini-batch. We set the group size to G=16 . We do not include a KL divergence regularization term in the loss calculation. Our max response length is 20,480 tokens. We use the same overlong penalty as in the DAPO recipe in all our experiments. We use AdamW optimizer with learning rate of 1\times 10^{-6} for all models. In AdamW, we set (\beta_{1},\beta_{2})=(0.9,0.95) , \epsilon=1\mathrm{e}{-15} , and a weight decay of 0.1, following the CISPO recipe [ 7 ] . We apply gradient norm clipping at 1.0 and use early truncation with repetition detection, as introduced in the CISPO recipe. All models were trained on H200 GPUs using the Verl codebase with vLLM rollout, requiring approximately 20,000 GPU-hours each. During training, we set the temperature to 1.0 and top-p to 1.0, while for inference we used temperature = 1.0 and top-p = 0.7 across all models. For inference on AIME’24, AIME’25, and AMC’23, we used the system prompt shown in Figure 9 , while for MATH-500 and Minerva we used the Qwen3-style system prompt shown in Figure 10 to ensure consistent parsing. Figure 9: The DAPO-style chat template. Figure 10: The Qwen3-style chat template. B.3 Evaluation We evaluate our models using five widely recognized mathematical reasoning benchmarks, namely, AIME’24 [ 21 ] , AIME’25 [ 21 ] , AMC’23 [ 20 ] , MATH-500 [ 12 ] , and Minerva [ 17 ] . These benchmarks assess the model’s capacity to solve complex problems across diverse domains and levels of difficulty. Each problem requires the generation of a final answer, which is usually a number, a simplified expression (e.g., p-q ), or a concise textual response (e.g., even ): \triangleright AIME’24 : 30 problems from the American Invitational Mathematics Examination in 2024 [ 21 ] . Each problem typically requires multiple steps of intricate reasoning and has an answer that is an integer between 0 and 999. \triangleright AIME’25: A forthcoming set of 30 problems from the AIME 2025 exam [ 21 ] , covering a similar range of topics in algebra, geometry, number theory, and combinatorics. \triangleright AMC’23: 40 problems from the American Mathematics Competitions in 2023 [ 20 ] . This exam serves as a precursor to the AIME and includes problems designed to test creative problem-solving across algebra, geometry, number theory, and probability. \triangleright MATH-500: A curated subset of 500 problems drawn from the MATH dataset [ 12 ] , selected as 18 . These problems cover seven subjects: prealgebra, algebra, number theory, counting and probability, geometry, intermediate algebra, and precalculus. \triangleright Minerva: A benchmark introduced by 17 , consisting of 272 advanced quantitative reasoning problems drawn from diverse sources such as research-level mathematics and science exams. To evaluate the correctness of the model’s outputs, we follow standard practices in mathematical LLM evaluation. We parse each generated solution using regular expressions to extract the final answer and compare it against the ground truth. For every question in the evaluation set, we generate 16 responses and report the average accuracy (denoted as Avg@16). We track the models’ Avg@16 accuracy on AIME’24 throughout training and select the checkpoint with the highest accuracy for final evaluation on all benchmarks. In addition, we compute auxiliary metrics—including token-level average entropy and response length—using the same set of 16 generated completions. Appendix C Additional experimental results Figures 11 and 12 show all learning curves of Qwen3-30B-A3B-Base and Qwen3-8B-Base, respectively. We see similar characteristics in all curves as in Figure 5 , showing the robustness of DISPO across different model sizes and architectures. Figure 11: Learning curves of the Qwen3-30B-A3B-Base runs. ⋆ indicates the maximum accuracy. Figure 12: Learning curves of the Qwen3-8B-Base runs. ⋆ indicates the maximum accuracy. Figure 13 shows an example of a token that is repeated in inference time, as mentioned in Regime 3 discussion in Section 5.3 . Figure 13: An example of Regime 3 degradation: when r_{i,t}(\theta)>\epsilon_{\text{high}}^{-} , Regime 3 update is suppressed, causing the token to become trapped in the high-probability region and repeatedly generated during evaluation. Figure 14 shows the response length curve of Qwen3-14B-Base. The response length curves of Regime 1 and Regime 2 runs are presented in Figures 15 and 16 , respectively. Figure 14: Response length curves of the Qwen3-14B-Base runs. Figure 15: Response length curves of the Regime 1 runs. Figure 16: Response length curves of the Regime 2 runs. Appendix D Additional discussion about Regimes 1 and 2 Figure 17 shows the results when we allow both r_{i,t}>1 and r_{i,t}<1 by setting both \epsilon^{+}_{\text{low}} and \epsilon^{+}_{\text{high}} to be non-zero. This configuration enables the model to simultaneously amplify learning signals (Regime 1) and reduce recovery signals (Regime 2), creating an interplay between exploration and distillation mechanisms and resulting in higher accuracy. The entropy dynamics reveal this competition between Regimes 1 and 2 clearly. The green curve ( \epsilon^{+}_{\text{low}}=0.28,\epsilon^{+}_{\text{high}}=1 ) in Figure 17 exhibits slower entropy growth compared to the orange curve (Regime 1 only with \epsilon^{+}_{\text{low}}=0.28 ), demonstrating how the simultaneous activation of Regime 2 counteracts the entropy-increasing effect of Regime 1. Figure 17: Learning curves of the Regime 1 + Regime 2 run. ⋆ in the accuracy panel indicates the maximum value.