Distributionally Robust Token Optimization in RLHF Yeping Jin, Jiaming Hu, Ioannis Ch. Paschalidis Abstract Large Language Models (LLMs) tend to respond correctly to prompts that align to the data they were trained and fine-tuned on. Yet, small shifts in wording, format, or language can trigger surprisingly large failures, especially on multi-step reasoning problems. To address this problem, we propose a Distributionally Robust Token Optimization (DRTO) approach, which combines token-level Reinforcement Learning from Human Feedback (RLHF) with Distributionally Robust Optimization (DRO) . DRTO bounds worst case token-wise rewards by constructing an f-divergence ambiguity set over a loss minibatch, leading to a theoretical robustness. Empirically, DRTO enhances consistency under distribution shifts in mathematical reasoning benchmarks, achieving 9.17% improvement on GSM8K and 2.49% improvement on MathQA . Our anonymized code is available at OSF . 1 Introduction Large language models (LLMs) have achieved strong performance on a wide range of language and reasoning tasks, and reinforcement learning from human feedback (RLHF) has become a practical and widely used approach to align these models with human preferences. In modern post-training pipelines, two mainstream choices are Proximal Policy Optimization (PPO) ( 18 ) and Direct Preference Optimization (DPO) ( 13 ) . Despite their success, both methods can be vulnerable to distributional shifts between the preference data used for alignment and the prompts or evaluation conditions encountered at deployment. In practice, such vulnerability may appear as training instability and reduced robustness. Small changes in phrasing, notation, or other superficial input variations can lead to substantial drops in accuracy. Due to this instability issue, recent work has begun to modify PPO and DPO models to enhance robustness ( 21 ; 9 ) . However, existing robust variants are often limited by increased conservatism or additional computational overhead, and they usually do not fully exploit the token-level credit assignment. As a result, they may struggle to improve out-of-distribution robustness without sacrificing strong in-distribution performance, especially on reasoning-intensive tasks. To address these limitations, we develop a distributionally robust extension of a recent token-level alignment framework, Reinforced Token Optimization (RTO) ( 24 ) , which already demonstrates consistent advantages over PPO and DPO by combining PPO-style policy optimization with a token-wise reward signal learned from preferences. We propose Distributionally Robust Token Optimization (DRTO), which adapts Distributionally Robust Optimization (DRO) to the RTO actor update through divergence-based ambiguity sets, including KL and \chi^{2} divergence. Such formulation yields more stable updates and improves invariance against out-of-distribution datasets, while retaining RTO’s performance advantages over DPO and PPO. Figure 1 : Visualization of DRTO performance on five benchmarks under distribution shifts. 1.1 Contributions This work enhances the robustness of token-level RLHF via a DRO approach. We summarize our main contributions as follows. • Distributionally robust token-level objectives. We introduce minibatch-level ambiguity sets over trajectory samples and derive robust objectives for the actor loss. To the best of our knowledge, this is the first work to apply DRO at the minibatch level within a token-level RLHF framework. This approach inherits the reasoning advantages of token-level alignment while significantly improving robustness. • Realization of DRO. We use two distinct f-divergences to measure the ambiguity set, leading to two different practical realizations. For KL-DRTO, we obtain an entropic objective that smoothly emphasizes high-loss trajectories through exponential tilting; for \chi^{2} -DRTO, we obtain a tractable surrogate based on the minibatch mean and standard deviation. • Empirical improvements under distribution shifts. Our practical implementations of KL-DRTO and \chi^{2} -DRTO use the same training pipeline as standard RTO, with little to no additional runtime or compute cost. Empirically, both methods yield more consistent performance under linguistic and symbolic shifts on math reasoning tasks, while preserving strong in-domain performance. Across five benchmarks, DRTO attains up to +11.83% in relative accuracy improvement over standard RTO, as indicated in Figure 1 . Subsequent ablation studies further validate DRTO’s efficiency. In summary, we develop a theoretically grounded and practically effective approach for robust token-level RLHF, which also inherits the reasoning advantages of token-level methods over standard PPO and DPO. 1.2 Related Work Reinforcement Learning from Human Feedback. RLHF ( 11 ; 1 ) is widely used to align LLMs with human preferences. Among all popular frameworks, PPO ( 18 ) remains a common choice for reinforcement-based fine-tuning, typically with a KL control term to limit deviation from a reference policy. As an alternative mainstream approach, DPO ( 13 ) reformulates preference optimization as a supervised objective on pairwise comparisons, avoiding explicit on-policy rollouts. Token-level RLHF. To enhance the two popular methods with sequence-level objectives, token-level alignment methods aim to improve credit assignment and make training less brittle. For instance, Token-level DPO (TDPO) ( 23 ) introduces a token-wise formulation of the preference objective. In this paper, we build our work on Reinforced Token Optimization (RTO) ( 24 ) , which adds a token-level reward that involves DPO model in the standard PPO framework, enabling more localized learning signals. Even so, token-level training can be sensitive to reward noise and minibatch composition, which motivates robust objectives that enhance stability for such uncertainty. Distributionally robust optimization for preference alignment. DRO provides a principled framework for learning under distributional uncertainty ( 3 ; 14 ; 4 ) , and divergence-based ambiguity sets yield tractable dual forms linked to variance-sensitive and entropic-risk objectives ( 10 ) . Recent alignment work has applied DRO to offline preference optimization: DrDPO ( 21 ) robustifies DPO against data noise via distributionally robust reweighting, and \chi^{2} -Preference Optimization ( 9 ) addresses offline over-optimization via \chi^{2} -based regularization. Group-robust preference optimization (GRPO) ( 15 ) maximizes worst-group performance when group labels are available. In contrast, DRTO enhances the robustness of on-policy RTO actor loss with either \chi^{2} ambiguity sets or KL-regularizations, where both methods directly stabilize token-level policy-gradient updates under reward perturbations and distribution shift, while they also inherit RTO’s advantages over DPO and PPO under in-distribution settings. Robustness and safety beyond DRO. A complementary line of work improves the model’s robustness and safety under adversarial prompting. Some methods exploit internal signals of the target model, such as GradSafe ( 22 ) and Gradient Cuff ( 7 ) , and they analyze safety-critical parameters or refusal loss landscapes by computing gradients with respect to input prompts, identifying adversarial patterns that resemble known unsafe behaviors. Robust Prompt Optimization (RPO) ( 25 ) formulates the construction of a protective suffix as a minimax defense objective and optimizes the suffix with gradient-based token optimization. Despite their methodological diversity, these approaches require white-box access, which is unrealistic for most closed-source LLMs. The limitation motivates alternatives that act at the interface without internal-state access. For instance, SmoothLLM ( 16 ) perturbs each input prompt multiple times at the character level (e.g., substitutions, insertions, deletions) to generate several noisy variants and aggregates the target model’s responses to decide whether the original prompt is malicious, and CCFC ( 6 ) proposes a dual-track prompt-level defense for jailbreak protection. These approaches are orthogonal to DRTO: they target the robustness of models for safety issues, whereas DRTO enhances the consistent performance of models on distribution shift problems. 2 Preliminaries In this section, we introduce the standard token-level optimization framework for RLHF and establish the notation used throughout the paper. 2.1 Proximal Policy Optimization In the classical RLHF pipeline, we define a policy \pi_{\theta} as a mapping from an input prompt to a distribution over output responses. Given a prompt x\sim\mathcal{X} , the policy induces a conditional distribution \pi_{\theta}(\cdot\mid x) and generates a response y\sim\pi_{\theta}(\cdot\mid x) . We would like to maximize a learned sentence-level reward while keeping the policy close to a reference model \pi_{\mathrm{ref}} . A common target objective is the KL-regularized form \max_{\theta}\ \mathbb{E}_{x\sim\mathcal{X},\ y\sim\pi_{\theta}(\cdot\mid x)}\left[r(x,y)\;-\;\beta\log\frac{\pi_{\theta}(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}\right], (1) where r(x,y) is a learned reward to evaluate the responses, and \beta>0 sets the strength of the KL penalty. Instead of directly optimizing ( 1 ), PPO performs clipped policy updates using on-policy rollouts and an advantage estimator. Given a prompt x , the policy \pi_{\theta} generates a response y=(y_{1},\ldots,y_{H}) token by token. At position h\in\{1,\ldots,H\} , we denote the response prefix generated up to position h as y_{1:h-1} . At each position, the policy samples y_{h}\sim\pi_{\theta}(\cdot\mid x,y_{1:h-1}) over the vocabulary \mathcal{V} . A rollout produces a full response together with its prefixes, where we will write such a rollout as \tau=(x,y) . For the i -th rollout with length H_{i} , we denote [H_{i}]:=\{1,\ldots,H_{i}\} as the answer-token positions at which we apply the trajectory loss. Given a minibatch of rollouts \mathcal{B}=\{\tau_{i}\}_{i=1}^{n} collected under \pi_{\theta} , we define the token-level state and action at position h as s_{i,h}:=(x_{i},y_{i,1:h-1}) and a_{i,h}:=y_{i,h} , respectively. For each trajectory pair, we quantify the policy difference between actor model \pi_{\theta} and reference model \pi_{\mathrm{ref}} by the policy ratio \rho_{i,h}(\theta)=\frac{\pi_{\theta}(a_{i,h}\mid s_{i,h})}{\pi_{\mathrm{ref}}(a_{i,h}\mid s_{i,h})}. Given a discount factor \gamma\in(0,1] , we denote the return from step h as G_{i,h}:=\sum_{t=h}^{H_{i}}\gamma^{t-h}r_{i,t}. For each rollout i , its advantage at step h is defined as A^{\pi}(s,a)\!:=\!\mathbb{E}_{\pi}\!\left[G_{i,h}\!\mid\!s_{i,h}\!=\!s,a_{i,h}\!=\!a\right]-\mathbb{E}_{\pi}\!\left[G_{i,h}\!\mid\!s_{i,h}=s\right], and we compute the advantage estimate \widehat{A}_{i,h} by GAE estimate ( 17 ) from the observed rewards. Then the PPO clipped surrogate defines the rollout loss \displaystyle L_{i}(\theta) \displaystyle:=-\frac{1}{H_{i}}\sum_{h\in[H_{i}]}\min\!\Big(\rho_{i,h}(\theta)\widehat{A}_{i,h}, (2) \displaystyle\mathrm{clip}\big(\rho_{i,h}(\theta),1-\epsilon,1+\epsilon\big)\widehat{A}_{i,h}\Big), where \epsilon>0 is the clipping parameter. In the upcoming sections, we will use PPO-Update to represent the update of minimizing the actor loss, which has the default setting as the average PPO loss L_{\mathrm{PPO}}(\theta):=\frac{1}{n}\sum_{i=1}^{n}L_{i}(\theta). 2.2 Reinforced Token Optimization RTO ( 24 ) is a token-level RLHF framework that keeps the PPO actor update above but uses a dense token-wise shaping reward to construct the advantage \widehat{A}_{i,h} . We first train a preference policy \pi_{\mathrm{dpo}} from offline preference data. We then start from a reference policy \pi_{\mathrm{ref}} and optimize the actor with PPO updates driven by a token-level shaping reward. This shaping reward is based on log-likelihood ratios relative to \pi_{\mathrm{dpo}} and \pi_{\mathrm{ref}} , with an additional terminal sentence-level reward. At an answer-token position h\in[H_{i}] , the shaping reward is \displaystyle r^{\mathrm{RTO}}_{h}(x,y_{1:h})=\beta_{1}\log\frac{\pi_{\mathrm{dpo}}(y_{h}\mid x,y_{1:h-1})}{\pi_{\mathrm{ref}}(y_{h}\mid x,y_{1:h-1})} (3) \displaystyle-\beta_{2}\log\frac{\pi_{\theta}(y_{h}\mid x,y_{1:h-1})}{\pi_{\mathrm{ref}}(y_{h}\mid x,y_{1:h-1})}+\beta_{3}\mathbf{1}_{\{h=H_{i}\}}\,r(x,y_{1:H_{i}}), where \beta_{1},\beta_{2},\beta_{3}>0 are tuning parameters and the last term is the terminal reward that is computed only if the current trajectory reaches the full length. Algorithm 1 formulates the procedure. At each iteration, we use DPO model to compute token-wise rewards as ( 3 ) and apply the PPO update. Algorithm 1 Standard RTO 1: Input: offline dataset \mathcal{D} ; \beta_{1},\beta_{2},\beta_{3}>0 ; DPO Alogirthm DPO ; PPO trainer PPO-Update . 2: Train \pi_{\mathrm{dpo}}\leftarrow\texttt{DPO}(\mathcal{D}) ; initialize \pi_{0}=\pi_{\mathrm{ref}} . 3: for t=1,\ldots,T do 4: Sample a batch of prompts D_{t} . 5: For each x\in D_{t} , generate response y\sim\pi_{t-1}(\cdot\mid x) . 6: Compute token-wise rewards \{r^{\mathrm{RTO}}_{h}(x,y_{1:h})\}_{h\in[H]} by ( 3 ). 7: Update \pi_{t}\leftarrow\texttt{PPO-Update}(\pi_{t-1},r^{\mathrm{RTO}},D_{t}) . 8: end for 9: Output: \pi_{T} . 3 Theoretical Results We now formulate a minibatch DRO objective that models distribution shift via adversarial reweighting over trajectories, which will serve as the starting point for the KL and \chi^{2} formulations. Our key insight is to allow an adversary to reweight the n trajectories in the minibatch. Let \mathbb{P}_{\mathcal{B}} denote the uniform distribution on indices \{1,\ldots,n\} . Given an ambiguity set \Omega(\mathbb{P}_{\mathcal{B}}) around \mathbb{P}_{\mathcal{B}} , DRTO replaces the mean surrogate loss with the robust value R(\theta;\rho):=\sup_{\mathbb{Q}\in\Omega(\mathbb{P}_{\mathcal{B}})}\ \mathbb{E}_{\tau_{i}\sim\mathbb{Q}}\big[L_{i}(\theta)\big], (4) which reduces to L_{\mathrm{PPO}}(\theta) when the ambiguity radius is \rho=0 . In the following sections, we instantiate ( 4 ) using KL and Pearson \chi^{2} divergence, yielding tractable objectives for KL-DRTO and \chi^{2} -DRTO. 3.1 DRTO on a KL-Divergence Ambiguity Set We first instantiate the minibatch DRO objective in ( 4 ) to a KL-divergence ambiguity set. Throughout this subsection, the per-trajectory PPO loss L_{i}(\theta) is defined in ( 2 ), and \mathbb{P}_{\mathcal{B}} denotes the uniform distribution over the n full-length trajectories in the minibatch \mathcal{B} . Given radius \rho\geq 0 , we describe the KL-ball around \mathbb{P}_{\mathcal{B}} as \Omega^{\mathrm{KL}}_{\rho}(\mathbb{P}_{\mathcal{B}})=\left\{\mathbb{Q}:\ D_{\mathrm{KL}}(\mathbb{Q}\|\mathbb{P}_{\mathcal{B}})\leq\rho\right\}. (5) On a finite minibatch, the inner maximization in ( 4 ) admits an exact one-dimensional dual ( 8 ) . For any \eta>0 , we define the entropic risk \displaystyle\Psi_{\eta}(\theta) \displaystyle:=\frac{1}{\eta}\log\mathbb{E}_{\tau_{i}\sim\mathbb{P}_{\mathcal{B}}}\left[\exp\!\left(\eta\,L_{i}(\theta)\right)\right] (6) \displaystyle=\frac{1}{\eta}\log\left(\frac{1}{n}\sum_{i=1}^{n}e^{\eta L_{i}(\theta)}\right), and the corresponding softmax weights \mathbb{Q}_{\eta}(i):=\frac{\exp(\eta L_{i}(\theta))}{\sum_{j=1}^{n}\exp(\eta L_{j}(\theta))},\qquad i\in\{1,\ldots,n\}. (7) Algorithm 2 KL-DRTO 1: Input: offline dataset \mathcal{D} ; \beta_{1},\beta_{2},\beta_{3},\rho>0 ; DPO algorithm DPO ; PPO trainer PPO-Update . 2: Train \pi_{\mathrm{dpo}}\leftarrow\texttt{DPO}(\mathcal{D}) ; initialize \pi_{0}=\pi_{\mathrm{ref}} . 3: for t=1,\ldots,T do 4: Sample a batch of prompts D_{t} . 5: For each x\in D_{t} , generate response y\sim\pi_{t-1}(\cdot\mid x) . 6: Compute token-wise rewards \{r^{\mathrm{RTO}}_{h}(x,y_{1:h})\}_{h\in[H]} by ( 3 ). 7: Compute PPO losses \{L_{i}(\theta)\}_{i=1}^{n} by ( 2 ). 8: Compute \eta^{\star}=\arg\min_{\eta>0}\Bigl\{\Psi_{\eta}(\theta)+\rho/\eta\Bigr\} by ( 6 ). 9: Compute optimal weights \mathbb{Q}^{\star}(i)=\mathbb{Q}_{\eta^{\star}}(i) by ( 7 ). 10: Replace the PPO loss with the KL-DRTO loss L_{\mathrm{KL}}(\theta)=\sum_{i=1}^{n}\mathbb{Q}^{\star}(i)\,L_{i}(\theta). 11: Update \pi_{t}\leftarrow\texttt{PPO-Update}\!\left(\pi_{t-1},r^{\mathrm{RTO}},L_{\mathrm{KL}},D_{t}\right) . 12: end for 13: Output: \pi_{T} . Theorem 3.1 (Guarantee of KL-DRTO) . Fix any minibatch \mathcal{B} of size n and any radius \rho\in[0,\log n] . Assume L_{i}(\theta) is finite for all i\in\{1,\ldots,n\} . Then the KL-robust value ( 4 ) admits the exact dual representation R_{\mathrm{KL}}(\theta;\rho)=\inf_{\eta>0}\left\{\Psi_{\eta}(\theta)+\frac{\rho}{\eta}\right\}. (8) Moreover, if the infimum in ( 8 ) is attained at some \eta^{\star}\in(0,\infty) , then the corresponding worst-case reweighting is \mathbb{Q}^{\star}=\mathbb{Q}_{\eta^{\star}} , and R_{\mathrm{KL}}(\theta;\rho)=\sum_{i=1}^{n}\mathbb{Q}^{\star}(i)\,L_{i}(\theta). (9) Proof. See Appendix A.1 . ∎ Theorem 3.1 reduces the inner KL-ball adversary in ( 4 ) from an n -dimensional constrained maximization to a scalar minimization over \eta>0 and yields an explicit worst-case reweighting. As a result, Algorithm 2 implements KL-DRTO by computing \eta^{\star} and \mathbb{Q}^{\star} from the minibatch losses \{L_{i}(\theta)\}_{i=1}^{n} , and then running RTO with the weighted loss L_{\mathrm{KL}}(\theta)=\sum_{i=1}^{n}\mathbb{Q}^{\star}(i)\,L_{i}(\theta) . 3.2 DRTO on a \chi^{2} -divergence based ambiguity set We now derive the DRO formulation in ( 4 ) with the ambiguity set measured by the Pearson \chi^{2} -divergence ( 12 ) . To begin with, we still start from the setup in Section 3.1 , where the trajectory loss L_{i}(\theta) is defined in ( 2 ). For a radius \rho\geq 0 , we describe the Pearson \chi^{2} ball around \mathbb{P}_{\mathcal{B}} as \Omega^{\chi^{2}}_{\rho}(\mathbb{P}_{\mathcal{B}})=\left\{\mathbb{Q}:\ \mathbb{E}_{\tau_{i}\sim\mathbb{P}_{\mathcal{B}}}\left[\left(\frac{\mathbb{Q}(\tau_{i})}{\mathbb{P}_{\mathcal{B}}(\tau_{i})}-1\right)^{2}\right]\leq\rho\right\}, (10) and the corresponding \chi^{2} -robust minibatch value is R_{\chi^{2}}(\theta;\rho):=\sup_{\mathbb{Q}\in\Omega^{\chi^{2}}_{\rho}(\mathbb{P}_{\mathcal{B}})}\ \mathbb{E}_{\tau_{i}\sim\mathbb{Q}}\!\left[L_{i}(\theta)\right]. (11) Algorithm 3 \chi^{2} -DRTO 1: Input: offline dataset \mathcal{D} ; \beta_{1},\beta_{2},\beta_{3}>0 , \rho,\delta\geq 0 ; DPO Algorithm DPO ; PPO trainer PPO-Update . 2: Train \pi_{\mathrm{dpo}}\leftarrow\texttt{DPO}(\mathcal{D}) ; initialize \pi_{0}=\pi_{\mathrm{ref}} . 3: for t=1,\ldots,T do 4: Sample a batch of prompts D_{t} . 5: For each x\in D_{t} , generate response y\sim\pi_{t-1}(\cdot\mid x) . 6: Compute token-wise rewards r^{\mathrm{RTO}} by ( 3 ). 7: Compute trajectory PPO losses \{L_{i}(\theta)\}_{i=1}^{n} by ( 2 ). 8: Compute \mathrm{Var}(L)=\frac{1}{n}\sum_{i=1}^{n}(L_{i}(\theta)-\bar{L}(\theta))^{2} . 9: Replace the PPO loss with the \chi^{2} -DRTO loss L_{\chi^{2}}(\theta)=\frac{1}{n}\sum_{i=1}^{n}L_{i}(\theta)+\sqrt{\rho}\,\sqrt{\mathrm{Var}(L)+\delta}. 10: Update \pi_{t}\leftarrow\texttt{PPO-Update}(\pi_{t-1},r^{\mathrm{RTO}},L_{\chi^{2}}) . 11: end for 12: Output: \pi_{T} . A tractable relaxation for \chi^{2} ambiguity set depends only on the minibatch mean and variance of \{L_{i}(\theta)\} . By the definition, such minibatch mean \bar{L}(\theta) coincides with the standard PPO actor loss L_{\mathrm{PPO}}(\theta) , and we denote the standard deviation as \sigma(\theta):=\sqrt{\mathrm{Var}_{\tau_{i}\sim\mathbb{P}_{\mathcal{B}}}(L_{i}(\theta))}. We thus have the following result. Theorem 3.2 (Relaxation for \chi^{2} -DRTO) . Fix any minibatch \mathcal{B} of size n with \mathbb{P}_{\mathcal{B}} uniform on \{1,\ldots,n\} and \chi^{2} radius \rho\geq 0 . Assume L_{i}(\theta)\in\mathbb{R} for all i . Then for any \theta , R_{\chi^{2}}(\theta;\rho)\;\leq\;\bar{L}(\theta)+\sigma(\theta)\sqrt{\rho}. (12) Moreover, if \sigma(\theta)>0 and the nonnegativity condition 1+\sqrt{\rho}\,\frac{L_{i}(\theta)-\bar{L}(\theta)}{\sigma(\theta)}\geq 0,\qquad\forall i\in\{1,\ldots,n\}, (13) holds, then the inequality in ( 12 ) is tight. Proof. See Appendix A.2 . ∎ Since L_{\mathrm{PPO}}(\theta)=\bar{L}(\theta) , the relaxation in ( 12 ) yields a tractable upper bound on the \chi^{2} -robust value R_{\chi^{2}}(\theta;\rho)\;\leq\;L_{\mathrm{PPO}}(\theta)+\sigma(\theta)\sqrt{\rho}. Therefore, we replace the standard PPO minibatch loss with a stabilized surrogate to form the \chi^{2}- DRTO objective: L_{\chi^{2}}(\theta):=L_{\mathrm{PPO}}(\theta)+\sqrt{\rho}\,\sqrt{\mathrm{Var}(L_{i}(\theta))+\delta}, (14) where \delta>0 is a small numerical stabilizer. As \delta\to 0 , ( 14 ) coincides with the bound in ( 12 ), and the bound is tight whenever ( 13 ) holds. Consequentially, Algorithm 3 implements the \chi^{2} -DRTO by computing ( 14 ) on each minibatch from the trajectory losses via their sample mean and variance. 4 Experimental Results In this section, we empirically evaluate our DRTO methods to assess their capability of improving robustness under distribution shift and maintaining strong performance on in-distribution data. 4.1 Robustness of DRTO Datasets and setup. We evaluate the robustness gains of KL-DRTO and \chi^{2} -DRTO against several widely used RLHF fine-tuning objectives. All methods use the same backbone Llama3-8B and are fine-tuned on 10,000 samples from OpenMathInstruct ( 19 ) , a widely used training dataset that spans diverse math topics and includes multiple solution formats. Such variety reduces reliance on any single prompt template. As a result, performance on external benchmarks more directly reflects robustness to distribution shift, rather than memorization of dataset-specific patterns. We use the ambiguity radius of 0.1 and 0.01 for \chi^{2} -DRTO and KL-DRTO respectively, where a hyperparameter analysis is included in Section 4.3 , and additional implementation details are included in Appendix B . We report flexible-match accuracy using lm-evaluation-harness ( 5 ) , and all evaluations follow a 5-shots regime. Table 1: Accuracy of Llama3-8B across five benchmarks. Superscripts report relative change (%) compared to standard RTO. Best results are in bold . Models Accuracy GSM8K MathQA GSM (CoT) GSM-Plus GSM (DE) RTO 0.732 0.441 0.744 0.530 0.608 SFT 0.614 -16.12% 0.421 -4.54% 0.648 -12.90% 0.393 -25.85% 0.476 -21.71% PPO 0.732 0.00% 0.438 -0.68% 0.752 +1.08% 0.528 -0.38% 0.620 +1.97% GRPO 0.727 -0.68% 0.441 0.00% 0.760 +2.15% 0.526 -0.75% 0.600 -1.32% DPO 0.739 +0.96% 0.445 +0.91% 0.764 +2.69% 0.546 +3.02% 0.620 +1.97% \chi^{2} -DRTO 0.799 +9.17% 0.449 +1.81% 0.832 +11.83% 0.572 +7.92% 0.656 +7.89% KL-DRTO 0.785 +7.24% 0.452 +2.49% 0.796 +6.99% 0.562 +6.04% 0.660 +8.55% Benchmarks and induced distribution shifts. To evaluate generalization beyond the training distribution, we test on a suite of math-reasoning benchmarks that differ from OpenMathInstruct in either problem style, format, or difficulty. Table 2 summarizes the benchmarks and the primary type of distribution shift they introduce, and we include more benchmarks with similar distribution shifts in Appendix D . Table 2 : Benchmarks and induced distribution shifts. Dataset Domain Distribution Shift GSM8K Grade School Math Standard arithmetic word problems MathQA Middle level Math More varied question templates and multi-step reasoning compared to training. GSM(CoT) Reasoning with assistant prompt Prompt-format shift induced by explicit chain-of-thought triggers. GSM-Plus Adversarial Variants Distractor-based perturbations designed to stress robustness under shifts of formats and numbers. GSM(DE) Multilingual German translation that changes wording and syntax. Baselines. To verify the performance of DRTO, we compare with various baseline models, including (i) RTO ( 24 ) , the token-level RLHF baseline that our methods extend; (ii) supervised fine-tuning (SFT) ( 11 ) on the same dataset; (iii) DPO ( 13 ) , a classic preference-based framework; (iv) PPO ( 18 ) , a classic policy-gradient RLHF baseline; (v) GRPO ( 15 ) , a recent group-based RL objective. Our main methods are reported at the bottom of Table 1 . DRTO improves robustness across all distribution shifts. Table 1 summarizes accuracy on five benchmarks. Both robust models outperform all baselines. \chi^{2} -DRTO improves over RTO by 11.83% on GSM(CoT) and 7.92% on GSM-Plus , and achieves the best score on three out of five benchmarks, while KL-DRTO achieves the best accuracy on the other two. On MathQA , gains are more modest, suggesting that this benchmark is less sensitive to the specific distributional shifts addressed by our robustness objectives under the current training budget. Notably, standard RTO is not uniformly stronger than simpler baselines under shift, indicating that token-wise reward shaping without robustness can be more vulnerable to distribution shift. In contrast, non-robust baselines yield comparatively small or inconsistent improvements. Table 3 : Comparison of DRPPO with DRTO. Best results in each column are in bold . Model GSM8K MathQA GSM(CoT) GSM-Plus GSM(DE) \chi^{2} -DRPPO 0.738 0.441 0.756 0.532 0.624 KL-DRPPO 0.738 0.441 0.764 0.527 0.620 \chi^{2} -DRTO 0.799 0.449 0.832 0.572 0.656 KL-DRTO 0.785 0.452 0.796 0.562 0.660 Table 4 : In-distribution evaluation on GSM8K. Method GSM8K SFT 0.633 PPO 0.781 DPO 0.768 RTO 0.802 KL-DRTO 0.801 \chi^{2} -DRTO 0.798 (a) Training return comparison. (b) Critic loss during training. (c) PTX loss comparison. Figure 2 : Training dynamics comparison across methods. Table 5 : KL-DRTO ablation over ambiguity radius \rho . \rho MathQA GSM(DE) GSM-Plus GSM8K GSM (CoT) 0.5 0.449 0.656 0.558 0.783 0.786 0.1 0.447 0.636 0.564 0.784 0.796 0.01 0.452 0.656 0.562 0.785 0.796 0.001 0.453 0.644 0.555 0.784 0.800 Table 6 : \chi^{2} -DRTO ablation over ambiguity radius \rho . \rho MathQA GSM(DE) GSM-Plus GSM8K GSM (CoT) 0.5 0.448 0.652 0.567 0.795 0.800 0.1 0.449 0.652 0.572 0.799 0.820 0.01 0.450 0.656 0.570 0.786 0.788 0.001 0.448 0.632 0.571 0.785 0.788 4.2 The vital role of token-wise reward: DRTO vs DRPPO Despite the large gains from DRO, a natural question is whether robustness alone is doing all the work. To evaluate whether token-wise shaping in RTO is still essential under distribution shift, we conduct the following ablation study. Since our robust objectives act purely at the trajectory level through the minibatch PPO losses \{L_{i}(\theta)\} , both KL and \chi^{2} variant can be plugged into standard PPO with minimal changes. Concretely, we train PPO with Algorithm 2 and Algorithm 3 under the same experimental setup as Section 4.1 , yielding KL-DRPPO and \chi^{2} -DRPPO. Table 4 compares these two robust PPO models against our KL-DRTO and \chi^{2} -DRTO models. Across all benchmarks, DRTO is consistently stronger than its PPO counterpart, which suggests the gains do not come from DRO alone. Instead, the results indicate that the token-wise reward in RTO provides a dense and fine-grained learning signal that is helpful for improving performance, while the DRO term stabilizes trajectory-level updates by downweighting or penalizing high-loss outliers within each minibatch. When we keep the same robustness mechanism but remove the token-wise shaping for DRPPO, the improvement is noticeably smaller and sometimes unstable under shift; when we combine both for DRTO, robustness and accuracy improve together. In short, DRO improves the reliability of the update, but the intrinsic token-wise reward in RTO still supplies an important part of the learning signal, and DRTO benefits from both. 4.3 Performance Analysis Besides the standard benchmark comparisons, we carry out the following ablation studies. In-distribution case. To verify that the robustness-oriented objectives do not distort the model behavior on the training distribution, we additionally evaluate an in-distribution setting, where we train on the GSM8K training split and evaluate on the corresponding test split. Table 4 shows that DRTO remains competitive in-distribution while improving robustness and training behavior in our other analyses. Both KL-DRTO and \chi^{2} -DRTO achieve strong scores, closely tracking the best-performing RTO and improving over standard PPO and DPO . Overall, this in-distribution sanity check supports our central claim that DRTO provides robustness and improved optimization characteristics without sacrificing stable behavior on the training distribution. Convergence and optimization effectiveness. Figure 2(a) compares the training return across methods. Among all approaches, KL-DRTO demonstrates the strongest optimization behavior, achieving the highest return and exhibiting a clear upward trend throughout training. In contrast, PPO and GRPO remain near zero, while RTO stays negative for most of training. \chi^{2} -DRTO consistently outperforms all baselines and avoids the degradation observed in RTO, which suggest that DRTO improves not only the final performance but also the effectiveness of the optimization trajectory. Critic stability. To assess training stability beyond return, we examine the critic loss in Figure 2(b) . Both DRTO variants converge to a noticeably lower critic loss than PPO, GRPO, and RTO. In particular, \chi^{2} -DRTO achieves the lowest and most stable critic loss, while KL-DRTO closely follows. This indicates that DRTO produces better-conditioned value learning signals, which is critical for stable policy optimization and partially explains its superior training dynamics. Preserving the pretraining objective. Figure 2(c) reports the auxiliary PTX loss, which reflects deviation from the pretraining distribution. Compared to PPO and RTO, both DRTO variants maintain consistently lower PTX loss. Notably, KL-DRTO achieves strong return improvements while keeping PTX loss lower than all non-DRTO baselines, suggesting that DRTO improves alignment performance without excessively drifting away from the base language model. Hyperparameter sensitivity. To assess sensitivity to the ambiguity radius, we sweep \rho for KL-DRTO and \chi^{2} -DRTO in Tables 6 and 6 . Overall, performance exhibits a moderate fluctuation across radii. For \chi^{2} -DRTO, the smallest radius provides limited robustness and leads to noticeably weaker results on several shifted benchmarks, while overly large radii can become more conservative and reduce performance on some tasks. Across the five benchmarks, \rho=0.1 yields the strongest and most balanced accuracy for \chi^{2} -DRTO, achieving the best scores on GSM8K and GSM (CoT) and the best performance on GSM-Plus . For KL-DRTO, performance is similarly stable, but different benchmarks peak at different radii. In practice, this variety indicates that a small grid over \rho is sufficient to locate a robust operating point, and that DRTO does not require extremely fine-tuned radii to obtain gains. 5 Conclusion In this work, we introduced Distributionally Robust Token Optimization (DRTO), a unified framework that connects RLHF token-level policy optimization with principled distributionally robust learning. By constructing divergence-based ambiguity sets around the empirical prompt distribution and deriving tight dual surrogates, DRTO optimizes a robust control objective that explicitly accounts for worst-case (adversarial) distribution shifts within the chosen divergence ball—a property not captured by standard PPO- or RTO-style objectives that primarily rely on reward shaping or fixed regularization. Our analysis further shows that \chi^{2} -DRTO and KL-DRTO are two concrete instantiations of the same robust optimization principle, offering a clean theoretical interpretation of adaptive-penalty mechanisms commonly used in practice. Empirically, DRTO improves robustness and training behavior. Across multiple math-reasoning benchmarks and perturbation settings, DRTO consistently strengthens out-of-distribution reasoning and factuality under distribution shifts, while remaining competitive on in-distribution evaluation. Beyond final scores, the training dynamics reveal that DRTO yields more effective and stable optimization trajectories. Moreover, these gains are achieved as a drop-in replacement in standard RLHF pipelines, without increasing model size or requiring architectural changes. Overall, our results suggest that distributional robustness is a fundamental and under-explored axis of alignment, since real-world deployment inevitably involves prompt and data shifts, and robustness cannot be reduced to the control from the reference model alone. DRTO clarifies the theoretical foundation of robust token-level RLHF and provides a practical method for improving reliability under distribution shift. Promising directions include extensions from single-turn prompts to multi-step conversational trajectories or integrating DRTO with preference-based, multi-reward, and/or verifier-guided RLHF. Impact Statement This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here. Appendix A Appendix of Proofs A.1 Proof of Theorem 3.1 We prove the exact one-dimensional dual representation of the minibatch KL-robust value and the induced softmax form of the worst-case reweighting when the dual infimum is attained at a finite temperature. If \rho=0 , then the constraint D_{\mathrm{KL}}(q\|\tfrac{1}{n}\mathbf{1})\leq 0 forces q=\tfrac{1}{n}\mathbf{1} , so R_{\mathrm{KL}}(\theta;0)=\frac{1}{n}\sum_{i=1}^{n}L_{i}(\theta) . Moreover, \Psi_{\eta}(\theta)\to\frac{1}{n}\sum_{i=1}^{n}L_{i}(\theta) as \eta\downarrow 0 , hence \inf_{\eta>0}\{\Psi_{\eta}(\theta)+0/\eta\}=\frac{1}{n}\sum_{i=1}^{n}L_{i}(\theta) , which matches ( 8 ) . In the rest of the proof, we assume \rho>0 . To begin with, we write the inner maximization ( 4 ) as the finite-dimensional convex program \max_{q\in\Delta_{n}}\ \sum_{i=1}^{n}q_{i}L_{i}(\theta)\quad\text{where }\quad D_{\mathrm{KL}}\!\left(q\Big\|\tfrac{1}{n}\mathbf{1}\right)\leq\rho,\qquad D_{\mathrm{KL}}\!\left(q\Big\|\tfrac{1}{n}\mathbf{1}\right)=\sum_{i=1}^{n}q_{i}\log(nq_{i}), and \Delta_{n}=\{q\in\mathbb{R}^{n}_{+}:\sum_{i}q_{i}=1\} . The point q=\tfrac{1}{n}\mathbf{1} is strictly feasible whenever \rho>0 , so Slater’s condition ( 2 ) holds and strong duality applies. Let \nu\geq 0 be the Lagrange multiplier for the KL constraint. The Lagrangian is \mathcal{L}(q,\nu)=\sum_{i=1}^{n}q_{i}L_{i}(\theta)+\nu\!\left(\rho-D_{\mathrm{KL}}\!\left(q\Big\|\tfrac{1}{n}\mathbf{1}\right)\right). By strong duality, \displaystyle R_{\mathrm{KL}}(\theta;\rho) \displaystyle=\inf_{\nu\geq 0}\ \sup_{q\in\Delta_{n}}\ \mathcal{L}(q,\nu) \displaystyle=\inf_{\nu\geq 0}\left\{\nu\rho+\sup_{q\in\Delta_{n}}\left(\sum_{i=1}^{n}q_{i}L_{i}(\theta)-\nu\,D_{\mathrm{KL}}\!\left(q\Big\|\tfrac{1}{n}\mathbf{1}\right)\right)\right\}. When \nu=0 , the inner supremum reduces to \max_{i}L_{i}(\theta) . For \nu>0 , we can apply the Gibbs variational principle ( 20 ) as follows: Lemma A.1 (van Handel, Lemma 4.10) . Let \mathbb{P} be a probability distribution supported on a finite set \{1,\ldots,n\} and let u=(u_{1},\ldots,u_{n})\in\mathbb{R}^{n} . For any \eta>0 , \sup_{\mathbb{Q}\ll\mathbb{P}}\left\{\mathbb{E}_{\tau_{i}\sim\mathbb{Q}}[u_{i}]-\frac{1}{\eta}D_{\mathrm{KL}}(\mathbb{Q}\|\mathbb{P})\right\}=\frac{1}{\eta}\log\mathbb{E}_{\tau_{i}\sim\mathbb{P}}\!\left[e^{\eta u_{i}}\right]. (15) Moreover, the supremum is attained, and any maximizer is given by the exponential tilt \mathbb{Q}_{\eta}(i)=\frac{\mathbb{P}(i)e^{\eta u_{i}}}{\sum_{j=1}^{n}\mathbb{P}(j)e^{\eta u_{j}}},\qquad i\in\{1,\ldots,n\}. (16) Now take any \nu>0 and set \eta:=1/\nu>0 . Note that \sum_{i=1}^{n}q_{i}L_{i}(\theta)-\nu\,D_{\mathrm{KL}}\!\left(q\Big\|\tfrac{1}{n}\mathbf{1}\right)=\sum_{i=1}^{n}q_{i}L_{i}(\theta)-\frac{1}{\eta}D_{\mathrm{KL}}(q\|\mathbb{P}_{\mathcal{B}}),\qquad\mathbb{P}_{\mathcal{B}}\equiv\tfrac{1}{n}. Applying Lemma A.1 with \mathbb{P}=\mathbb{P}_{\mathcal{B}} and u_{i}=L_{i}(\theta) yields \sup_{q\in\Delta_{n}}\left(\sum_{i=1}^{n}q_{i}L_{i}(\theta)-\frac{1}{\eta}D_{\mathrm{KL}}(q\|\mathbb{P}_{\mathcal{B}})\right)=\Psi_{\eta}(\theta), and the maximizer is q=\mathbb{Q}_{\eta} with the exponential-tilt form ( 7 ) . Substituting \nu=1/\eta gives the dual form ( 8 ); the \nu=0 case corresponds to the \eta\to\infty limit. Finally, if the infimum in ( 8 ) is attained at some finite \eta^{\star}\in(0,\infty) , then \nu^{\star}:=1/\eta^{\star}>0 is dual optimal. By strong duality and the KKT conditions ( 2 ) , there exists a primal optimal q^{\star} that maximizes \mathcal{L}(q,\nu^{\star}) over q\in\Delta_{n} and is feasible for the KL constraint. The inner maximizer at \nu^{\star} is unique, hence q^{\star}=\mathbb{Q}_{\eta^{\star}} . Therefore \mathbb{Q}^{\star}=\mathbb{Q}_{\eta^{\star}} attains the primal supremum, which gives ( 9 ) and completes the proof. A.2 Proof of Theorem 3.2 We first show the following bound, which also characterizes an optimizer when the nonnegativity condition holds. Theorem A.2 ( \chi^{2} -robust bound) . Fix a minibatch \mathcal{B} and assume L_{i}(\theta)\in\mathbb{R} for all i . Let \bar{L}(\theta)=\mathbb{E}_{\tau_{i}\sim\mathbb{P}_{\mathcal{B}}}[L_{i}(\theta)] and \sigma(\theta)=\sqrt{\mathrm{Var}_{\tau_{i}\sim\mathbb{P}_{\mathcal{B}}}(L_{i}(\theta))} . Then for any \rho>0 , R_{\chi^{2}}(\theta;\rho)\leq\bar{L}(\theta)+\sigma(\theta)\sqrt{\rho}. (17) Moreover, given z_{i}(\theta)=L_{i}(\theta)-\bar{L}(\theta) , if 1+\sqrt{\rho}\,\frac{z_{i}(\theta)}{\sigma(\theta)}\geq 0\quad\text{for all }i, (18) then the bound in ( 17 ) is tight and the supremum in ( 11 ) equals \bar{L}(\theta)+\sigma(\theta)\sqrt{\rho} . In this case, an optimal adversary is \mathbb{Q}^{\star}(i)=\mathbb{P}_{\mathcal{B}}(\tau_{i})\left(1+\sqrt{\rho}\,\frac{z_{i}(\theta)}{\sigma(\theta)}\right). (19) Proof. Let w(i)=\mathbb{Q}(i)/\mathbb{P}_{\mathcal{B}}(\tau_{i}) . Then \mathbb{E}_{\mathbb{P}_{\mathcal{B}}}[w]=1 and \mathbb{E}_{\mathbb{P}_{\mathcal{B}}}[(w-1)^{2}]\leq\rho . We have \mathbb{E}_{\mathbb{Q}}[L_{i}(\theta)]=\mathbb{E}_{\mathbb{P}_{\mathcal{B}}}[wL_{i}(\theta)]=\bar{L}(\theta)+\mathbb{E}_{\mathbb{P}_{\mathcal{B}}}\big[(w-1)(L_{i}(\theta)-\bar{L}(\theta))\big]. By Cauchy–Schwarz, \mathbb{E}_{\mathbb{P}_{\mathcal{B}}}\big[(w-1)(L_{i}(\theta)-\bar{L}(\theta))\big]\leq\sqrt{\mathbb{E}_{\mathbb{P}_{\mathcal{B}}}[(w-1)^{2}]}\ \sqrt{\mathbb{E}_{\mathbb{P}_{\mathcal{B}}}[(L_{i}(\theta)-\bar{L}(\theta))^{2}]}\leq\sqrt{\rho}\,\sigma(\theta), which implies ( 17 ). If \sigma(\theta)=0 , then L_{i}(\theta)=\bar{L}(\theta) for all i , so R_{\chi^{2}}(\theta;\rho)=\bar{L}(\theta) and the theorem holds. Now assume \sigma(\theta)>0 and ( 18 ) holds. Define w^{\star}(i)=1+\sqrt{\rho}\,z_{i}(\theta)/\sigma(\theta) and \mathbb{Q}^{\star}(i)=\mathbb{P}_{\mathcal{B}}(\tau_{i})w^{\star}(i) . Then \mathbb{Q}^{\star} is a valid distribution and \mathbb{E}_{\mathbb{P}_{\mathcal{B}}}[w^{\star}]=1 . Moreover, \mathbb{E}_{\mathbb{P}_{\mathcal{B}}}[(w^{\star}-1)^{2}]=\mathbb{E}_{\mathbb{P}_{\mathcal{B}}}\left[\rho\frac{z_{i}(\theta)^{2}}{\sigma(\theta)^{2}}\right]=\rho, so \mathbb{Q}^{\star}\in\Omega^{\chi^{2}}_{\rho}(\mathbb{P}_{\mathcal{B}}) . Finally, \mathbb{E}_{\mathbb{Q}^{\star}}[L_{i}(\theta)]=\bar{L}(\theta)+\mathbb{E}_{\mathbb{P}_{\mathcal{B}}}[(w^{\star}-1)z_{i}(\theta)]=\bar{L}(\theta)+\sqrt{\rho}\,\frac{\mathbb{E}_{\mathbb{P}_{\mathcal{B}}}[z_{i}(\theta)^{2}]}{\sigma(\theta)}=\bar{L}(\theta)+\sigma(\theta)\sqrt{\rho}, which matches the upper bound and proves tightness. ∎ Theorem 3.2 thus follows by applying Theorem A.2 to R_{\chi^{2}}(\theta;\rho) . Appendix B Implementation Details This appendix provides implementation details for all baselines and robust variants in our DRTO study, including the training pipeline, shared hyperparameters, and method-specific configurations. Training pipeline. All experiments use OpenRLHF/Llama-3-8b-sft-mixture as the SFT initialization for the actor. For methods with PPO updates, we use a shared reward model checkpoint as the reward function and as the critic initialization. For RTO-style methods, we additionally load a DPO policy checkpoint to provide token-level preference shaping. Datasets. Preference learning (DPO) uses the UltraFeedback preference dataset. Reinforcement learning (PPO/RTO/DRTO) uses a prompt-only OpenMath dataset. Unless stated otherwise, we use max_samples=10000 prompts and seed=42 . Shared RLHF setup. Across PPO/RTO/DRTO runs, We apply a cosine learning rate schedule with 3% warming steps and 10% minimum learning rate. We use DeepSpeed ZeRO-2 with adam_offload , bf16 , flash_attn , and gradient_checkpointing . PPO implementation details. Our PPO-based runs follow the OpenRLHF implementation and enable reward normalization. We use GAE with lambd=0.95 and gamma=1.0 . We list the hyperparameters for each method below. DPO (UltraFeedback) Learning rate 5e-7 Batch size 256 Maximum sequence length 2048 DPO KL coefficient ( \beta ) 0.1 RTO (OpenMath) Max samples / seed 10000 / 42 Prompt / response length 512 / 1024 Rollout batch size 512 Train batch size 256 Micro rollout / train batch size 8 / 8 Actor / critic learning rate 1e-5 / 5e-5 Initial KL coefficient 0.01 DPO reward scale / clip 0.05 / 0.05 PTX coefficient 0.05 PPO (OpenMath) Max samples / seed 10000 / 42 Prompt / response length 512 / 1024 Rollout batch size 512 Train batch size 256 Micro rollout / train batch size 8 / 8 Actor / critic learning rate 1e-5 / 5e-5 Initial KL coefficient 0.01 KL target 0.1 PPO clip ( \epsilon ) / value clip 0.2 / 0.2 GAE \lambda / discount \gamma 0.95 / 1.0 PTX coefficient 0.05 Max grad norm 1.0 KL-DRTO (OpenMath) Max samples / seed 10000 / 42 Prompt / response length 512 / 1024 Rollout batch size 320 Train batch size 160 Micro rollout / train batch size 8 / 8 Actor / critic learning rate 1e-5 / 5e-5 Initial KL coefficient 0.005 PTX coefficient 0.05 DPO reward scale / clip 0.05 / 0.05 Max epochs 1 KL radius \rho 0.01 Numerical eps 1e-8 \eta bounds min=1e-12, max=1e6 \eta iterations 30 \chi^{2} -DRTO (OpenMath) Max samples / seed 10000 / 42 Prompt / response length 512 / 1024 Rollout batch size 128 Train batch size 64 Micro rollout / train batch size 8 / 8 Actor / critic learning rate 1e-5 / 5e-5 Initial KL coefficient 0.005 PTX coefficient 0.05 DPO reward scale / clip 0.05 / 0.05 Max epochs 1 \chi^{2} radius \rho 0.1 Numerical \delta 1e-8 System and memory optimizations. All runs use DeepSpeed ZeRO-2 with adam_offload . We additionally enable bf16 , flash_attn , and gradient_checkpointing . Appendix C Response comparison To further illustrate qualitative differences among methods, we present a representative evaluation example and compare the responses produced by RTO and both DRTO variants on the same prompt in Table 7 . Table 7 : Example responses on the same prompt. RTO misses the pack price multiplication, while both DRTO variants return the correct total cost. Question. Terry eats 2 yogurts a day. They are currently on sale at 4 yogurts for $5.00. How much does he spend on yogurt over 30 days? RTO. Terry eats 2 yogurts a day, so over 30 days he eats 2*30=<<2*30=60>>60 yogurts. They are currently on sale at 4 yogurts for $5.00, so Terry spends 60/4=$<<60/4=15>>15.00 on yogurt over 30 days. #### 15 \chi^{2} -DRTO ( \rho=0.1 ). Terry eats 2 yogurts a day, so over 30 days he eats 2*30=<<2*30=60>>60 yogurts. They are currently on sale at 4 yogurts for $5.00, so Terry spends $5.00 for every 4 yogurts. To find out how much Terry spends on 60 yogurts, we can divide 60 by 4 to find out how many sets of 4 yogurts he needs: 60/4=<<60/4=15>>15 sets of 4 yogurts. Since each set of 4 yogurts costs $5.00, Terry spends 15*$5.00=<<15*5=75>>$75.00 on yogurt over 30 days. #### 75 KL-DRTO ( \rho=0.01 ). Terry eats 2 yogurts a day, so over 30 days he eats 2*30=<<2*30=60>>60 yogurts. The sale is 4 yogurts for $5.00, so the price per yogurt is $5.00/4 = $<<5/4=1.25>>1.25. Therefore, Terry spends 60 * $1.25 = $<<60*1.25=75>>75 on yogurt over 30 days. #### 75 Appendix D Additional Experiments We compare the models’ performances on GSM’s Chinese, German, Spanish and French translations, respectively. The result is shown in Table 8 . Table 8: Accuracy on GSM across four languages. Superscripts report relative change (%) compared to standard RTO within each language. Best results in each column are in bold . Models Accuracy GSM(ZH) GSM(DE) GSM(ES) GSM(FR) RTO 0.556 0.608 0.676 0.608 SFT 0.540 -2.88% 0.616 +1.32% 0.648 -4.14% 0.584 -3.95% PPO 0.548 -1.44% 0.620 +1.97% 0.668 -1.18% 0.596 -1.97% GRPO 0.540 -2.88% 0.600 -1.32% 0.656 -2.96% 0.580 -4.61% DPO 0.556 0.00% 0.620 +1.97% 0.696 +2.96% 0.608 0.00% \chi^{2} -DRTO 0.580 +4.32% 0.656 +7.89% 0.716 +5.92% 0.604 -0.66% KL-DRTO 0.572 +2.88% 0.660 +8.55% 0.720 +6.51% 0.640 +5.26% Appendix E Tailed Behavior of DRTO vs RTO Aggregate accuracy can obscure catastrophic errors on difficult examples. To directly measure worst-case robustness, we examine the hardest 20% of the GSM8K test set—specifically, the 264 examples where the baseline RTO model achieves 0% accuracy . These cases reveal systematic weaknesses of standard RLHF, where optimizing average reward often neglects minority or adversarial examples. Table 9 reports performance on both the full GSM8K test set and this hard subset. Table 9: Accuracy on GSM8K and the hardest 20%. \chi^{2} -DRTO provides substantial gains on tail cases. Method Full Acc. Hard Acc. Abs. Gain Correct / Total RTO 73.2% 0.00% — 0/264 KL-DRTO 78.5% 6.44% +6.44% 17/264 \chi^{2} -DRTO 79.9% 34.47% +34.47% 91/264 The baseline RTO model exhibits a critical weakness: despite achieving strong average accuracy, it fails on all hard examples. This outcome powerfully illustrates a core limitation of mean-reward RLHF: a model can appear reliable in aggregate while remaining brittle on rare but important, high-complexity cases. In sharp contrast, \chi^{2} -DRTO dramatically improves performance, solving 91 of these previously intractable examples and boosting hard-example accuracy from 0% to 34.47\% . Crucially, this significant gain in robustness is achieved without sacrificing overall performance; the \chi^{2} -DRTO model maintains an accuracy level comparable to KL-DRTO and superior to the original RTO. The practical distinction lies in how the divergence objectives operate: • KL-DRTO employs a lighter-tailed correction, which yields only modest gains in robustness. • \chi^{2} -DRTO applies a heavier-tailed correction, assigning substantially greater weight to hard or misaligned examples. • This specific weighting mechanism is key to \chi^{2} -DRTO’s consistent advantage across the reasoning benchmarks with distributional shifts(Table 2 ).