Safe RLHF Beyond Expectation: Stochastic Dominance for Universal Spectral Risk Control

Yaswanth Chittepu, Ativ Joshi, Rajarshi Bhattacharjee, Scott Niekum · 2026

Read Safe RLHF Beyond Expectation: Stochastic Dominance for Universal Spectral Risk Control by Yaswanth Chittepu, Ativ Joshi, Rajarshi Bhattacharjee, Scott Niekum (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-10938.txt.

Introduction

Yaswanth Chittepu and coauthors argue that keeping language models safe through expected cost limits is too weak. An average hides rare but catastrophic outcomes. Their Risk-sensitive Alignment via Dominance takes the place of the average. It uses first-order stochastic dominance constraints. These compare whole cost distributions against a reference policy. Quantile-weighted versions are shown to control a broad family of spectral risk measures. Tests show less harm than baselines. Helpfulness stays on par. Out-of-distribution robustness is stronger too.

Catalog record

CREATORYaswanth Chittepu, Ativ Joshi, Rajarshi Bhattacharjee, Scott Niekum
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61928
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.10938

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Safe RLHF Beyond Expectation: Stochastic Dominance for Universal Spectral Risk Control Yaswanth Chittepu, Ativ Joshi, Rajarshi Bhattacharjee, Scott Niekum Abstract Safe Reinforcement Learning from Human Feedback (RLHF) typically enforces safety through expected cost constraints, but the expectation captures only a single statistic of the cost distribution and fails to account for distributional uncertainty, particularly under heavy tails or rare catastrophic events. This limitation is problematic when robustness and risk sensitivity are critical. Stochastic dominance offers a principled alternative by comparing entire cost distributions rather than just their averages, enabling direct control over tail risks and potential out-of-distribution failures that expectation-based constraints may overlook. In this work, we propose Risk-sensitive Alignment via Dominance (RAD), a novel alignment framework that replaces scalar expected cost constraints with First-Order Stochastic Dominance (FSD) constraints. We operationalize this constraint by comparing the target policy’s cost distribution to that of a reference policy within an Optimal Transport (OT) framework, using entropic regularization and Sinkhorn iterations to obtain a differentiable and computationally efficient objective for stable end-to-end optimization. Furthermore, we introduce quantile-weighted FSD constraints and show that weighted FSD universally controls a broad class of Spectral Risk Measures (SRMs), so that improvements under weighted dominance imply guaranteed improvements in the corresponding spectral risk. This provides a principled mechanism for tuning a model’s risk profile via the quantile weighting function. Empirical results demonstrate that RAD improves harmlessness over baselines while remaining competitive in helpfulness, and exhibits greater robustness on out-of-distribution harmlessness evaluations. 1 Introduction Large Language Models (LLMs) are now used across a wide range of applications, so it is important that their outputs are helpful, reliable, and safe. This need is especially acute in high-stakes domains such as legal reasoning ( 31 ), medical consultation ( 60 ; 37 ), and educational support ( 30 ; 33 ), where harmful generations—including toxicity and misinformation—can have serious consequences ( 26 ; 57 ; 24 ). Broadly, we want LLMs to be helpful and aligned with human preferences, while being harmless , i.e. prevent toxic outputs. But there may be scenarios when users may request assistance with potentially harmful activities ( 7 ; 27 ). Thus, harmlessness and helpfulness can conflict. The standard technique for aligning LLMs with human preferences, Reinforcement Learning from Human Feedback (RLHF), typically optimizes a single reward model for both objectives ( 40 ; 8 ), or heuristically mixes separate reward models ( 27 ; 55 ; 38 ). This creates trade-offs: stronger harmlessness can lead to refusals, while stronger helpfulness can increase unsafe outputs ( 8 ). Recent work addresses this by decoupling preference data and enforcing harmlessness as a constraint, an approach known as Safe RLHF ( 16 ). However, Safe RLHF typically typically constrains the expected cost of a policy, providing no guarantees on worst-case or tail outcomes. This is inadequate in high-stakes deployments where tail risk is critical — such as code generation 43 and open-ended dialogue where rare but severe harms including toxic generations and private data leakage can impact users ( 44 ; 26 ) . Thus, we argue that a stronger notion of constraining the cost of the policy is needed: the cost distribution of the learned policy should be stochastically smaller than that of the reference policy, not merely cheaper on average. This means that the learned policy should assign less probability to high-cost outcomes across the distribution, not just reduce the mean cost. This motivates Risk-sensitive Alignment via Dominance (RAD) , which enforces a First-Order Stochastic Dominance (FSD) constraint on the cost distribution of the learned policy relative to a reference policy. A key insight of RAD is that by reweighting quantiles of the FSD objective, we recover Spectral Risk Measures (SRMs) — a family of risk measures defined as a weighted integral of the cost quantiles, \int w(q)Q_{X}(q)dq , where Q_{X}(q) is the quantile function of the cost distribution and the weighting function w(q) allows practitioners to express diverse risk preferences over the cost distribution. For instance, concentrating weight on upper quantiles recovers tail-sensitive measures such as CVaR, while uniform weighting recovers the mean — both expressible within our unified framework. This is practically valuable: a medical deployment may demand near-zero tolerance for harmful outputs, while a general assistant may tolerate a more permissive tradeoff, and both are accommodated by choosing w appropriately. Optimizing FSD constraints directly is challenging, so we relax them to an asymmetric FSD-violation surrogate that aggregates positive quantile gaps (see Equation 3 ). We interpret stochastic dominance through Optimal Transport (OT) and derive an efficient REINFORCE-style policy gradient using Sinkhorn iterations, yielding an end-to-end differentiable approach. Empirically, RAD yields models that are more robust to safety violations while achieving competitive helpfulness across multiple spectral risk measures relative to Safe RLHF. Our main contributions are: (i) we formulate safe alignment as a dominance-constrained objective over the full cost distribution rather than only its expectation, (ii) we derive a RAD policy-gradient estimator using entropic OT and Sinkhorn iterations, making optimization of the objective end-to-end differentiable, (iii) we connect quantile reweighting in our framework to SRMs, enabling controllable risk-sensitive alignment profiles, and (iv) through extensive experiments, we show that RAD improves robustness to safety violations while maintaining competitive helpfulness relative to expected cost based baselines. The rest of the paper is organized as follows: Section 2 reviews Safe RLHF, stochastic dominance, and SRMs; Section 3 presents RAD and its optimization procedure; Section 4 reports empirical evaluations safety–helpfulness trade-offs; and Sections 5 and 6 is related work and conclusion respectively. 2 Background and Problem Setting 2.1 Reinforcement Learning from Human Feedback (RLHF) The most popular method of aligning LLMs with human preferences is Reinforcement Learning from Human Feedback (RLHF). The standard RLHF pipeline consists of three phases as described in 13 ; 40 . The first phase is a supervised fine-tuning (SFT) phase where the outputs of a pre-trained model are aligned with human responses. This is followed by a reward modeling phase, where a reward model is trained using human preferences and finally, a reinforcement learning phase where the model is optimized using the learned reward function. Formally, let x\sim D_{x} denote prompts and y\sim\pi_{\theta}(\cdot\mid x) denote model responses. SFT produces a reference policy \pi_{\mathrm{sft}} , and we set \pi_{\mathrm{ref}}:=\pi_{\mathrm{sft}} unless stated otherwise. A reward model r_{\phi}(x,y) is learned from preference comparisons using the Bradley-Terry model ( 10 ) . Details are standard and deferred to Appendix B . RLHF then optimizes a KL-regularized objective to prevent reward overoptimization and preserve language quality ( 40 ; 51 ; 25 ; 47 ) : \displaystyle\max_{\theta}\;\mathbb{E}_{x\sim D_{x},\,y\sim\pi_{\theta}(\cdot\mid x)}\!\big[r_{\phi}(x,y)\big]\;-\;\beta\,D_{\mathrm{KL}}\!\Big(\pi_{\theta}(\cdot\mid x)\,\|\,\pi_{\mathrm{ref}}(\cdot\mid x)\Big), (1) equivalently maximizing the KL-regularized reward \tilde{r}(x,y)=r_{\phi}(x,y)-\beta\log\frac{\pi_{\theta}(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)} . In practice, this objective is optimized with PPO/GRPO/REINFORCE-style methods ( 58 ; 3 ; 50 ; 49 ) . 2.2 Safe RLHF Standard RLHF optimizes a single reward function learned from human preferences, which may be inadequate when attempting to balance competing goals such as helpfulness and harmlessness . In contrast, Safe RLHF ( 16 ) separates helpfulness and harmlessness by learning (i) a reward model r_{\phi}(x,y) , (parameterized by \phi and for input prompt x and model output y ) from helpfulness preferences and (ii) a cost model c_{\psi}(x,y) , (parameterized by \psi ) from harmfulness

Read the complete text