BSO: Safety Alignment Is Density Ratio Matching
Tien-Phat Nguyen, Truong Nguyen, Thin Nguyen, Duy Minh Ho Nguyen, Ngoc-Thanh Dinh, Trung Le · 2026
Read BSO: Safety Alignment Is Density Ratio Matching by Tien-Phat Nguyen, Truong Nguyen, Thin Nguyen, Duy Minh Ho Nguyen, Ngoc-Thanh Dinh, Trung Le (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12339.txt.
Introduction
Nguyen and five colleagues simplify safety tuning for language models by recasting it as matching density ratios. Current pipelines juggle separate reward and cost models plus online reinforcement learning, while simpler preference methods bolt safety on with ad-hoc margins. They show the optimal safe policy's likelihood ratio splits in closed form. Minimising Bregman divergences between data and model ratios yields BSO, a single-stage family of losses with one extra hyperparameter. Safety benchmarks show a better safety-helpfulness balance.
Catalog record
| CREATOR | Tien-Phat Nguyen, Truong Nguyen, Thin Nguyen, Duy Minh Ho Nguyen, Ngoc-Thanh Dinh, Trung Le |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56847 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.12339
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BSO: Safety Alignment Is Density Ratio Matching Tien-Phat Nguyen, Truong Nguyen, Thin Nguyen, Duy Minh Ho Nguyen, Ngoc-Thanh Dinh, Trung Le Abstract Aligning language models for both helpfulness and safety typically requires complex pipelines—separate reward and cost models, online reinforcement learning, and primal-dual updates. Recent direct preference optimization approaches simplify training but incorporate safety through ad-hoc modifications such as multi-stage procedures or heuristic margin terms, lacking a principled derivation. We show that the likelihood ratio of the optimal safe policy admits a closed-form decomposition that reduces safety alignment to a density ratio matching problem. Minimizing Bregman divergences between the data and model ratios yields Bregman Safety Optimization (BSO) , a family of single-stage loss functions, each induced by a convex generator, that provably recover the optimal safe policy. BSO is both general and simple: it requires no auxiliary models, introduces only one hyperparameter beyond standard preference optimization, and recovers existing safety-aware methods as special cases. Experiments across safety alignment benchmarks show that BSO consistently improves the safety–helpfulness trade-off. 1 Introduction The two central objectives of language model alignment—helpfulness and safety—are often in direct tension. A model that follows instructions brilliantly but occasionally produces harmful content fails in deployment; one that refuses every sensitive query is safe but useless. These objectives are not merely complementary: the most helpful response to a dangerous prompt is often the most unsafe, and the preference data that teaches helpfulness can actively reward unsafe behavior when the preferred response happens to be harmful ( 4 ) . Alignment is therefore not one problem but two entangled ones, and any method that treats safety as an afterthought risks optimizing one objective at the expense of the other. The dominant approach to resolving this tension is constrained optimization: maximize expected helpfulness subject to a safety constraint. Safe RLHF ( 4 ) instantiates this idea by training separate reward and cost models, converting the safety constraint into a Lagrangian with adaptive multipliers, and solving the resulting objective via proximal policy optimization (PPO) ( 17 ) . While principled in formulation, this approach demands a multi-stage pipeline—reward model training, cost model training, online RL, and primal-dual updates—each stage introducing its own instabilities and computational overhead. The gap between the elegance of the constrained formulation and the complexity of its implementation has motivated a search for simpler alternatives. Direct Preference Optimization (DPO) ( 16 ) offered a dramatic simplification for helpfulness alignment by eliminating reward modeling and online RL entirely, reducing the problem to supervised learning on preference pairs. Naturally, several methods have sought to extend this simplicity to the safety setting. SACPO ( 19 ) decomposes alignment into sequential helpfulness and safety stages. CDPO ( 11 ) and CAN ( 8 ) introduce Lagrangian penalties but still require auxiliary reward and cost models to construct training data. SafeDPO ( 9 ) goes furthest toward simplicity—transforming preference pairs using binary safety labels and augmenting DPO with a safety margin—yet its margin term is introduced as a heuristic rather than derived from the optimization objective. A pattern emerges across these methods: each incorporates safety through a different ad-hoc modification—a multi-stage pipeline here, an auxiliary model there, an unexplained margin term elsewhere—without a unifying principle that reveals when and why these modifications work. The field lacks a theoretical framework from which safety-aware preference optimization methods can be systematically derived and compared, one that explains existing approaches as special cases and suggests principled alternatives. In this paper, we provide such a framework. Our starting point is a structural observation: the likelihood ratio of the optimal safe policy decomposes into three interpretable factors—the reference model ratio, the helpfulness preference ratio, and a safety correction term. This decomposition reveals that safety alignment can be cast as a density ratio matching problem, where driving the model’s likelihood ratio toward the data ratio is sufficient to recover the optimal safe policy. By minimizing Bregman divergences ( 3 ) between these ratios, we derive a family of tractable loss functions—each induced by a different convex generator h —that provably converge to the optimal safe policy under sufficient model capacity. We call the resulting framework Bregman Safety Optimization (BSO) . Our contributions are: • We propose BSO, a unified framework for safety-aware preference optimization based on density ratio matching under Bregman divergences. Each choice of convex generator yields a distinct, single-stage training algorithm—requiring no auxiliary models and only one hyperparameter beyond DPO—while all members of the family share the same optimality guarantee. We show that existing methods, including SafeDPO, arise as special cases. • We provide extensive experiments and analysis demonstrating the effectiveness of SBPO across safety alignment benchmarks, including ablation studies on the safety penalty strength and the generator choice. 2 Background 2.1 Preference Alignment Reinforcement Learning from Human Feedback (RLHF) ( 15 ) established the dominant paradigm for aligning language models with human preferences. It first trains a reward model from pairwise comparisons under the Bradley–Terry model ( 2 ) , p(y_{w}\succ y_{l}\mid x)=\sigma\bigl(r_{\phi}(x,y_{w})-r_{\phi}(x,y_{l})\bigr), (1) and then optimizes a KL-regularized policy, typically via proximal policy optimization (PPO) ( 17 ) : \max_{\pi}\;\mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi(\cdot\mid x)}\bigl[r_{\phi}(x,y)\bigr]-\beta\,\mathrm{KL}\bigl(\pi\,\|\,\pi_{\mathrm{ref}}\bigr). (2) Direct Preference Optimization (DPO) ( 16 ) bypasses explicit reward modeling by leveraging the closed-form solution of the KL-constrained objective, \pi^{*}(y\mid x)\propto\pi_{\mathrm{ref}}(y\mid x)\exp\bigl(r(x,y)/\beta\bigr) , to reparameterize the reward as r(x,y)=\beta\log\frac{\pi_{\theta}(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}+\beta\log Z(x) . Substituting into the Bradley-Terry likelihood yields the DPO loss: \mathcal{L}_{\mathrm{DPO}}(\theta)=-\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}}\left[\log\sigma\!\left(\beta\log\frac{\pi_{\theta}(y_{w}\mid x)\,\pi_{\mathrm{ref}}(y_{l}\mid x)}{\pi_{\theta}(y_{l}\mid x)\,\pi_{\mathrm{ref}}(y_{w}\mid x)}\right)\right], (3) which can be optimized with standard supervised learning, eliminating the need for a separate reward model or online RL. Many follow-up methods modify this direct-optimization view, including IPO ( 1 ) , KTO ( 5 ) , and SimPO ( 13 ) ; we summarize these variants in Appendix A . 2.2 Safety Alignment While the methods above improve helpfulness and instruction-following, aligning for preferences alone is insufficient; models must also be steered away from generating harmful content. We assume access to a joint helpfulness–safety dataset \mathcal{D} of the form (x,y_{w},y_{l},s_{w},s_{l})\sim\mathcal{D}, where s_{w}=\mathbf{1}_{\{c(x,y_{w})>0\}},\qquad s_{l}=\mathbf{1}_{\{c(x,y_{l})>0\}} are binary safety indicators . Given this augmented data, safety alignment is naturally formulated as a constrained optimization problem ( 4 ) : \displaystyle\max_{\theta} \displaystyle\mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi_{\theta}(\cdot\mid x)}\left[r(x,y)-\beta D_{\mathrm{KL}}\bigl(\pi_{\theta}(\cdot\mid x)\,\|\,\pi_{\mathrm{ref}}(\cdot\mid x)\bigr)\right], (4) \displaystyle\text{s.t.} \displaystyle c(x,y)\leq 0,\qquad\forall x\sim\mathcal{D},\ y\sim\pi_{\theta}(\cdot\mid x). Existing safety-alignment methods instantiate this constrained view in different ways. Safe RLHF ( 4 ) uses separate reward and cost models with online RL and primal-dual updates, while direct alternatives such as SACPO ( 19 ) , CDPO ( 11 ) , CAN ( 8 ) , and SafeDPO ( 9 ) incorporate safety through multi-stage training, auxiliary models, relabeling, or margin terms. We provide a fuller comparison in Appendix A . In contrast to these method-specific modifications, our goal is to derive safety-aware preference optimization from a unified ratio-matching principle. 2.3 Density Ratio Matching Given two distributions p_{\text{de}}(x) and p_{\text{nu}}(x) , density ratio estimation seeks a parametric model R_{\theta}(x) that approximates the true ratio R_{\text{data}}(x):=p_{\text{nu}}(x)/p_{\text{de}}(x) from i.i.d. samples. Classical approaches include probabilistic classification via logistic regression ( 6 ) , the Kullback–Leibler importance estimation procedure (KLIEP) ( 14 ) , and least-squares importance fitting (LSIF) ( 7 ) . A key insight from 18 is that these techniques can be unified through Bregman divergence minimization ( 3 ) : \displaystyle D_{h}\!\left(R_{\mathrm{data}}(\mathbf{x})\middle\|R_{\theta}(\mathbf{x})\right) \displaystyle=\int p_{\mathrm{de}}(\mathbf{x})\,B_{h}\!\left(R_{\mathrm{data}}(\mathbf{x})\middle\|R_{\theta}(\mathbf{x})\right)\,d\mathbf{x}