Relative Score Policy Optimization for Diffusion Language Models
Zichao Yu, Shengze Xu, Bingqing Jiang, Wenyi Zhang, Difan Zou · 2026
Read Relative Score Policy Optimization for Diffusion Language Models by Zichao Yu, Shengze Xu, Bingqing Jiang, Wenyi Zhang, Difan Zou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10218.txt.
Introduction
Diffusion language models generate text by gradual denoising rather than word by word, which makes standard reinforcement learning hard to apply. Zichao Yu and four colleagues close that gap. Their 2026 paper proposes Relative Score Policy Optimization, or RSPO, a method for training these models with verifiable rewards, meaning answers that can be checked as right or wrong.
Standard methods need sequence-level log-ratios, the change in likelihood between policies, which diffusion models cannot compute directly. Existing workarounds use noisy approximations that can destabilize training when rewards are high. RSPO instead treats the reward advantage as a target for the relative log-ratio and updates the policy toward closing the gap. On math reasoning and planning benchmarks it gained most strongly on planning tasks and stayed competitive on math.
Catalog record
| CREATOR | Zichao Yu, Shengze Xu, Bingqing Jiang, Wenyi Zhang, Difan Zou |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.10218
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Relative Score Policy Optimization for Diffusion Language Models Zichao Yu, Shengze Xu, Bingqing Jiang, Wenyi Zhang, Difan Zou Abstract Diffusion large language models (dLLMs) offer a promising route to parallel and efficient text generation, but improving their reasoning ability requires effective post-training. Reinforcement learning with verifiable rewards (RLVR) is a natural choice for this purpose, yet its application to dLLMs is hindered by the absence of tractable sequence-level log-ratios, which are central to standard policy optimization. The lack of tractable sequence-level log-ratios forces existing methods to rely on high-variance ELBO-based approximations, where high verifier rewards can amplify inaccurate score estimates and destabilize RL training. To overcome this issue, we propose R elative S core P olicy O ptimization (RSPO), a simple RLVR method that uses verifiable rewards to calibrate noisy likelihood estimates in dLLMs. The core of our algorithm relies on a key observation: a reward advantage can be interpreted not only as an update direction, but also as a target for the relative log-ratio between the current and reference policies. Accordingly, RSPO calibrates this noisy relative log-ratio estimate by comparing its reward advantage with the reward-implied target relative log-ratio, updating the policy according to the gap between the current estimate and the target rather than the raw advantage alone. Experiments on mathematical reasoning and planning benchmarks show that RSPO yields especially strong gains on planning tasks and competitive mathematical-reasoning performance. 1 Introduction Reinforcement learning with verifiable rewards (RLVR) ( He et al. 2025 ; Shao et al. 2024 ; Yu et al. 2025 ; Zheng et al. 2025a ) has become an effective post-training paradigm for improving language models on reasoning tasks: sample candidate answers, verify their correctness, and update the model so that correct answers become more likely ( Sutton et al. 1998 ; Ouyang et al. 2022 ; Jaech et al. 2024 ; Guo et al. 2025 ) . For autoregressive language models, this update is naturally expressed through policy log-ratios, namely the change in log-likelihood of a sampled response under the current policy relative to a reference or previous policy. Because autoregressive models admit an explicit left-to-right factorization, these log-ratios are tractable, enabling methods such as Proximal Policy Optimization (PPO) ( Schulman et al. 2017 ) and Group Relative Policy Optimization (GRPO) ( Shao et al. 2024 ) to combine reward improvement with Kullback–Leibler (KL) control. Diffusion large language models (dLLMs) ( Sahoo et al. 2024 ; Shi et al. 2024 ; Nie et al. ; Yang et al. 2025 ; Bie et al. 2025 ) break this convenient interface. Their generation process proceeds through iterative denoising rather than left-to-right prediction, so the sequence likelihood of a completed response is not directly available. As a result, the sequence-level policy log-ratios used by standard RL objectives become difficult to compute. Recent RL methods for dLLMs therefore replace exact log-ratios with tractable surrogates, such as one-step mean-field estimates ( Zhao et al. 2025 ) , ELBO-based token or sequence scores ( Ou et al. 2025 ) , trajectory-level formulations ( Wang et al. 2025b ) , and evidence-bound objectives ( Wang et al. 2025a ; Lin et al. 2025 ) . These substitutes make policy optimization possible, but they also introduce a new source of uncertainty: in RLVR, the verifier reward is a reliable task signal, whereas the model-side relative score used to apply that reward is only an approximate and noisy estimate of the true likelihood change. Figure 1 : Overview of RSPO: verifier rewards define centered relative-score targets, and the residual to the current ELBO-based score provides policy-update feedback. This mismatch raises a basic question: how should dLLM RLVR combine a reliable verifier reward with a noisy likelihood-ratio surrogate? Standard advantage-weighted objectives address only part of this question. They multiply a model-side score by a reward or advantage coefficient, thereby specifying which responses should be encouraged. When the score is an ELBO-based log-ratio surrogate, however, the coefficient does not depend on the current relative score. A high-advantage response can therefore continue to amplify a noisy likelihood estimate even after its estimated relative likelihood has moved far from the reference. Thus, the reward provides a preference direction, but the objective lacks a calibrated target for the relative score itself. Our key observation is that KL-regularized policy improvement provides exactly this target. The optimal KL-regularized policy assigns each response a current-reference log-ratio determined by its reward, up to a prompt-level normalization term. Within a comparison group, this normalization term is shared and disappears after centering. The centered verifier rewards therefore specify the relative likelihood changes that the improved policy should assign to sampled responses. Based on this principle, we propose R elative S core P olicy O ptimization (RSPO), a simple RLVR objective for diffusion language models. RSPO estimates a centered current-reference score with ELBO differences, derives a centered relative-score target from verifier rewards, and uses the residual gap as feedback in the policy update. In this way, reliable rewards calibrate noisy likelihood-ratio estimates instead of serving only as fixed multiplicative weights. Figure 1 gives an overview of this feedback loop: the policy generates a group of responses for each prompt, the reference model is used to form ELBO-based log-ratio scores, and micro-batch centering turns these scores into relative feedback. RSPO compares this current centered score with the verifier-induced target and optimizes the resulting residual objective; details are deferred to Section 3 . Experiments on mathematical reasoning and planning benchmarks show that RSPO yields large planning gains and competitive mathematical-reasoning performance against existing RL baselines. Our contributions are summarized as follows: • We identify a calibration problem in dLLM RLVR. Verifier rewards provide reliable task-level feedback, but the ELBO-based log-ratios used for policy optimization are noisy likelihood-ratio surrogates. Standard advantage-weighted objectives use these rewards as fixed multiplicative coefficients, which can keep pushing already high-scoring responses and over-amplify noisy relative-score estimates. • We derive a principled relative-score target from KL-regularized policy improvement. Although the optimal policy log-ratio contains an unknown normalization term, this term is shared within a comparison group and can be removed by centering. The resulting centered form shows that verifier rewards specify not only which responses should be preferred, but also the relative likelihood changes an improved policy should assign to them. • We propose RSPO, a simple stop-gradient objective for diffusion language models. Instead of using rewards only as fixed advantages, RSPO compares the current centered ELBO-based relative score with the reward-implied target and updates according to the remaining gap. We further provide a local first-order analysis showing that the RSPO update admits a finite centered-score target and is first-order equivalent to a matched quadratic target objective. • We validate RSPO on mathematical reasoning and planning benchmarks. Across these settings, RSPO improves diffusion language models over existing dLLM RL baselines, supporting our key observations and demonstrating the superior performance of the proposed algorithm. 2 Preliminaries Masked Diffusion Language Models. Masked diffusion models (MDMs) ( Sahoo et al. 2024 ) generate sequences by reversing a gradual masking process. Given a prompt \bm{q} and a clean response \bm{y}=(y^{1},\ldots,y^{L}) , the forward process corrupts \bm{y} into \bm{z}_{t}=(z_{t}^{1},\ldots,z_{t}^{L}) at timestep t\in[0,1] , while keeping \bm{q} fixed. Each token is independently replaced by the mask token [\textsc{mask}] , whose one-hot