DARE: Diffusion Large Language Models Alignment and Reinforcement Executor

Jingyi Yang & Yuxian Jiang & XuHao Hu, Shuang Cheng, Biqing Qi & Jing Shao · 2026

Read DARE: Diffusion Large Language Models Alignment and Reinforcement Executor by Jingyi Yang & Yuxian Jiang & XuHao Hu, Shuang Cheng, Biqing Qi & Jing Shao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-04215.txt.

Introduction

Jingyi Yang and team presented DARE, an open framework for post-training and testing diffusion language models. The open-source world for these models is fragmented across families and post-training pipelines, slowing work and fair tests. Built on verl and OpenCompass, DARE unifies supervised fine-tuning, parameter-efficient fine-tuning, preference optimization, and diffusion reinforcement learning under one execution stack for masked and block models. Broad tests across LLaDA, Dream, SDAR, and LLaDA2.x models mark DARE as a reusable base for building, comparing, and shipping post-training methods.

Catalog record

CREATORJingyi Yang & Yuxian Jiang & XuHao Hu, Shuang Cheng, Biqing Qi & Jing Shao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29453
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.04215

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DARE: Diffusion Large Language Models Alignment and Reinforcement Executor Jingyi Yang & Yuxian Jiang & XuHao Hu, Shuang Cheng, Biqing Qi & Jing Shao Abstract Diffusion large language models (dLLMs) are emerging as a compelling alternative to dominant autoregressive models, replacing strictly sequential token generation with iterative denoising and parallel generation dynamics. However, their open-source ecosystem remains fragmented across model families and, in particular, across post-training pipelines, where reinforcement learning objectives, rollout implementations and evaluation scripts are often released as paper-specific codebases. This fragmentation slows research iteration, raises the engineering burden of reproduction, and makes fair comparison across algorithms difficult. We present DARE ( d LLMs A lignment and R einforcement E xecutor), an open framework for post-training and evaluating dLLMs. Built on top of verl 44 and OpenCompass 15 , DARE unifies supervised fine-tuning, parameter-efficient fine-tuning, preference optimization, and dLLM-specific reinforcement learning under a shared execution stack for both masked and block diffusion language models. Across representative model families including LLaDA, Dream, SDAR, and LLaDA2.x, DARE provides broad algorithmic coverage, reproducible benchmark evaluation, and practical acceleration. Extensive empirical results position that DARE serves as a reusable research substrate for developing, comparing, and deploying post-training methods for current and emerging dLLMs. Code: https://github.com/yjyddq/DARE 1 Introduction Diffusion large language models (dLLMs) 38 ; 53 ; 9 ; 31 ; 19 ; 45 have rapidly evolved from an intriguing alternative to autoregressive language modeling into a growing family of practical model architectures. Early masked diffusion large language models such as LLaDA 38 and Dream 53 , as well as recent block diffusion or semi-autoregressive variants such as SDAR 9 and LLaDA2.x 6 ; 5 , show that diffusion-style generation can support flexible token order, bidirectional conditioning, and parallelism. As these models mature, however, the main bottleneck is shifting away from model definition and toward post-training and evaluation infrastructure. The current open-source dLLM ecosystem is highly fragmented. Most dLLM-based reinforcement learning (RL) methods are released as paper-specific repositories, each with its own model fork, rollout implementation, reward interface, and evaluation scripts. This fragmentation creates at least three problems. First, it slows research iteration because integrating a new model or objective may requires re-implementing the surrounding infrastructure. Second, it makes cross-paper comparison unreliable because algorithmic differences are entangled with execution and evaluation differences. Third, it raises the engineering barrier for researchers who want to study, use and extend prior work. This systems gap is becoming more important, not less. Importantly, it cannot be solved by directly reusing LLM RL frameworks. Most LLM post-training pipelines assume left-to-right generation, sequence log-probabilities over a single decoding trajectory, and rollout engines built for autoregressive serving. dLLMs instead require diffusion-aware forward and reverse processes, denoising-state likelihood surrogates, and model-family-specific rollout backends. To address this need, we present DARE ( d LLMs A lignment and R einforcement E xecutor), a unified post-training and evaluation framework for diffusion large language models. DARE is built on top of verl 44 for training and OpenCompass 15 for evaluation, while adding the missing dLLM-specific execution layers or modules. It integrates a broad set of post-training recipes, including supervised fine-tuning (SFT), parameter-efficient fine-tuning (PEFT), preference optimization, and multiple dLLM-specific RL algorithms, and exposes them through a shared pipeline. The framework supports both masked diffusion language models (MDLMs) and block diffusion language models (BDLMs), enabling one infrastructure stack to serve LLaDA 38 , Dream 53 , SDAR 9 , LLaDA-MoE 60 , and LLaDA2.x 6 ; 5 families. DARE also treats systems optimization as a first-class part of dLLM post-training. For MDLMs, the framework decouples the attention backend used during rollout from the one used during actor updates: rollout uses Fast-dLLM 49 together with FlashAttention backends 17 ; 18 , while training uses variable-length backends of FlashAttention to reduce padding overhead. For BDLMs, DARE integrates LMDeploy 14 ; 55 , SGLang 16 , and fused loss kernels for acceleration, and integrates FlexAttention 20 for training. In addition, DARE extends OpenCompass 15 with dLLM-aware evaluation so that benchmarking is part of the same reusable framework. DARE is not proposed at the method-level. Instead, it is a reusable research substrate that makes it possible to integrate, compare, and evaluate rapidly growing dLLM post-training methods within a unified execution environment. Our contributions are four-fold: • We present DARE, a unified framework that brings together dLLM post-training and evaluation under a single open-source stack, spanning masked and block diffusion language model families. • We integrate a broad range of post-training methods, including SFT, PEFT, preference optimization, and multiple dLLM-tailored RL algorithms, enabling fairer algorithm comparison. • We implement extensive dLLM-specific system optimizations for rollout, training, and evaluation. • We empirically show that DARE provides broad model and benchmark coverage while turning fragmented paper-specific implementations into a more reproducible and comparable dLLM research workflow. 2 Related Works 2.1 Diffusion Language Models Masked Diffusion Language Models. D3PM 3 formulates the discrete diffusion by modeling a sequence of corruption processes over categorical variables, for example with an absorbing state ( <MASK> ) or uniform noise 3 ; 50 ; 42 ; 35 ; 58 ; 22 ; 39 ; 37 . For masked diffusion language models, the categorical distribution q(x_{t}^{\ell}|x_{0}^{\ell}) is parameterized by a linear interpolation between the original one-hot vector \mathbf{x}_{0}^{\ell} and the absorbing vector \mathbf{m} ( <MASK> token): q(x_{t}^{\ell}|x_{0}^{\ell})=\text{Cat}(x_{t}^{\ell};\alpha_{t}\mathbf{x}_{0}^{\ell}+(1-\alpha_{t})\mathbf{m}), (1) where \alpha_{t} denotes the noise schedule at timestamp t . The mask predictor p_{\theta} is trained to reverse the corruption process by minimizing the negative evidence lower bound objective (NELBO): \displaystyle\mathcal{L}_{\theta}= \displaystyle\mathbb{E}_{x_{0}\sim p_{\text{data}},{x}_{t}\sim q({x}_{t}|{x}_{0}),t\sim\mathcal{U}(0,1]}\left[-\frac{1}{t}\sum_{\ell=1}^{L}\mathbb{I}[{x}_{t}^{\ell}=\texttt{<MASK>}]\log p_{\theta}({x}_{0}^{\ell}|{x}_{t})\right]. (2) Recent work 38 ; 53 ; 31 ; 19 ; 45 has scaled this paradigm to large language modeling. These models motivate diffusion-specific rollout and training pipelines because their generation process differs fundamentally from autoregressive left-to-right decoding. Block Diffusion Language Models. In parallel, block diffusion paradigms 25 ; 2 ; 21 ; 6 ; 5 have produced models such as SDAR 9 and LLaDA2.x 6 ; 5 , which combine intra-block diffusion with inter-block autoregression and support variable-length generation with kv-caching. Specifically, a sequence x is partitioned into B contiguous, non-overlapping blocks, \{x^{1},\dots,x^{B}\} , each containing L^{\prime}=\frac{L}{B} tokens. The likelihood factorizes over block as \log p_{\theta}(x)=\sum_{b=1}^{B}\log p_{\theta}(x^{b}|x^{<b}) , and block-wise NELBO is given by: \displaystyle\mathcal{L}_{\theta}= \displaystyle\mathbb{E}_{x_{0}\sim p_{\text{data}},b\sim\mathcal{U}[1,B],t\sim\mathcal{U}(0,1]}\left[-\frac{1}{t}\sum_{\ell=1}^{L^{\prime}}\mathbb{I}[{x}_{t}^{b,\ell}=\texttt{<MASK>}]\log p_{\theta}({x}_{0}^{b,\ell}|{x}_{t}^{b},x^{<b})\right] (3) where {x}_{0}^{b,\ell} and {x}_{t}^{b} denote the clean and corrupted sequences of block b , respectively. The p_{\theta}(x^{b}|x^{<b}) is trained to recover the clean block x^{b} from its noisy counterpart conditioned on preceding clean blocks x^{<b} . This regime introduces different systems requirements from masked diffusion, especially for rollout backends, block-wise verification, and online policy updates. The existence of both MDLM and BDLM families is one reason a unified dLLM executor is needed. 2.2 Post-Training Frameworks and Reinforcement Learning for dLLMs Since 2025, dLLM post-training has developed rapidly. Early work has adapted policy gradient-based and RL algorithms to masked diffusion large language models (MDLMs). For example, VRPO 59 is designed for human preference alignment in dLLM. d1 57 and Coupled-GRPO 23 directly adapt GRPO 7 ; 43 ; 24 for one-step denoising optimization. MDPO 27 addressed the training-inference mismatch by optimizing the progressive refinement schedule used at inference. CJ-GRPO 51 emphasized consistency between rollout and optimization trajectories, while DiFFPO 56 studied off-policy surrogate policies and joint optimization

Read the complete text