dVLA-RL: Reinforcement Learning over Denoising Trajectories for Discrete Diffusion Vision-Language-Action Models
Yuhao Wu, Yitian Liu, Weijie Shen , Mishuo Han, Wenjie Xu, Haotian Liang, Zhongshan Liu, Yinan Mao, Lei Xu, Xinping Guan, Ru Ying, Ran Zheng, Wei Sui, Xiaokang Yang, Wenbo Ding, Yao Mu · 2026
Read dVLA-RL: Reinforcement Learning over Denoising Trajectories for Discrete Diffusion Vision-Language-Action Models by Yuhao Wu, Yitian Liu, Weijie Shen , Mishuo Han, Wenjie Xu, Haotian Liang, Zhongshan Liu, Yinan Mao, Lei Xu, Xinping Guan, Ru Ying, Ran Zheng, Wei Sui, Xiaokang Yang, Wenbo Ding, Yao Mu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-23623.txt.
Introduction
Wu and colleagues bring reinforcement learning to a new class of robot control models, in a 2026 paper on embodied AI. Discrete diffusion vision language action models handle images, instructions, and robot actions as tokens in one shared space, refining actions through repeated denoising steps. Until now they were trained only by supervised fine tuning, imitating demonstrations. Reinforcement learning could improve them, but the probability of the final action is intractable, which blocks standard methods.
The fix is to score the whole denoising path instead of the final action alone. Modeling each denoising step as a decision turns the path probability into a product of step probabilities that can be optimized. A step scheduling scheme gives harder tasks more denoising steps. The method reaches near ninety nine point seven percent success on the LIBERO benchmark and improves a supervised baseline by about thirty points on RoboTwin.
Catalog record
| CREATOR | Yuhao Wu, Yitian Liu, Weijie Shen , Mishuo Han, Wenjie Xu, Haotian Liang, Zhongshan Liu, Yinan Mao, Lei Xu, Xinping Guan, Ru Ying, Ran Zheng, Wei Sui, Xiaokang Yang, Wenbo Ding, Yao Mu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.23623
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
dVLA-RL: Reinforcement Learning over Denoising Trajectories for Discrete Diffusion Vision-Language-Action Models Yuhao Wu, Yitian Liu, Weijie Shen , Mishuo Han, Wenjie Xu, Haotian Liang, Zhongshan Liu, Yinan Mao, Lei Xu, Xinping Guan, Ru Ying, Ran Zheng, Wei Sui, Xiaokang Yang, Wenbo Ding, Yao Mu Abstract Vision-Language-Action (VLA) models have established a powerful paradigm for generalist robotic manipulation by grounding control into the semantic reasoning of large Vision-Language Models (VLMs). Prevailing architectures typically model actions continuously via diffusion or flow processes, or discretely through either autoregressive generation or parallel decoding. Recently, Discrete Diffusion VLAs (dVLAs) have emerged as a distinct alternative, unifying vision, language, and action into a single discrete token space via masked generative modeling. While this paradigm successfully combines iterative action refinement with unified representations, its training has thus far been restricted to Supervised Fine-Tuning (SFT), leaving the potential of Reinforcement Learning (RL) for further policy refinement largely unexplored. A fundamental challenge in RL for dVLAs is that the marginal probability of the final action generated by dVLAs remains intractable. To solve this problem, we propose dVLA-RL , shifting the learning objective from the marginal action probability to the joint probability of the sampled generation path. Specifically, by modeling the denoising process as a Markov Decision Process (MDP), we mathematically formulate this path probability as a product of step-wise transitions. This trajectory-level objective provides a unified formulation that natively accommodates variable denoising steps. Leveraging this intrinsic flexibility, we introduce a unified step scheduling approach for complex multi-task learning, tailoring denoising steps to specific task complexities to maximize both success rates and computational efficiency. Extensive evaluations demonstrate that our approach achieves a success rate of 99.7% on LIBERO. Furthermore, it establishes strong VLA-based results on RoboTwin 2.0 by delivering a 30.6% improvement over the SFT baseline, remaining competitive with strong World-Action Model baselines. Figure 1: Overview of UDVLA-RL. (a) We present a unified probability modeling approach for denoising trajectories, seamlessly integrating 1-step parallel decoding and multi-step denoising paradigms. (b) Extensive evaluations on the LIBERO and RoboTwin 2.0 benchmarks demonstrate that our RL-finetuned policy outperforms both SFT-only baselines and remains highly competitive among modern VLA architectures, achieving performance comparable to World-Action Models. (c) Our hybrid version exhibits superior sample efficiency, achieving higher success rates with substantially fewer environment interaction steps. 1 Introduction Grounding multimodal perception into precise physical actions remains a central challenge in Embodied AI. Vision-Language-Action (VLA) models have advanced robotic control by framing it as a sequence generation task ( 4 ; 3 ; 14 ; 12 ) . However, their reliance on Supervised Fine-Tuning (SFT) exposes them to distributional shift, leading to compounding errors during real-world execution. Reinforcement Learning (RL) provides a principled solution to this bottleneck. By shifting the optimization target from mimicking static datasets to maximizing environment rewards, RL directly targets task success, unlocking unprecedented capability upper bounds for VLA policies ( 15 ) . However, applying policy-gradient RL to VLA policies requires a tractable policy likelihood for the generated action, making the underlying action-generation paradigm a crucial design factor. Existing VLA action-generation paradigms employ distinct RL optimization strategies tailored to their underlying probability representations. Autoregressive and parallel decoding policies formulate action generation through token-wise or single-step predictions, allowing standard policy-gradient algorithms to directly utilize their exact analytical action likelihoods. Conversely, continuous diffusion and flow-matching policies focus on the expressive modeling of complex, multimodal action distributions. To facilitate RL in these continuous spaces, current methods typically evaluate policy gradients through likelihood approximations, variational objectives, or auxiliary stochastic formulations ( 24 ; 20 ; 7 ) . Recently, Discrete Diffusion VLAs (dVLAs) have emerged as a promising new paradigm, unifying multimodal inputs and action chunks into a single discrete token space via masked generative modeling ( 18 ) . While recent works demonstrate strong Supervised Fine-Tuning (SFT) performance comparable to continuous counterparts ( 17 ; 18 ; 6 ; 29 ; 33 ; 21 ) , Reinforcement Learning (RL) for native dVLAs remains largely underexplored. The primary bottleneck lies in the inherent mismatch between the multi-step denoising process and the exact action likelihood required by standard policy-gradient algorithms. Although a dVLA policy outputs a tractable categorical distribution via standard softmax at each individual denoising step, evaluating the exact marginal probability \pi_{\theta}(a_{t}\mid s_{t}) of the final executed action requires marginalizing over all possible intermediate paths within the iterative, K -step Markovian denoising process. This operation is combinatorially expensive; moreover, simple last-step approximations fail to account for the intermediate states that actually dictate the rollout, thereby preventing the direct application of standard RL objectives to dVLA policies. In this work, we address this gap by formulating RL directly over the denoising path executed by the dVLA. Instead of optimizing the intractable marginal probability of the final action, we unroll the denoising process and express the probability of the sampled generation path as a product of step-wise denoising transitions. This trajectory-level formulation aligns the PPO objective with the actual action-generation process used during rollout and deployment. In practice, each transition consists of a mask-scheduling decision and token-generation probabilities. Since the scheduler is a discrete selection process, we treat it as part of the generation dynamics and compute policy-gradient updates only on the newly denoised action tokens along the executed path. This yields a tractable and stable objective for applying standard PPO to multi-step discrete diffusion VLA policies. Importantly, this trajectory-level formulation also unifies different denoising horizons. A 1-step decoder can be viewed as a degenerate denoising path with a single transition, while 2-step and 4-step decoding correspond to longer generation paths under the same probability framework. This flexibility enables a hybrid denoising-step strategy: different tasks can use different denoising horizons according to their initial SFT performance and refinement requirements. For tasks where low-step decoding already provides a strong initial success rate, fewer denoising steps reduce rollout and inference cost. For tasks that require more iterative refinement, larger denoising horizons provide better initial actions and more informative online exploration. In this way, the same trajectory-level RL framework can balance action quality, sample efficiency, and computational cost. Built on these ideas, we present dVLA-RL , a reinforcement learning framework for native discrete diffusion VLA policies. dVLA-RL enables PPO-style online RL by optimizing the probability of sampled denoising paths rather than approximating the marginal likelihood of final actions. Empirically, we evaluate dVLA-RL on LIBERO and RoboTwin 2.0. The results show that dVLA-RL consistently improves its SFT-only discrete diffusion VLA backbone, achieves state-of-the-art performance on LIBERO, and obtains strong VLA-based performance on RoboTwin 2.0 while remaining competitive with strong World-Action Model baselines. Further analyses demonstrate the necessity of trajectory-level probability modeling and show that the hybrid denoising-step strategy improves efficiency while preserving the benefits of multi-step action refinement. In summary, our core contributions are threefold: • We formulate reinforcement learning for native discrete diffusion VLAs by optimizing the probability of sampled denoising trajectories instead of the intractable marginal likelihood of final actions. This provides a simple and tractable way to integrate standard policy-gradient RL with multi-step discrete denoising policies. • We introduce dVLA-RL , a unified denoising-step training strategy enabled by our trajectory-level probability formulation. By assigning task-wise denoising horizons according to task complexity and initial policy performance, our method reduces computational cost on