Beyond Human Demonstrations: Diffusion-Based Reinforcement Learning to Generate Data for VLA Training
Rushuai Yang, Hangxing Wei, Ran Zhang, Zhiyuan Feng, Xiaoyu Chen, Tong Li, Chuheng Zhang, Li Zhao, Jiang Bian, Xiu Su, Yi Chen · 2025
Read Beyond Human Demonstrations: Diffusion-Based Reinforcement Learning to Generate Data for VLA Training by Rushuai Yang, Hangxing Wei, Ran Zhang, Zhiyuan Feng, Xiaoyu Chen, Tong Li, Chuheng Zhang, Li Zhao, Jiang Bian, Xiu Su, Yi Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-19752.txt.
Introduction
Robot models that see, read, and act need huge sets of demonstrations, and filming humans is slow and costly. Rushuai Yang and ten co-authors let reinforcement learning manufacture the demonstrations instead. A diffusion-based policy explores varied, smooth behaviors on long manipulation tasks. Its rollouts become training data for a vision-language-action model.
Ordinary reinforcement learning often fails on such tasks because rewards are sparse. The denoising process steadies the generated trajectories, lowering their variance. Tests use LIBERO, a benchmark of one hundred thirty long tasks. Synthetic demonstrations could scale robot training the way synthetic text scaled language models. The pipeline keeps a human-free loop from practice to policy. Quality of the generated data, the authors argue, is what makes the swap work.
Catalog record
| CREATOR | Rushuai Yang, Hangxing Wei, Ran Zhang, Zhiyuan Feng, Xiaoyu Chen, Tong Li, Chuheng Zhang, Li Zhao, Jiang Bian, Xiu Su, Yi Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-19752
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Human Demonstrations: Diffusion-Based Reinforcement Learning to Generate Data for VLA Training Rushuai Yang, Hangxing Wei, Ran Zhang, Zhiyuan Feng, Xiaoyu Chen, Tong Li, Chuheng Zhang, Li Zhao, Jiang Bian, Xiu Su, Yi Chen Abstract Vision-language-action (VLA) models have shown strong generalization across tasks and embodiments; however, their reliance on large-scale human demonstrations limits their scalability owing to the cost and effort of manual data collection. Reinforcement learning (RL) offers a potential alternative to generate demonstrations autonomously, yet conventional RL algorithms often struggle on long-horizon manipulation tasks with sparse rewards. In this paper, we propose a modified diffusion policy optimization algorithm to generate high-quality and low-variance trajectories, which contributes to a diffusion RL-powered VLA training pipeline. Our algorithm benefits from not only the high expressiveness of diffusion models to explore complex and diverse behaviors but also the implicit regularization of the iterative denoising process to yield smooth and consistent demonstrations. We evaluate our approach on the LIBERO benchmark, which includes 130 long-horizon manipulation tasks, and show that the generated trajectories are smoother and more consistent than both human demonstrations and those from standard Gaussian RL policies. Further, training a VLA model exclusively on the diffusion RL-generated data achieves an average success rate of 81.9%, which outperforms the model trained on human data by +5.3% and that on Gaussian RL-generated data by +12.6%. The results highlight our diffusion RL as an effective alternative for generating abundant, high-quality, and low-variance demonstrations for VLA models. I Introduction Vision-language-action (VLA) is a promising model toward general-purpose robots capable of generalizing across a wide array of manipulation tasks [ 1 , 2 , 3 ] . However, this paradigm is critically dependent on massive datasets of human demonstrations, such as the Open X-Embodiment dataset [ 4 ] . The process of collecting this data via manual teleoperation is notoriously expensive and labor-intensive. The reliance on manual data collection fundamentally caps the scalability of VLA models, presenting a major bottleneck to further progress. Reinforcement learning (RL) has emerged as a powerful paradigm for enabling robots to acquire sophisticated physical skills directly through environmental interaction. The fundamental strength of reinforcement learning stems from its trial-and-error process: by optimizing for a reward signal, an agent can autonomously discover highly effective and efficient strategies that often surpass what can be learned by simply mimicking human demonstrations. However, a significant limitation of this approach is that the resulting policies are often highly specialized. A policy trained to excel under one specific set of conditions typically struggles to adapt or generalize its skills when faced with new task variations or different environmental setups [ 5 ] . However, making a general RL algorithm effective enough to generate high-quality data across diverse, complex manipulation tasks is challenging. The long-horizon, sparse-reward tasks prevalent in benchmarks like LIBERO [ 6 ] expose critical weaknesses of conventional RL algorithms, often leading to unstable learning process or high-variance and suboptimal trajectories [ 7 ] . Fig. 2 : Our diffusion RL method generates a superior data distribution for VLA training. The figure contrasts three types of generated trajectories. (a) Human demonstrations, while successful, are inconsistent and exhibit high-variance, multi-modal behaviors; (b) A conventional Gaussian RL policy produces jittery, suboptimal trajectories that result in poor task success; (c) In contrast, our method consistently generates high-quality, low-variance trajectories that are both smooth and optimal; (d) This superior data quality directly translates to downstream performance, as the VLA model trained on our data significantly outperforms those trained on the others. Quantitative results can be found in Sec. V . To this end, we propose a general framework that utilizes a modified diffusion policy optimization algorithm for diffusion RL-powered data generation. We find that diffusion offer a superior alternative for this problem. First, diffusion policy provide good expressiveness to fit complex expert distribution. Compared with Gaussian RL, diffusion-based RL provides more space for RL exploration when interacting with environment. Second, the inherent structure of the iterative denoising process acts as a powerful implicit regularizer on the action space, The model is trained to predict the noise for the entire action chunk at every step of the denoising process. This forces the model to learn the underlying structure of smooth, physically plausible motions. A single, jerky movement in the final action would require a very specific and complex sequence of denoising steps, which is less likely to be learned than a smooth, coherent refinement process. This naturally encourages the generation of temporally smooth, low-variance motion. We further enhance this process with a stabilized fine-tuning regimen, incorporating modifications to the architecture and training strategies to ensure robust performance across the 130 challenging tasks in LIBERO. This property allows our RL agent to explore more effectively and converge to near-optimal and low-variance policies. Our experiments yield a clear message: A VLA model trained exclusively on our RL-generated data consistently and significantly surpasses the ones trained on human data and Gaussian RL, both on in-distribution tasks and in challenging OOD generalization. A quantitative analysis reveals the mechanism behind this success: Our generated trajectories are smoother and less variable, providing a more stable learning signal for VLA training. Our contributions are threefold: • A diffusion RL-powered VLA training pipeline for autonomously generating high-quality and low-variance data for VLA training, including validated effective modifications on the model architecture and training strategies. • Compelling empirical evidence on the 130 complex manipulation tasks of the LIBERO benchmark shows that our synthetic data provides superior training signal to human demonstrations, significantly improving both the in-distribution success rates and out-of-distribution generalization of VLA models. • An in-depth quantitative analysis that relates trajectory-level properties (e.g., efficiency, smoothness, and consistency) with the performance of fine-tuned VLA, providing a clear explanation for why optimized data is more effective. Fig. 3 : Overview of our diffusion RL framework. (Top-left) The policy network architecture integrates proprioceptive and visual observations via a ResNet backbone and FiLM conditioning to parameterize the denoising process. (Top-right) The policy network samples actions through iterative diffusion denoising, while the value network estimates the value of these actions to guide the policy network’s updates. (Bottom) The training loop executes diffusion-based action sampling within parallel environment rollouts, collecting task-completion rewards to update the networks and improve policy performance. II Related Work II-A Data Generation for VLA Models While many recent VLA models have demonstrated impressive capabilities, their performance often relies on the availability of large-scale datasets. Open-source datasets, such as Open X-Embodiment [ 4 ] and LIBERO [ 6 ] , provide a strong foundation for the field, with most of this data being collected through human teleoperation. A key challenge is that scaling this data collection process requires significant human labor. On another front, some model-based methods generate trajectories for manipulation tasks by solving optimization problems [ 8 ] , but this often requires careful, task-specific algorithmic design. Seeking a more scalable and autonomous approach, we explore using RL for data generation. The ability of RL to discover near-optimal behaviors through online interaction presents a promising path towards generating high-quality data. II-B Reinforcement Learning in Robotic Foundation Models Recent works have started to explore how RL algorithms can be applied to enhance robotic foundation models. One key challenge is how to train