D-VLA: A High-Concurrency Distributed Asynchronous Reinforcement Learning Framework for Vision-Language-Action Models

Yucheng Guo, Yongjian Guo, Zhong Guan, Wen Huang, Haoran Sun, Haodong Yue, Xiaolong Xiang, Shuai Di, Zhen Sun, Luqiao Wang, Junwu Xiong, Yicheng Gong · 2026

Read D-VLA: A High-Concurrency Distributed Asynchronous Reinforcement Learning Framework for Vision-Language-Action Models by Yucheng Guo, Yongjian Guo, Zhong Guan, Wen Huang, Haoran Sun, Haodong Yue, Xiaolong Xiang, Shuai Di, Zhen Sun, Luqiao Wang, Junwu Xiong, Yicheng Gong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13276.txt.

Introduction

Guo and colleagues present D-VLA, a distributed training framework for vision language action models in robots. Training such models clashes with high fidelity simulation over memory and bandwidth, choking throughput. D-VLA separates fast training data from slow weight control and runs sampling, inference, gradients and weight sharing in parallel lanes. Tests show large gains in throughput and sampling efficiency, with stable near linear speedups at trillion parameter scale.

Catalog record

CREATORYucheng Guo, Yongjian Guo, Zhong Guan, Wen Huang, Haoran Sun, Haodong Yue, Xiaolong Xiang, Shuai Di, Zhen Sun, Luqiao Wang, Junwu Xiong, Yicheng Gong
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33765
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.13276

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

D-VLA: A High-Concurrency Distributed Asynchronous Reinforcement Learning Framework for Vision-Language-Action Models Yucheng Guo, Yongjian Guo, Zhong Guan, Wen Huang, Haoran Sun, Haodong Yue, Xiaolong Xiang, Shuai Di, Zhen Sun, Luqiao Wang, Junwu Xiong, Yicheng Gong Abstract The rapid evolution of Embodied AI has enabled Vision-Language-Action (VLA) models to excel in multimodal perception and task execution. However, applying Reinforcement Learning (RL) to these massive models in large-scale distributed environments faces severe systemic bottlenecks, primarily due to the resource conflict between high-fidelity physical simulation and the intensive VRAM/bandwidth demands of deep learning. This conflict often leaves overall throughput constrained by execution-phase inefficiencies. To address these challenges, we propose D-VLA , a high-concurrency, low-latency distributed RL framework for large-scale embodied foundation models. D-VLA introduces "Plane Decoupling," physically isolating high-frequency training data from low-frequency weight control to eliminate interference between simulation and optimization. We further design a four-thread asynchronous "Swimlane" pipeline, enabling full parallel overlap of sampling, inference, gradient computation, and parameter distribution. Additionally, a dual-pool VRAM management model and topology-aware replication resolve memory fragmentation and optimize communication efficiency. Experiments on benchmarks like LIBERO show that D-VLA significantly outperforms mainstream RL frameworks in throughput and sampling efficiency for billion-parameter VLA models. In trillion-parameter scalability tests, our framework maintains exceptional stability and linear speedup, providing a robust system for high-performance general-purpose embodied agents. 1 Introduction Figure 1 : Placement Strategies across Different Training Frameworks Embodied AI, regarded as a pivotal pathway toward Artificial General Intelligence (AGI), is undergoing a profound paradigm shift driven by the emergence of Vision-Language-Action (VLA) models 28 ; 2 ; 7 ; 11 ; 20 such as OpenVLA 11 , \pi_{0} 2 , and GR00T 1 . These models achieve a significant transition from manually designed explicit models to data-driven implicit models by integrating visual perception, language understanding, and action generation into a unified end-to-end framework 14 ; 26 ; 25 . Through continuous scaling across massive computational resources and datasets, VLA models have demonstrated unprecedented potential in cross-task and cross-morphology adaptation. However, despite these notable advancements, current training paradigms remain heavily reliant on imitation learning based on Supervised Fine-Tuning (SFT). Existing frameworks like LeRobot 3 and GR00T 1 primarily utilize expert demonstration data to fine-tune policies via behavior cloning. This SFT-centric path faces multiple severe challenges in practical applications: first, large-scale human-collected robot trajectory data is both costly and difficult to obtain, which strictly limits the scaling of models 19 ; 9 ; second, processing high-dimensional data and complex multimodal architectures imposes a heavy burden on training cycles and inference latency. Furthermore, constrained by the limited diversity of offline datasets, SFT models often exhibit weak generalization capabilities when encountering distribution shifts and unseen tasks 4 ; 21 . Unlike human learning mechanisms, SFT cannot support agents in discovering new action patterns through autonomous exploration beyond demonstration data. Consequently, the research community is increasingly pivoting toward Reinforcement Learning (RL) frameworks to break through these limitations via online interaction. In response to the deficiencies of SFT, several RL frameworks have emerged, optimizing different dimensions of the training pipeline. For instance, RLinf 24 provides a universal distributed training and evaluation interface, offering a unified framework for multimodal agents and VLAs. RL-VLA 3 8 implements a three-stage asynchronous pipeline to decouple data collection, policy inference, and model updates, thereby maximizing hardware utilization and training throughput. SimpleVLA-RL 12 designs rule-based outcome rewards and interactive trajectory sampling for VLA models, proving that performance can surpass SFT even with minimal demonstrations. Additionally, Dexbotix 22 focuses on integrating tactile feedback for high-degree-of-freedom dexterous hands, while Vlab 5 is dedicated to building specialized sim-to-real transfer environments. Unlike traditional online RL, Embodied AI training involves deep coupling between high-fidelity physics simulation and large-scale deep learning models. The former features high-frequency, fragmented occupation of computational resources, while the latter demands extreme throughput for GPU memory capacity and communication bandwidth. Existing distributed training frameworks, such as RLinf-VLA 24 and RL-VLA 3 8 , introduce hybrid resource allocation and fine-grained asynchronous mechanisms to alleviate computational pressure. However, they have yet to fundamentally resolve the resource contention and execution conflicts between simulation tasks and model optimization at the underlying architecture level. As a result, the overall system throughput remains bottlenecked by the slowest physics stepping or synchronization overhead. The current performance bottlenecks in embodied RL systems primarily stem from the high degree of coupling between simulation logic and learning logic on the execution plane. On one hand, frequent memory allocation and deallocation by physics engines easily lead to severe memory fragmentation in deep learning frameworks. On the other hand, the frequent transfer of massive multimodal environment data (e.g., high-resolution images) between sampling and inference components introduces significant serialization overhead and communication latency. This systemic "blocking" effect is particularly severe when handling long-sequence interaction tasks, restricting the sample acquisition efficiency of agents in complex scenarios. To address these challenges, we propose D-VLA , a high-performance distributed embodied reinforcement learning framework. The core innovation of this framework lies in the design philosophy of "Plane Decoupling," which physically isolates the high-frequency Data Plane from the low-frequency weight Control Plane during the training process, eliminating interference between simulation and training tasks from the ground up. Based on this concept, we have constructed a four-thread asynchronous execution pipeline—the "Swimlane" model. By parallelizing sampling, weight reception, gradient training, and parameter distribution, we achieve full overlap of computation and communication. To further optimize heterogeneous resource utilization, D-VLA introduces a dual-pool memory management model and a zero-copy data exchange mechanism, supporting various flexible placement strategies including co-located, separated, and hybrid deployments, as shown in Figure 1 . By combining Group Relative Policy Optimization (GRPO) 18 with local topology replication scaling techniques, D-VLA successfully breaks through the scalability bottlenecks of large-scale interactive data processing, providing stable support for the training of ultra-large-scale VLA models. The primary contributions of this paper are summarized as follows: • "Plane Decoupling" and Four-Thread Asynchronous Pipeline Architecture: We propose a system design that physically isolates the high-frequency data interaction plane from the low-frequency weight control plane. Through an innovative four-thread "Swimlane" parallel mechanism, we achieve complete computational overlap of data sampling, policy inference, gradient training, and parameter distribution, resolving resource conflicts between embodied simulation and model optimization at the architectural level. • Hierarchical Memory Management and Topology-Aware Scaling Strategies: we introduce a dual-pool GPU memory management model and a zero-copy data exchange mechanism to effectively alleviate memory fragmentation caused by physics engines. Meanwhile, through local topology replication and control plane offloading techniques, we significantly reduce cross-node communication latency and optimize the communication-to-computation ratio while maintaining global consistency for trillion-parameter models. • Performance Breakthrough and Validation in Large-Scale Embodied Tasks: By integrating the GRPO algorithm into the D-VLA framework and conducting extensive validation on complex benchmarks such as LIBERO, we demonstrate that the system possesses superior stability and sampling efficiency when handling long-sequence, large-scale interactive data, significantly outperforming existing mainstream distributed RL baselines. 2 Related Work Evolution of Vision-Language-Action Models In the field of Embodied AI, the evolution of Vision-Language-Action (VLA) models is undergoing a paradigm shift from fundamental Supervised Fine-Tuning (SFT) 27 toward Reinforcement Learning (RL) 5 frameworks with enhanced generalization capabilities. Early foundational VLA models such as RT-2 28 , OpenVLA 11 , and \pi_{0} 2 achieved

Read the complete text