AcceRL: A Distributed Asynchronous Reinforcement Learning and World Model Framework for Vision-Language-Action Models
Chengxuan Lu, Shukuan Wang, Yanjie Li, Wei Liu, Shiji Jin, Fuyuan Qian, [0.5ex] Peiming Li, Baigui Sun, Yang Liu · 2026
Read AcceRL: A Distributed Asynchronous Reinforcement Learning and World Model Framework for Vision-Language-Action Models by Chengxuan Lu, Shukuan Wang, Yanjie Li, Wei Liu, Shiji Jin, Fuyuan Qian, [0.5ex] Peiming Li, Baigui Sun, Yang Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-18464.txt.
Introduction
Lu and co-authors present AcceRL, a distributed asynchronous framework for training vision-language-action models with world models. Synchronous training wastes hardware on synchronization barriers and costly environment data. AcceRL isolates rollouts, inference, and gradient updates so none waits on the others. Plug-and-play world models join the pipeline in a modular way. It runs 2.4 times faster than synchronous baselines and, with a pretrained world model, uses up to 200 times fewer online samples.
Catalog record
| CREATOR | Chengxuan Lu, Shukuan Wang, Yanjie Li, Wei Liu, Shiji Jin, Fuyuan Qian, [0.5ex] Peiming Li, Baigui Sun, Yang Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53028 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.18464
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AcceRL: A Distributed Asynchronous Reinforcement Learning and World Model Framework for Vision-Language-Action Models Chengxuan Lu, Shukuan Wang, Yanjie Li, Wei Liu, Shiji Jin, Fuyuan Qian, [0.5ex] Peiming Li, Baigui Sun, Yang Liu Abstract Reinforcement learning (RL) for large-scale Vision-Language-Action (VLA) models faces significant challenges in computational efficiency and data acquisition. We propose AcceRL, a fully asynchronous and decoupled RL framework designed to eliminate synchronization barriers by physically isolating training, inference, and rollouts. Crucially, AcceRL is the first to integrate a plug-and-play, trainable world model into a distributed asynchronous RL pipeline to generate virtual experiences. Experiments on the LIBERO [ 16 ] benchmark demonstrate that AcceRL achieves state-of-the-art (SOTA) performance. Systematically, it exhibits super-linear scaling in throughput and highly efficient hardware utilization. Algorithmically, the world-model-augmented variant delivers unprecedented sample efficiency and robust training stability in complex control tasks. Code is publicly available at https://github.com/distanceLu/AcceRL . 1 1 footnotetext: Wolf 1069B, Sany Group. 1 1 footnotetext: Equal contribution. 2 2 footnotetext: Corresponding authors: Baigui Sun ( [email protected] ) and Yang Liu ( [email protected] ). K eywords Asynchronous Reinforcement Learning \cdot Embodied Intelligence \cdot World Model Figure 1: Overview of AcceRL . (Top Left) The framework’s throughput exhibits super-linear scaling with the number of trainer GPUs, enabled by ZeRO [ 18 ] optimizations. (Bottom Left) Experimental results on the LIBERO benchmark demonstrate that AcceRL achieves SOTA performance across all evaluation categories. (Middle) A conceptual illustration of the world model in reinforcement learning, representing the agent “learning in imagination” to significantly boost data utilization. (Right) Performance comparison between model-based (AcceRL with WM) and model-free (AcceRL) approaches. By leveraging a world model pre-trained on 1,000 offline trajectories, the model-based architecture improves online sample efficiency by 200\times . 1 Introduction The field of embodied artificial intelligence has been profoundly transformed by the advent of Vision-Language-Action (VLA) models. These models promise to create agents capable of executing complex tasks based on natural language instructions. While imitation learning has enabled significant capabilities, it suffers from poor generalization and compounding errors known as covariate shift. To overcome these barriers, the research community has turned to reinforcement learning (RL) to optimize policies for long-term success and online fine-tuning. However, applying RL to large-scale VLAs introduces severe systemic bottlenecks primarily due to the synchronization barriers in traditional frameworks. In a conventional synchronous setup, the system must frequently switch between environment interaction, policy inference, and model updates. This lockstep execution forces high-performance GPUs to idle while waiting for slow physical simulators. This creates a significant long-tail problem where the overall throughput is capped by the slowest rollout worker. Furthermore, even with optimized asynchronous execution, the training process remains constrained by the inherent low frequency of physical simulators and the extreme data hunger of large-scale VLA architectures. Relying solely on real-world or simulated interactions imposes a prohibitive temporal cost because the sampling speed is strictly limited by the simulator stepping and the model inference latency. There is an urgent need for a mechanism that can break free from the temporal constraints of physical simulation by generating high-fidelity synthetic experiences to accelerate policy refinement. To bridge this gap, we introduce AcceRL, a modular and fully asynchronous framework tailored for large-scale VLA models. AcceRL achieves a paradigm shift by physically isolating training, inference, and sampling into independent asynchronous streams. Our key contributions are as follows: • High-throughput asynchronous RL architecture: We propose a fully decoupled infrastructure that removes synchronization barriers. By eliminating frequent mode-switching and mitigating long-tail latencies, AcceRL maximizes hardware utilization and achieves super-linear scaling in system throughput. • First distributed asynchronous framework with a trainable world model: AcceRL seamlessly incorporates both world model training and plug-and-play inference pools directly into its pipeline. By enabling “learning in imagination,” the framework bypasses high-cost real-world interactions and continuously refines the world model alongside the policy. This architectural innovation improves online sample efficiency by an astonishing 200\times (20,000%). • Empirical excellence and SOTA performance: Evaluated on the challenging LIBERO benchmark, AcceRL achieves state-of-the-art (SOTA) task performance. The framework demonstrates both highly efficient systemic scalability and robust training stability in complex embodied control tasks. 2 Related work 2.1 Evolution of Distributed Reinforcement Learning Frameworks Historically, scaling reinforcement learning has relied on distributed frameworks to accelerate data acquisition. Even contemporary large-scale systems, such as the default veRL framework [ 22 ] and the synchronous baselines evaluated in RLinf [ 24 ] [ 25 ] , still rely on a synchronous mechanism. In this lockstep paradigm, the central learner must wait for all distributed rollout workers to finish collecting trajectories before performing a gradient update. While this strict synchronization prevents severe trajectory staleness and maintains the bounded policy lag required by algorithms like PPO [ 20 ] , the overall system throughput is dictated by the slowest worker. This phenomenon is known as the straggler effect [ 5 ] , which is particularly exacerbated in large-scale generative tasks due to highly variable trajectory lengths. As a result, expensive compute resources are frequently left idle. To mitigate these synchronization overheads, frameworks evolved towards asynchronous architectures. General-purpose systems like Ray RLlib [ 15 ] introduced asynchronous task scheduling. However, their underlying architectures were fundamentally designed for lightweight models. When applied to multi-billion parameter Vision-Language-Action (VLA) models, they lack native deep integration with large-model optimization techniques such as efficient parameter offloading or model parallelism, leading to prohibitive memory bottlenecks. Addressing this coupling, IMPALA [ 6 ] pioneered actor-learner decoupling; yet, by retaining model inference on rollout workers, it still incurs massive memory costs when deploying such large models. Later, Laminar [ 21 ] introduced a fully decoupled architecture with trajectory-level asynchrony to solve long-tail latency. However, its design is tailored for closed-loop autoregressive generation within memory, making it structurally misaligned with the external, step-by-step causal dependencies of physical simulators required by embodied agents. Most recently, RL-VLA 3 [ 7 ] adapted these principles to Embodied AI by separating environment simulation from inference to saturate GPU utilization. While optimizing execution, it remains fundamentally bottlenecked by the slow speed of the physical simulator itself. This historical evolution highlights a critical unmet need for a fully decoupled architecture that can bypass the inherent latency of physical simulators to scale embodied foundation models. However, their design is primarily tailored for closed-loop autoregressive generation entirely within memory, making them structurally misaligned with the external, step-by-step causal dependencies required by embodied agents interacting with physical simulators. To bridge this gap, domain-specific frameworks have emerged. For instance, RLinf-VLA [ 25 ] introduces a hybrid fine-grained pipeline to overlap rendering and training for VLA models. Yet, it remains fundamentally bottlenecked by the inherently slow stepping speed of the external physical simulator itself. Similarly, RL-VLA 3 [ 7 ] attempts to saturate GPU utilization by strictly decoupling environment simulation from model inference via asynchronous queues. While optimizing raw execution speed, this asynchronous architecture exposes two critical vulnerabilities: it inevitably introduces severe trajectory staleness and massive cross-node communication overhead for high-dimensional multimodal data, thereby significantly degrading sample efficiency. In contrast, AcceRL extends this architectural decoupling to a fundamentally new paradigm by integrating a plug-and-play World Model. This allows the system to bypass the high-latency physical simulators through zero-offload, distributed imagination rollouts. By permanently keeping both the policy and world models on dedicated GPU