Thousand-GPU Large-Scale Training and Optimization Recipe for AI-Native Cloud Embodied Intelligence Infrastructure
2026
Read Thousand-GPU Large-Scale Training and Optimization Recipe for AI-Native Cloud Embodied Intelligence Infrastructure (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-11101.txt.
Introduction
Yongjian Guo and a large team describe a cloud platform that trains embodied intelligence models. It runs on a thousand graphics processors and builds on the LeRobot framework. They reworked the data pipeline, training stack, and infrastructure together. This cut one model's single-round training time from fifteen hours to twenty-two minutes. Storage, an RDMA network, and an elastic data lake keep data, communication, and computation in step. An end-to-end evaluation loop closes the path from training to simulation to assessment.
Catalog record
| YEAR | 2026 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52264 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.11101
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Thousand-GPU Large-Scale Training and Optimization Recipe for AI-Native Cloud Embodied Intelligence Infrastructure Abstract Embodied intelligence is a key step towards Artificial General Intelligence (AGI), yet its development faces multiple challenges including data, frameworks, infrastructure, and evaluation systems. To address these issues, we have, for the first time in the industry, launched a cloud-based, thousand-GPU distributed training platform for embodied intelligence, built upon the widely adopted LeRobot framework, and have systematically overcome bottlenecks across the entire pipeline. At the data layer, we have restructured the data pipeline to optimize the flow of embodied training data. In terms of training, for the GR00T-N1.5 model, utilizing thousand-GPU clusters and data at the scale of hundreds of millions, the single-round training time has been reduced from 15 hours to just 22 minutes, achieving a 40-fold speedup. At the model layer, by combining variable-length FlashAttention and Data Packing, we have moved from sample redundancy to sequence integration, resulting in a 188% speed increase; π 0.5 \pi_{0.5} Attention optimization has accelerated training by 165%; and FP8 quantization has delivered a 140% speedup. On the infrastructure side, relying on high-performance storage, a 3.2T RDMA network, and a Ray-driven elastic AI data lake, we have achieved deep synergy among data, storage, communication, and computation. At the training level, we propose and implement a fully asynchronous strategy training pipeline for the first time — RL-VLA 3 , a triple-level asynchronous architecture covering the entire process from environment interaction and trajectory generation to policy network updates, achieving a maximum throughput increase of 126.67%. We have also built an end-to-end evaluation system, creating a closed loop from training to simulation to assessment. This framework has already been fully validated on thousand-GPU clusters, laying a crucial technical foundation for the development and application of next-generation autonomous intelligent robots, and is expected to accelerate the arrival of the era of human-machine integration. 1]AI Infra Team at JDT 2 Tsinghua University 3 Peking University 4 Tianjin University 5 Beihang University 6 University of Science and Technology of China \contribution Full author list in Contributions \correspondence Junwu Xiong at 1 Introduction Figure 1 : AI-Native Cloud Embodied Intelligence Infrastructure Designing agents with physical-world action capabilities that reach or surpass human-level is one of the key technical paths toward Artificial General Intelligence (AGI) [ 16 ] , a direction known as “Embodied AI.” Embodied intelligence is currently experiencing a period of market expansion: Driven by global innovation and market demand, the embodied AI industry is expanding rapidly at an annual growth rate achieving 39% [ 19 ] , with the market size expected to surpass \$100 billion by 2035 [ 10 ] . In the field of embodied intelligence, traditional dynamics-based embodied strategies mainly address pre-defined task sets in controlled environments, but they have weak generalization abilities and are sensitive to noise in real-world scenarios [ 6 ] . In recent years, breakthroughs in multimodal understanding, reasoning, and utilization of vision and language foundation models [ 23 ] have propelled the rapid development of Vision-Language-Action (VLA) models. VLA models, with powerful cross-modal sequential modeling capabilities, integrate perception, understanding, reasoning, and action generation end-to-end, enabling direct mapping from visual observation and language instructions to physical actions. Existing research has demonstrated their outstanding versatility, flexibility, and generalization to complex environments [ 34 , 3 , 2 ] . Simultaneously, the embodied intelligence ecosystem continues to integrate with world models [ 12 ] , robotic hardware [ 9 ] , open-source datasets [ 5 ] , simulation environments [ 21 , 31 ] , training algorithms [ 22 ] , and computational resource [ 35 ] supply, collectively forming the AI-Native Embodied Intelligence Infrastructure (as shown in Figure 1 ), which provides a solid foundation for the continuous optimization and iteration of VLA-based embodied intelligence systems. Success stories such as the game of Go [ 24 ] and LLMs [ 1 ] show that large-scale, general-purpose computation is essential to move embodied intelligence from scientific research to industrial application. However, achieving large-scale training for embodied intelligence models poses higher demands and challenges for AI infrastructure. Training framework challenges: For thousand-GPU scale embodied training, there is a lack of industrial-grade systems that seamlessly connect simulation, training, and evaluation; multi-dimensional parallelism is complex, making communication and load balancing difficult; large batch data loading is easily blocked by I/O, leading to unstable training and low compute utilization, which hinders efficient large-scale training. Data engine challenges: Mixed storage of multimodal files increases system complexity, and high concurrency puts pressure on metadata processing; frequent file operations lead to increased latency and reduced throughput, affecting GPU data supply; uneven load among nodes may block distributed training; the complexity of data preprocessing further burdens storage. Traditional data lakes struggle to dynamically allocate large files, and serial processing can lead to resource idleness and task blockage, lacking elastic scalability and failing to meet cloud-native high-availability requirements. Model computation challenges: Insufficient dynamic computation and memory optimization, padding in traditional attention mechanisms generates invalid tokens, causing wasted compute and memory redundancy; inefficient training data organization leads to resource waste and low hardware utilization due to short data being padded to fixed lengths; model inference and edge deployment are constrained by real-time requirements and compute resources, making efficient compression and acceleration of small-parameter models key for deployment. Based on these challenges, the AI Infra team at JDT has proposed and implemented a thousand-GPU distributed training framework for embodied intelligence based on the JD Cloud JoyBuilder platform, using the open-source LeRobot framework as its foundation, as shown in Figure 2 . It is equipped with a 3.2T RDMA backend network supporting up to ten thousand GPUs and a flexible VPC frontend network, systematically overcoming bottlenecks across the entire pipeline. Through Yunhai high-performance storage and a Ray-driven elastic AI data lake, the training data pipeline is collaboratively optimized to meet embodied model training needs. For the GR00T-N1.5 [ 2 ] model, in thousand-GPU clusters and billion-scale data scenarios, single-round training time is reduced from 15 hours to 22 minutes, a 40-fold speedup; combining variable-length FlashAttention and Data Packing achieves sequence integration from sample redundancy, accelerating by 188%; custom \pi_{0.5} architecture and post-training quantization improve training and inference efficiency by 165%; FP8 quantization accelerates by 140%; and an end-to-end evaluation system is built to connect the training-simulation-evaluation pipeline. At the VLA training level, we propose and implement a fully asynchronous strategy training pipeline for the first time — RL-VLA 3 [ 11 ] , a triple-level asynchronous architecture covering the entire process from environment interaction and trajectory generation to policy network updates. On the LIBERO benchmark, this framework achieves a maximum throughput improvement of 59.25% compared to existing synchronous training strategies. After further optimization through a decoupling strategy, the throughput improvement reaches 126.67%. This framework has been validated at scale on 256 GPUs clusters, providing robust infrastructure support for the industrialization of embodied intelligence. 2 Core Architecture and Model Optimization 2.1 Overall Architecture Design Figure 2 : The overview of Joybuilder AI-Native Cloud Embodied Intelligence Infrastructure With the continuous growth in the scale of embodied intelligence models and the complexity of data, efficient distributed frameworks are playing an increasingly important role in large-scale training and multi-scenario adaptation. These frameworks integrate VLA models [