Benchmarking Massively Parallelized Multi-Task Reinforcement Learning for Robotics Tasks
Viraj Joshi, Zifan Xu, Bo Liu, Peter Stone, Amy Zhang · 2025
Read Benchmarking Massively Parallelized Multi-Task Reinforcement Learning for Robotics Tasks by Viraj Joshi, Zifan Xu, Bo Liu, Peter Stone, Amy Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-23172.txt.
Introduction
Viraj Joshi and co-authors release a benchmark for teaching one robot policy many tasks at once, trained in massively parallel simulation. Their 2025 paper introduces MTBench, built on a GPU simulator that can run thousands of robot scenes side by side. It bundles fifty manipulation tasks and twenty locomotion tasks into one testbed.
The suite pairs four base reinforcement learning algorithms with seven multi-task methods and architectures under one roof. That standardization matters because multi-task research has mostly used slower, off-policy methods in small setups. On-policy methods, which can reach higher final performance, need fresh data constantly, exactly what parallel simulation supplies. Experiments show the benchmark evaluates approaches far faster than serial setups. They also expose fresh friction when scale meets task variety, which the authors catalogue for the field.
Catalog record
| CREATOR | Viraj Joshi, Zifan Xu, Bo Liu, Peter Stone, Amy Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-23172
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Benchmarking Massively Parallelized Multi-Task Reinforcement Learning for Robotics Tasks Viraj Joshi, Zifan Xu, Bo Liu, Peter Stone, Amy Zhang Abstract Multi-task Reinforcement Learning (MTRL) has emerged as a critical training paradigm for applying reinforcement learning (RL) to a set of complex real-world robotic tasks, which demands a generalizable and robust policy. At the same time, massively parallelized training has gained popularity, not only for significantly accelerating data collection through GPU-accelerated simulation but also for enabling diverse data collection across multiple tasks by simulating heterogeneous scenes in parallel. However, existing MTRL research has largely been limited to off-policy methods like SAC in the low-parallelization regime. MTRL could capitalize on the higher asymptotic performance of on-policy algorithms, whose batches require data from the current policy, and as a result, take advantage of massive parallelization offered by GPU-accelerated simulation. To bridge this gap, we introduce a massively parallelized M ulti- T ask Bench mark for robotics (MTBench), an open-sourced benchmark featuring a broad distribution of 50 manipulation tasks and 20 locomotion tasks, implemented using the GPU-accelerated simulator IsaacGym. MTBench also includes four base RL algorithms combined with seven state-of-the-art MTRL algorithms and architectures, providing a unified framework for evaluating their performance. Our extensive experiments highlight the superior speed of evaluating MTRL approaches using MTBench, while also uncovering unique challenges that arise from combining massive parallelism with MTRL. Code is available at https://github.com/Viraj-Joshi/MTBench 1 Introduction Figure 1 : MTBench is a benchmark that leverages massive parallelism for MTRL in two robotics domains, Parkour and Meta-World, and provides MTRL implementations developed over the years. On the left, we see that IsaacGym’s Tensor API enables us to assign blocks of environments to a desired task within the domain of interest, allowing for the setting and getting of the required information for RL training. Deep reinforcement learning has been successfully applied to a wide range of decision-making tasks, including Atari games ( Mnih et al. 2013 ) , the game of Go ( Silver et al. 2016 ) , and continuous control tasks ( Hwangbo et al. 2019 ; Wurman et al. 2022 ) . While these applications have achieved remarkable task-specific performance, recent research trends have shifted towards developing general-purpose agents capable of solving multiple tasks or adapting to diverse environments ( Cobbe et al. 2020 ; Kirk et al. 2023 ; Park et al. 2024 ) . This transition is partly motivated by the demands of real-world robotics applications, where versatility and robustness are essential. For example, tabletop manipulation often requires acquiring multiple skills to accomplish complex tasks ( Pinto & Gupta 2016 ; Yu et al. 2021 ) and legged locomotion demands adaptability to traverse challenging terrains ( Lee et al. 2020 ; Liang et al. 2024 ) . To facilitate the learning of a general-purpose robotic agent, massively parallelized training (>>1000 simulations) has gained popularity with the advancement of GPU-accelerated simulators ( Liang et al. 2018b ; Freeman et al. 2021 ; Makoviychuk et al. 2021 ; Mittal et al. 2023 ; Tao et al. 2024 ; Zakka et al. 2025 ) . These simulators have significantly mitigated hardware and runtime constraints for learning single tasks , reducing experiment durations from days to minutes ( Liang et al. 2018b ; Rudin et al. 2022 ) . However, in the multi-task setting, no out-of-the-box solution exists to allocate a fixed number of environments per task on a single GPU, allowing for simultaneous diverse data collection and end-to-end MTRL training. Additionally, massively parallelized online batched RL introduces new, non-trivial algorithmic challenges. For example, on-policy methods like PPO reach a saturation point beyond which additional parallelization no longer improves performance ( Singla et al. 2024 ) . Meanwhile, off-policy methods such as SAC and Q-Learning become unstable, losing their sample efficiency compared to on-policy methods as interaction with parallel environments unbalances the replay ratio ( D’Oro et al. 2022 ; Li et al. 2023 ; Gallici et al. 2024 ) . On the other hand, learning general-purpose robotic agents has also motivated multi-task RL (MTRL), which aims to learn a single policy that maximizes average performance across multiple tasks. By leveraging task similarities ( Pinto & Gupta 2016 ) , MTRL often enhances sample efficiency, requiring fewer transitions to match the performance of single-task counterparts. Prior research has primarily focused on addressing optimization challenges introduced by multiple learning signals, either from a gradient-based perspective ( Yu et al. 2020 ; Liu et al. 2024 ; Liu et al. 2023a ) or through neural architecture design ( Yang et al. 2020 ; Sodhani et al. 2021 ; Sun et al. 2022 ; Hendawy et al. 2024 ) . However, these MTRL approaches have been limited to using off-policy methods in low-parallelization settings using libraries like Ray ( Liang et al. 2018a ) . With massive parallelization applied to MTRL, we no longer need to deal with how to distribute experience collection and learning, instead utilizing on-policy algorithms, whose batches require data from current experience and as a result, take advantage of the parallelization offered by GPU-based simulators. To support large-scale MTRL experiments and advance the development of general-purpose robotic agents, we introduce a massively parallelized M ulti- T ask Bench mark for robotics (MTBench). This open-source benchmark includes a diverse set of 50 manipulation tasks and 20 locomotion tasks (right side of Figure 1 ), implemented using the GPU-accelerated simulator IsaacGym. Each task allows for procedurally generating infinitely many variations by modifying factors such as initial states and terrain configurations. Additionally, MTBench integrates four base RL algorithms with seven state-of-the-art MTRL algorithms and architectures, providing a unified framework to evaluate their performance. Based on our experiments, we highlight the following major observations: (O1) On-Policy > Off-Policy: Choosing between on-policy RL methods or off-policy methods affects performance more than the MTRL scheme applied in massively parallel training. Off-policy RL’s asymptotic performance struggles to match on-policy RL in this regime. (O2) Prioritize Wall-Clock Time over Sample Efficiency: In the massively parallel regime, wall-clock efficiency is more critical than sample efficiency, as experience collection scales easily with more GPUs. (O3) Value Learning is the Key Bottleneck in MTRL: Multi-task RL struggles primarily with value estimation rather than policy learning, as gradient conflicts mostly impact the critic function. (O4) Curriculum Learning is Crucial for Sparse-Reward Tasks: MTRL alone does not help exploration in sparse-reward tasks; curriculum learning is essential for overcoming early stagnation. 2 Background 2.1 GPU Accelerated Simulation Traditionally, simulators used for online RL rely on the coordination between CPU and GPU where the CPU handles physics simulation and observation/reward calculations while the GPU handles neural network training and inference, leading to frequent slow memory transfers between the two many times during the RL training process. Now, GPU-accelerated simulators provide access to the results of physics simulation on the GPU, and as a result, we have all relevant data - observations, actions, and rewards - remaining on the GPU throughout the learning process. This development allows for massive parallelization and as a result, dramatically reduces MTRL training time from days or weeks on thousands of CPU cores to just hours on a single GPU. Specifically, NVIDIA IsaacGym offers a Tensor API that directly exposes the physics state of the world