MO -Playground 1 : Massively Parallelized M ulti- O bjective Reinforcement Learning for Robotics
Neil Janwani, Ellen Novoseller, Vernon J. Lawhern, Maegan Tucker · 2026
Read MO -Playground 1 : Massively Parallelized M ulti- O bjective Reinforcement Learning for Robotics by Neil Janwani, Ellen Novoseller, Vernon J. Lawhern, Maegan Tucker (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-09237.txt.
Introduction
Neil Janwani and colleagues present MO-Playground, pairing a fast GPU-native algorithm for multi-objective reinforcement learning with a kit of GPU-run environments. Older methods fail to use large-scale parallel simulation, so training on hard robotics problems stays slow. Their MORLAX algorithm approximates Pareto sets within minutes, with large speed-ups over CPU approaches and better front quality. A custom BRUCE humanoid setup shows strong locomotion trade-offs across six objectives such as smoothness and efficiency.
Catalog record
| CREATOR | Neil Janwani, Ellen Novoseller, Vernon J. Lawhern, Maegan Tucker |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36721 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.09237
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MO -Playground 1 : Massively Parallelized M ulti- O bjective Reinforcement Learning for Robotics Neil Janwani, Ellen Novoseller, Vernon J. Lawhern, Maegan Tucker Abstract Multi-objective reinforcement learning (MORL) is a powerful tool to learn Pareto-optimal policy families across conflicting objectives. However, unlike traditional RL algorithms, existing MORL algorithms do not effectively leverage large-scale parallelization to concurrently simulate thousands of environments, resulting in vastly increased computation time. Ultimately, this has limited MORL’s application towards complex multi-objective robotics problems. To address these challenges, we present 1) MORLAX , a new GPU-native, fast MORL algorithm, and 2) MO-Playground, a pip -installable playground of GPU-accelerated multi-objective environments. Together, MORLAX and MO-Playground approximate Pareto sets within minutes, offering 21-270x speed-ups compared to legacy CPU-based approaches whilst achieving superior Pareto front hypervolumes. We demonstrate the versatility of our approach by implementing a custom BRUCE humanoid robot environment using MO-Playground and learning Pareto-optimal locomotion policies across 6 realistic objectives for BRUCE, such as smoothness, efficiency and arm swinging. I Introduction Reinforcement learning (RL) has established itself as an effective paradigm for synthesizing complex robotic controllers by enabling agents to learn directly from interactions with their environment [ 12 , 22 , 17 ] . However, this methodology notoriously relies on crafting a single-objective, scalar reward function that trades off between conflicting objectives such as energy efficiency and tracking accuracy in a hard-coded manner. This reward shaping process is time-consuming, requires domain knowledge, and implies that the prioritization of conflicting objectives is known at training time, presenting a major barrier to real-world deployment [ 10 ] . Several paradigms have been proposed to address the problem of RL with unknown rewards, for instance, learning from human interaction, e.g. via demonstrations, pairwise preferences, or ratings [ 20 , 14 , 17 , 28 , 16 , 30 ] . Yet, these methods learn behavior reflecting a single reward function, and do not learn to intelligently trade off between conflicting objectives. This lack of on-demand flexibility is especially problematic for real-world robotic systems, which inherently have competing objectives with optimal weightings that vary across different scenarios. For instance, human-robot systems such as lower-limb exoskeletons or assistive driving require this tunability for user-specific personalization [ 28 , 3 , 13 ] . 1 1 footnotetext: MO-Playground: https://github.com/dynamicmobility/moplayground Multi-Objective Reinforcement Learning (MORL) provides a principled framework to address this gap and enable the exploration of trade-offs between objectives. In contrast to the single-objective RL setting, which uses a single, scalarized reward, the MORL paradigm measures performance with a vectorized reward in which each component represents a different objective. With this vectorized reward, MORL enables practitioners to flexibly tune robot behavior post-training by finding and exploring the Pareto set , a collection of high quality solutions that are optimal under different objective prioritizations, capturing critical trade-offs among objectives. To this end, a wide variety of MORL algorithms have been developed to approximate or explore the Pareto set efficiently [ 10 , 6 ] , including evolutionary approaches [ 31 ] , context-conditioned policy exploration [ 1 , 2 ] and hypernetwork-based approaches [ 24 , 18 ] . While these methods have all contributed significant methodological advances, modern MORL algorithms still require long training times, sometimes on the order of days [ 1 ] . Meanwhile, single-objective RL has benefited from hardware-accelerated differentiable physics and vectorized computation frameworks such as JAX [ 4 ] and MuJoCo JaX (MJX) [ 32 ] . These tools enable large-scale on-device parallelization and just-in-time compilation through the simulation loop. Most notably, recent toolboxes such as MuJoCo Playground [ 32 ] are capable of running thousands of MuJoCo-native simulations simultaneously on consumer GPUs, vastly decreasing training times and bolstering a community shift away from CPU-based, small-scale parallelism. MORL algorithms have yet to leverage such computational frameworks effectively. Most current implementations rely on CPU-based simulation or limited concurrency, resulting in training times that are an order of magnitude slower than state-of-the-art single-objective RL. This critical computational gap has fundamentally hindered MORL’s application to complex, high-dimensional robot morphologies and settings involving more than a few objectives. In this paper, we remove this barrier by introducing a GPU-native MORL algorithm that takes advantage of hypernetworks [ 24 ] . Our highlighted contributions are as follows: 1. MORLAX : a JAX-compatible framework for scalable MORL that integrates multi-objective optimization with JAX’s vectorized computation to achieve 21-270x speedups and improved (larger) hypervolumes compared to existing MORL baselines. 2. MO-Playground: an open-source toolbox I featuring MORLAX and a modernized set of multi-objective MJX environments for systematic benchmarking. 3. A custom application of the BRUCE humanoid robot across six realistic objectives, demonstrating the versatility and performance of MO-Playground. II Background Our work sits at the intersection of MORL and modern hardware-accelerated computation. Before presenting our approach, we review the foundational Multi-Objective Markov Decision Process (MOMDP) framework underlying all MORL algorithms, outline the details of existing algorithms and toolboxes for MORL, and then introduce the parallelizable computational paradigm that we leverage for our novel, open-source MORL framework, MO-Playground. II-A Multi-Objective Markov Decision Processes We adopt the theoretical foundation for multi-objective sequential decision-making given by the MOMDP [ 10 ] . As with a regular Markov Decision Process (MDP), a MOMDP is defined via a tuple (\mathcal{S},\mathcal{A},\mathcal{P},\bm{R},\gamma) , where \mathcal{S} is a set of states, \mathcal{A} is a set of actions, \mathcal{P}(s^{\prime}\mid s,a) is the transition probability function of state s\in\mathcal{S} and action a\in\mathcal{A} to state s^{\prime}\in\mathcal{S} , and \gamma\in[0,1) is a discount factor. The key distinction from ordinary MDPs lies in the vector-valued reward \bm{R}:\mathcal{S}\times\mathcal{A}\to\mathbb{R}^{m} , with m denoting the number of objectives. Consequently, the value function V_{\pi} also becomes a vector \bm{V}_{\pi}\in\mathbb{R}^{m} , representing the expected cumulative return for each objective: \displaystyle\bm{V}_{\pi}(s)=\mathbb{E}_{\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}\bm{R}(s_{t},a_{t})\mid a_{t}\sim\pi(s_{t})\right]\in\mathbb{R}^{m}. (1) We seek to maximize the expected returns over the initial state distribution \mathcal{D}_{s_{0}} : \displaystyle\bm{J}_{\pi}=\mathbb{E}_{s_{0}\sim\mathcal{D}_{s_{0}}}\left[\bm{V}_{\pi}(s_{0}))\right]\in\mathbb{R}^{m}, (2) with \bm{J}_{\pi} being shorthand for \bm{J}(\pi)\in\mathbb{R}^{m} . In most scenarios, no single policy can simultaneously maximize all components of \bm{J}_{\pi} , as trade-offs can exist between the objectives. Thus, the goal is not to find a single optimal policy, but rather to find the set of Pareto-optimal policies, which optimally trade off among the objectives. Pareto optimality is mathematically quantified using the notion of Pareto dominance : a policy \pi^{\prime} dominates a policy \pi if \bm{J}_{\pi^{\prime}}\geq\bm{J}_{\pi} , where the comparison is taken elementwise, and \bm{J}_{\pi^{\prime}}\neq\bm{J}_{\pi} for at least one component. A policy \pi^{*} is then defined as Pareto optimal if it is not dominated by any other policy \pi^{\prime} . The set of all such optimal policies forms the Pareto set , and their corresponding objectives form the Pareto front in the objective space. While early research focused on analyzing the existence of stationary Pareto-optimal policies and solving small-scale MOMDPs via value iteration or linear programming, the challenge for complex robotic systems lies in efficiently approximating the Pareto set with high granularity. Towards this end, several MORL algorithms have been proposed, which we will discuss next. II-B Multi-Objective Reinforcement Learning for Robotics Modern MORL approaches focus on generating high-quality approximations of the Pareto set in high-dimensional state and action spaces. One of the first such works, Prediction-Guided MORL (PG-MORL) [ 31 ] , developed an evolutionary algorithm that utilizes a prediction model to choose promising