SLAC: Simulation-Pretrained Latent Action Space for Whole-Body Real-World RL

Jiaheng Hu, Peter Stone, Roberto Martín-Martín · 2025

Read SLAC: Simulation-Pretrained Latent Action Space for Whole-Body Real-World RL by Jiaheng Hu, Peter Stone, Roberto Martín-Martín (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-04147.txt.

Introduction

Robots with arms on moving bases have many joints to control, and learning that control directly in the real world is slow and risky. Jiaheng Hu and co-authors present SLAC in this 2025 paper, which first practices in a cheap simulator to learn a compact set of reusable moves.

That move set then becomes the control interface for learning real tasks through trial and error. The practice phase is tuned to favor moves that are safe, distinct, and stretched over time. On two-armed mobile manipulation tasks, the method beat rivals and learned contact-heavy whole-body skills in under an hour of real practice, with no demonstrations.

Catalog record

CREATORJiaheng Hu, Peter Stone, Roberto Martín-Martín
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-04147

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SLAC: Simulation-Pretrained Latent Action Space for Whole-Body Real-World RL Jiaheng Hu, Peter Stone, Roberto Martín-Martín Abstract Building capable household and industrial robots requires mastering the control of versatile, high-degree-of-freedom (DoF) systems such as mobile manipulators. While reinforcement learning (RL) holds promise for autonomously acquiring robot control policies, scaling it to high-DoF embodiments remains challenging. Direct RL in the real world demands both safe exploration and high sample efficiency, which are difficult to achieve in practice. Sim-to-real RL, on the other hand, is often brittle due to the reality gap. This paper introduces SLAC, a method that renders real-world RL feasible for complex embodiments by leveraging a low-fidelity simulator to pretrain a task-agnostic latent action space . SLAC trains this latent action space via a customized unsupervised skill discovery method designed to promote temporal abstraction, disentanglement, and safety, thereby facilitating efficient downstream learning. Once a latent action space is learned, SLAC uses it as the action interface for a novel off-policy RL algorithm to autonomously learn downstream tasks through real-world interactions. We evaluate SLAC against existing methods on a suite of bimanual mobile manipulation tasks, where it achieves state-of-the-art performance. Notably, SLAC learns contact-rich whole-body tasks in under an hour of real-world interactions, without relying on any demonstrations or hand-crafted behavior priors. More information, code, and videos at robo-rl.github.io Keywords: Real-world RL, Latent Action, Whole-body Manipulation 1 Introduction Future robots are expected to perform diverse tasks in unstructured environments. Achieving this requires controlling high-degree-of-freedom, multi-purpose systems like mobile manipulators and humanoids. These robots offer unique opportunities to handle complex and ambitious tasks: their many degrees of freedom allow them to pursue multiple control objectives simultaneously, including sophisticated contact-rich interactions (e.g., wiping a board while avoiding obstacles), and their combined locomotion and manipulation capabilities provide large workspaces for long-horizon operation. However, these opportunities come with significant challenges. The combination of long-horizon, multi-objective tasks and high-dimensional action spaces makes policy optimization particularly difficult. Moreover, performing contact-rich interactions with the whole body increases the risk of damaging the robot and its surroundings, making safety a critical concern. As a result, prior policy learning methods that have shown success in simpler settings often struggle to scale to these high-DoF, multi-purpose robots. On the one hand, successes in Imitation Learning (IL) with simple fixed-base robot arm [ 1 , 2 , 3 ] are hard to replicate for high DoF robot systems due to the difficulty both in creating a teleoperation interface and in collecting a sufficient number of high-quality demonstrations. On the other hand, while Reinforcement Learning (RL) algorithms can learn without expert demonstrations, they require many environment steps before convergence, and has primarily been conducted in simulation [ 4 , 5 , 6 , 7 , 8 , 9 , 10 ] , with policies transferred zero-shot to the real world. However, these approaches often face significant challenges in bridging the reality gap [ 11 , 12 , 13 , 14 , 15 ] , which often widen as the complexity of robots and tasks increases, despite costly domain randomization or the tedious construction of high-fidelity digital twins [ 16 , 17 , 4 ] . Figure 1: SLAC uses a task-agnostic action space trained in low-fidelity simulation ( left ) to learn downstream tasks in the real world. This latent action space is safe, temporally extended, and disentangled, enabling a bimanual mobile manipulator to solve challenging contact-rich whole-body tasks ( right ) with less than an hour of autonomous real-world interactions. Real-world RL offers a promising alternative: by enabling robots to learn directly through trial-and-error interactions with the physical world, we may bypass both the reality gap and the need for costly human demonstrations. Unfortunately, existing real-world RL approaches [ 18 , 19 , 20 , 21 ] remain largely limited to simple domains such as tabletop manipulation and quadruped locomotion. They do not scale effectively to more complex tasks and embodiments, due to fundamental challenges in ensuring safe exploration given larger workspaces and frequent physical contact with the environment, and in achieving sample-efficient learning in the presence of high-dimensional action spaces, long-horizon tasks, and intricate reward structures. These factors severely limit the applicability of current real-world RL techniques to complex robotics problems. This paper introduces SLAC : S imulation-Pretrained L atent AC tion Space for Real-World RL, which utilizes low-fidelity simulator to make real-world downstream RL feasible for high DoF robots such as mobile manipulators. The SLAC framework introduces a two-step procedure to circumvent the main challenges of real-world RL, namely unsafe exploration and sample inefficiency. In the first step, SLAC learns a task-agnostic latent action space in a coarsely aligned, low-fidelity simulator via Unsupervised Skill Discovery (USD) [ 22 , 23 , 24 , 25 ] . SLAC utilizes a novel USD objective that shapes this latent action space to be (1) temporally extended , enabling more effective exploration than when directly using the low-level action space by reducing decision frequency; (2) disentangled , allowing each latent action dimension to independently affect the states, thereby facilitating joint optimization of multiple objectives without conflict; and (3) safe , avoiding dangerous behaviors that could damage the robot. In the second step, the learned SLAC latent action space is used by a novel off-policy RL algorithm to efficiently learn downstream tasks directly in the real world. Critically, this design offers robustness to the reality gap: even if latent actions exhibit slight behavioral mismatches between simulation and the real world, the downstream policy can still learn to solve the task by directly selecting effective latent actions based on real-world reward signals. We evaluate our method on a complex, high-DoF, bimanual mobile manipulator, where SLAC can learn contact-rich whole-body tasks in less than an hour of real-world interactions, using only onboard sensor signals. To the best of our knowledge, SLAC is the first algorithm that enables a high-DoF mobile manipulator to learn with RL in the real world without relying on any demonstrations/mocap data [ 26 , 27 ] or hand-crafted behavior priors [ 26 , 28 , 29 ] . 2 Related Work Robots can be trained to perform tasks using four main approaches: (1) sim-to-real reinforcement learning, (2) real-world reinforcement learning, (3) classical motion planning and control, and (4) learning from demonstrations. In this section, we review the first two approaches, which are most relevant to SLAC. We defer discussion of the latter two [ 2 , 3 , 1 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 ] to the appendix. Sim-to-Real Reinforcement Learning: While Reinforcement Learning (RL) provides a way for agents to learn sophisticated behaviors from trial and error, popular algorithms like PPO [ 50 ] are quite sample-inefficient and can require billions of samples before they converge. Many works have therefore resorted to performing the RL training completely in simulation [ 4 , 14 , 5 , 6 , 7 , 8 , 13 , 9 , 10 , 11 ] , and zero-shot transfer the learned policy into the real-world. Such a procedure requires the simulation to

Read the complete text