Learning Sim-to-Real Humanoid Locomotion in 15 Minutes
Younggyo Seo Carmelo Sferrazza Juyue Chen Guanya Shi Rocky Duan Pieter Abbeel [0.25cm]Amazon FAR (Frontier AI & Robotics) · 2025
Read Learning Sim-to-Real Humanoid Locomotion in 15 Minutes by Younggyo Seo Carmelo Sferrazza Juyue Chen Guanya Shi Rocky Duan Pieter Abbeel [0.25cm]Amazon FAR (Frontier AI & Robotics) (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-01996.txt.
Introduction
Teaching a humanoid robot to walk inside a simulation is fast now, thanks to massively parallel computing that cut training from days to minutes. Younggyo Seo and co-authors present a 2025 recipe that makes the transfer to a real robot both quick and reliable, despite the jump from virtual to physical.
The transfer is hard because humanoid control is high-dimensional, and randomizing the simulation to cover reality adds further strain. Their recipe builds on off-policy reinforcement learning algorithms, which reuse past experience efficiently. The title's promise is training in about a quarter of an hour. Fast, dependable transfer would let robot skills be developed at software speed.
Catalog record
| CREATOR | Younggyo Seo Carmelo Sferrazza Juyue Chen Guanya Shi Rocky Duan Pieter Abbeel [0.25cm]Amazon FAR (Frontier AI & Robotics) |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-01996
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning Sim-to-Real Humanoid Locomotion in 15 Minutes Younggyo Seo Carmelo Sferrazza Juyue Chen Guanya Shi Rocky Duan Pieter Abbeel [0.25cm]Amazon FAR (Frontier AI & Robotics) Abstract Massively parallel simulation has reduced reinforcement learning (RL) training time for robots from days to minutes. However, achieving fast and reliable sim-to-real RL for humanoid control remains difficult due to the challenges introduced by factors such as high dimensionality and domain randomization. In this work, we introduce a simple and practical recipe based on off-policy RL algorithms, i.e., FastSAC and FastTD3 , that enables rapid training of humanoid locomotion policies in just 15 minutes with a single RTX 4090 GPU. Our simple recipe stabilizes off-policy RL algorithms at massive scale with thousands of parallel environments through carefully tuned design choices and minimalist reward functions. We demonstrate rapid end-to-end learning of humanoid locomotion controllers on Unitree G1 and Booster T1 robots under strong domain randomization, e.g., randomized dynamics, rough terrain, and push perturbations, as well as fast training of whole-body human-motion tracking policies. We provide videos and open-source implementation at: https://younggyo.me/fastsac-humanoid . Figure 1: Summary of results. We introduce a simple recipe based on off-policy RL algorithms, i.e., FastSAC and FastTD3, that learns robust humanoid locomotion policies in 15 minutes on a single RTX 4090 GPU, with strong domain randomization including randomized dynamics, rough terrain, and push perturbations. We also show that our recipe based on off-policy RL algorithms is scalable and accelerates the training of whole-body tracking policies: trained with 4 \times L40s GPUs and 16384 parallel environments, FastSAC and FastTD3 learn to complete the full sequence of dancing motion much faster than PPO under the same condition. For sim-to-real deployment with Unitree G1 and Booster T1, we used the checkpoints saved at the points we marked as \bigstar . 1 Introduction In recent years, reinforcement learning (RL) has undergone a dramatic shift driven by the emergence of massively parallel simulation frameworks ( 38 ; 20 ) . By scaling environment throughput to thousands of environments, these frameworks have reduced wall-clock training time from many hours to mere minutes for a wide range of benchmark tasks ( 29 ; 31 ; 50 ) . This shift has had an outsized impact on robotics, where sim-to-real development is inherently iterative: a policy is trained in simulation, deployed on hardware, and reveals mismatches such as unmodeled dynamics or sensing inaccuracies ( 52 ) . These discrepancies must then be corrected by improving the simulation environment, requiring the entire pipeline to be retrained ( 7 ) . Because these cycles repeat until the policy is reliable, fast simulation becomes essential for making such iteration feasible. Despite the speed offered by modern parallel simulators, these iterative cycles remain expensive in practice, especially for high-dimensional systems such as humanoids. Achieving robust transfer of policies to the real world typically requires expanding domain randomization ( 39 ; 49 ; 35 ) , randomizing terrain properties ( 38 ) , or shaping curricula that encourage low-effort, stable whole-body behavior. Such components complicate exploration and reduce sample efficiency, pushing training for humanoid locomotion or tracking back into the multi-hour regime. Thus, despite dramatic gains in raw throughput, achieving fast, reliable sim-to-real iteration for humanoid control remains a challenge. This work introduces a simple and practical recipe that brings sim-to-real iteration time for humanoid robots back to the order of minutes. At the core of this recipe are FastSAC and FastTD3 ( 43 ) , efficient variants of popular off-policy RL algorithms, i.e., Soft Actor-Critic ( 14 ) and TD3 ( 13 ) , which have been shown to learn humanoid control policies faster than on-policy RL algorithms such as PPO ( 40 ) . While 43 demonstrated the first sim-to-real deployment of FastTD3 policies to real humanoid hardware, the results were limited to relatively simple controllers for humanoids with only a subset of joints. In this work, we show that with careful design choices and hyperparameters, FastSAC and FastTD3 can scale to full-body humanoid control, enabling rapid sim-to-real iterations for training locomotion policies with all joints or whole-body tracking policies that follow human motion. Another important aspect of our fast sim-to-real recipe is its simplicity in reward design. By adopting reward functions with essential terms, we can quickly sweep hyperparameters, isolate what matters for transfer, and avoid the brittle engineering often required in humanoid locomotion setups. With our recipe, we train a full-fledged humanoid locomotion policy with randomized dynamics, rough terrain, push perturbations, and an automatic action-rate curriculum, all end-to-end in 15 minutes on a single RTX 4090 GPU. The code for this recipe is available in the Holosoma repository ( 12 ) at https://github.com/amazon-far/holosoma . 2 Recipe 2.1 FastSAC and FastTD3: Off-Policy RL for Humanoid Control Our recipe is based on off-policy RL algorithms tuned for large-scale training with massively parallel simulation, i.e., FastTD3 and FastSAC ( 43 ) , instead of PPO ( 40 ) that has been a standard algorithm for sim-to-real RL due to the ease of scaling up with parallel simulation. This is motivated by recent work that have demonstrated off-policy algorithms can also scale effectively and be faster than PPO in various benchmark tasks ( 25 ; 37 ; 43 ; 45 ) by effectively re-using the data from simulation. Notably, 43 report the first sim-to-real deployment of humanoid control policies trained with off-policy RL to real humanoid hardware. However, its results were limited to humanoid robots with a subset of joints. This section describes how we train FastTD3 and FastSAC to achieve full-body humanoid control, enabling rapid sim-to-real iterations for training locomotion policies with all joints or whole-body tracking policies that follow human motion. In particular, we improve the recipe of 43 in training FastSAC, which learns how to explore environments from data with its maximum entropy learning scheme instead of deterministic policies with fixed noise schedules (FastTD3), mitigating the exploration challenges of training with strong domain randomization. Scaling up off-policy RL with massively parallel simulation Similar to prior work ( 25 ; 45 ; 37 ; 43 ) , we use massively parallel simulation for training FastSAC and FastTD3 agents. We find that the effect of using more environments is particularly visible in challenging whole-body tracking tasks (see Figure 2f ). We also find that most of observations in 43 with regard to scaling up off-policy RL also holds for full-body humanoid control as well. For instance, we find that using large batch size up to 8K consistently improves performance. We also find that taking more gradient steps per each simulation step usually ends up in a faster training, and slow simulation speed often becomes a bottleneck with more challenging setups such as training robots in non-flat terrains (see Figure 2b ). This makes off-policy RL, which can re-use data from previous interactions instead of discarding it, a more attractive choice for fast training. (a) Clipped double Q-learning (b) Number of update steps (c) Normalization (d) Effect of \gamma (e) Effect of \gamma (WBT) (f) Number of environments (WBT) Figure 2: FastSAC: Analyses. We investigate the effect of (a) Clipped double Q-learning, (b) number of update steps, (c) normalization techniques, and (d) discount factor \gamma on a Unitree G1 locomotion task with rough terrain.