Learning Whole-Body Humanoid Locomotion via Motion Generation and Motion Tracking

Zewei Zhang, Kehan Wen, Michael Xu, Junzhe He, Chenhao Li, Takahiro Miki, Clemens Schwarke, Chong Zhang, Xue Bin Peng, Marco Hutter · 2026

Read Learning Whole-Body Humanoid Locomotion via Motion Generation and Motion Tracking by Zewei Zhang, Kehan Wen, Michael Xu, Junzhe He, Chenhao Li, Takahiro Miki, Clemens Schwarke, Chong Zhang, Xue Bin Peng, Marco Hutter (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-17335.txt.

Introduction

Zhang and nine colleagues teach a humanoid robot to walk whole-body style over rough ground. A diffusion model predicts terrain-aware reference motion, and a separate tracker learns to follow it. A final closed-loop fine-tuning round hardens the tracker against imperfect predictions. The system runs on a Unitree G1 robot using only onboard sensing and computing, and hardware trials crossed boxes, hurdles, stairs, and mixed ground.

Catalog record

CREATORZewei Zhang, Kehan Wen, Michael Xu, Junzhe He, Chenhao Li, Takahiro Miki, Clemens Schwarke, Chong Zhang, Xue Bin Peng, Marco Hutter
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37307
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2604.17335

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learning Whole-Body Humanoid Locomotion via Motion Generation and Motion Tracking Zewei Zhang, Kehan Wen, Michael Xu, Junzhe He, Chenhao Li, Takahiro Miki, Clemens Schwarke, Chong Zhang, Xue Bin Peng, Marco Hutter Abstract Whole-body humanoid locomotion is challenging due to high-dimensional control, morphological instability, and the need for real-time adaptation to various terrains using onboard perception. Directly applying reinforcement learning (RL) with reward shaping to humanoid locomotion often leads to lower-body-dominated behaviors, whereas imitation-based RL can learn more coordinated whole-body skills but is typically limited to replaying reference motions without a mechanism to adapt them online from perception for terrain-aware locomotion. To address this gap, we propose a whole-body humanoid locomotion framework that combines skills learned from reference motions with terrain-aware adaptation. We first train a diffusion model on retargeted human motions for real-time prediction of terrain-aware reference motions. Concurrently, we train a whole-body reference tracker with RL using this motion data. To improve robustness under imperfectly generated references, we further fine-tune the tracker with a frozen motion generator in a closed-loop setting. The resulting system supports directional goal-reaching control with terrain-aware whole-body adaptation, and can be deployed on a Unitree G1 humanoid robot with onboard perception and computation. The hardware experiments demonstrate successful traversal over boxes, hurdles, stairs, and mixed terrain combinations. Quantitative results further show the benefits of incorporating online motion generation and fine-tuning the motion tracker for improved generalization and robustness. I Introduction Developing whole-body perceptive humanoid locomotion remains a challenging yet important step toward expanding the operational range of humanoid robots. While deep reinforcement learning (RL) has achieved remarkable success in enabling highly dynamic behaviors on quadrupedal systems [ 10 , 15 , 5 , 20 , 35 , 34 ] , extending these capabilities to humanoid robots remains considerably difficult, as humanoids possess higher degrees of freedom together with more complex morphological constraints. As a result, training an RL-based whole-body humanoid controller solely through reward shaping can be non-trivial and often suffers from inefficient exploration. Without explicit structural guidance, vanilla RL-based humanoid controllers frequently converge to locomotion strategies with limited whole-body coordination, making it difficult to acquire coordinated behaviors for challenging terrain traversal. This limitation becomes especially evident in tasks such as humanoid parkour, where coordinated whole-body movements are essential for traversing large and complex obstacles. To alleviate the difficulty of reward shaping and inefficient exploration, motion imitation via RL has emerged as a promising alternative [ 18 , 2 , 12 , 9 , 32 , 31 ] . By leveraging reference data, motion tracking can efficiently transfer highly coordinated whole-body skills to the robot. However, pure motion trackers are inherently limited to replaying choreographed trajectories and generally lack the adaptability and reactivity required for operation in unstructured and diverse environments. Achieving human-level perceptive locomotion therefore requires a higher-level composition mechanism that can dynamically adjust locomotion style across varied obstacles based on real-time perceptive inputs. One direct solution for skill composition is to distill multiple specialized teacher policies [ 28 ] , or to compose multiple skills from separate reference trajectories within a single imitation-based teacher framework [ 29 , 27 ] . By learning distinct expert skills for different terrain conditions, a unified distilled policy can learn to transition among them using exteroceptive observations. However, current methods have not yet demonstrated strong scalability to a diverse set of skills or robust generalization to new terrains. Fig. 1: Whole-body humanoid locomotion on the Unitree G1 robot enabled by diffusion-based motion generation and RL-based motion tracking. Our hardware experiments show behaviors including box climbing, box descent, vaulting, and traversal over mixed terrain sequences. https://wholebodylocomotion.github.io/ . Alternatively, generative models have recently shown strong potential for scalable motion control. Training generative models on large-scale motion datasets has proven effective in both character control [ 8 , 26 , 25 ] and robot control [ 4 , 12 ] , naturally reducing the engineering effort required as the motion repertoire expands. However, directly deploying expressive generated kinematic trajectories on real humanoid locomotion systems may introduce motion artifacts, such as foot sliding or temporal discontinuities, which can lead to catastrophic failure. To mitigate such issues, recent work [ 25 , 22 , 30 ] combines diffusion-based motion generation with RL-based motion tracking, allowing the tracker to output physically plausible actions instead of relying solely on kinematic generation. Despite this progress, their applicability to real humanoid robots in locomotion tasks requiring whole-body coordination remains insufficiently explored, and their deployment may potentially be hindered by slow inference and limited robustness to real-world disturbances. To address these limitations, we propose a whole-body humanoid locomotion system that adapts a motion generation and motion tracking framework to real-time robot control for challenging humanoid locomotion tasks. In particular, we repurpose coordinated human whole-body skills learned from motion data for perceptive humanoid locomotion, and build a control framework that combines diffusion-based motion generation with RL-based motion tracking. The resulting system supports directional goal-reaching control together with terrain-aware whole-body adaptation. Our hardware experiments show that our approach enables efficient skill execution and generalization across diverse terrains, including terrains unseen during training. Our results further demonstrate the effectiveness of using a motion generator trained on human motion data to guide an RL-based controller through challenging terrains requiring diverse skills, without heavily engineered distillation strategies. II Related work II-A Whole-body Motion Tracking for Humanoids In simulated whole-body character control, prior works [ 18 , 3 ] have explored RL-based frameworks in which characters learn to imitate reference motion clips. By training a control policy to minimize the discrepancy between the reference motion and the executed motion, the controller is able to reproduce a wide range of dynamic maneuvers while remaining consistent with the physical constraints of the controlled system. More recently, this paradigm has been widely adopted in quadruped and humanoid robot control [ 31 , 12 , 13 , 11 ] . By designing imitation rewards and regularization terms, and optimizing the policy with the PPO algorithm [ 21 ] , recent works have demonstrated strong capability in imitating large-scale motion datasets and even enabling real-time teleoperation [ 32 , 33 , 6 ] . However, for general locomotion tasks, pure motion tracking remains fundamentally limited in its adaptability to varying terrains. II-B Motion Generation for Motion Control Motion generation with generative models provides an effective way to sample feasible motion trajectories from curated motion datasets. Prior work [ 26 ] has demonstrated that diffusion models can generate smooth kinematic motion trajectories conditioned on text descriptions by incorporating additional geometric losses. Building on this line of work, Tevet et al. [ 25 ] further extend it to physical character control by coupling the generated motion with an RL-based motion tracker, which helps correct artifacts introduced by the generative model. Serifi et al. [ 22 ] also combine motion generation with RL-based motion tracking, and further fine-tune the generator using value information from the critic network of the tracking policy. Xu et al. [ 30 ] adopt a similar design philosophy to [ 25 ] and extend it to humanoid parkour in simulation, demonstrating impressive whole-body motion control conditioned on surrounding terrain observations. However, these methods mainly target simulated character control, and may still be constrained by the inference latency

Read the complete text