Direct Dynamic Retargeting for Humanoid Imitation Learning from Videos

Constant Roux, Ludovic De Matteïs, Armand Jordana, Valentin Guillet, Nicolas Mansard, Olivier Stasse, Philippe Souères · 2026

Read Direct Dynamic Retargeting for Humanoid Imitation Learning from Videos by Constant Roux, Ludovic De Matteïs, Armand Jordana, Valentin Guillet, Nicolas Mansard, Olivier Stasse, Philippe Souères (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-23762.txt.

Introduction

Roux and colleagues propose Direct Dynamic Retargeting, a plan making paths from videos for humanoid imitation learning. Turning human motion into humanoid motion must beat shape gaps, and retargeting pipes add shape bias, narrowing search and yielding weak moves. Direct Dynamic Retargeting frames the task in task space and uses a Model Predictive Control solver in a simulator, optimizing contact runs while easing input drift. Skipping shape bias lets it beat baselines in demo tracking accuracy. Giving workable guides to reward agents speeds training and lifts agile moves.

Catalog record

CREATORConstant Roux, Ludovic De Matteïs, Armand Jordana, Valentin Guillet, Nicolas Mansard, Olivier Stasse, Philippe Souères
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36292
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.23762

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Direct Dynamic Retargeting for Humanoid Imitation Learning from Videos Constant Roux, Ludovic De Matteïs, Armand Jordana, Valentin Guillet, Nicolas Mansard, Olivier Stasse, Philippe Souères Abstract Imitation Learning from monocular video demonstrations provides a scalable approach for teaching complex skills to humanoid robots. However, translating human motion to humanoids requires overcoming significant morphological mismatches. Standard approaches rely on Geometric Retargeting or Indirect Dynamic Retargeting pipelines. We identify that these intermediate kinematic projections introduce a geometric bias, restricting the search space and yielding suboptimal dynamic behaviors. In this paper, we propose Direct Dynamic Retargeting (DDR), a novel single-stage framework that generates high-fidelity, dynamically feasible trajectories directly from expert videos. By formulating the problem in the task space and leveraging a sampling-based Model Predictive Control solver within a physics simulator, DDR natively optimizes over complex contact sequences while mitigating input drift. Our experiments demonstrate that bypassing the geometric bias allows DDR to outperform state-of-the-art baselines in demonstration tracking accuracy. Furthermore, we establish that providing such physically viable references to RL agents accelerates training convergence and enhances the final execution of agile and balancing behaviors. Source code will be made publicly available. I Introduction Humanoid robotics has witnessed major breakthroughs in recent years, demonstrating unprecedented capabilities in locomotion and loco-manipulation [ 2 ] . This progress has been largely catalyzed by the rise of Reinforcement Learning (RL) [ 3 ] . However, for complex tasks, manual reward engineering becomes a major bottleneck, proving to be both tedious and time-consuming. To bypass this difficulty, Imitation Learning (IL) has emerged as a highly effective alternative. Techniques such as Behavioral Cloning, DeepMimic [ 4 ] , and Adversarial Motion Priors [ 5 ] enable robots to learn complex skills directly from expert demonstrations. Nevertheless, the efficiency of the resulting policies is fundamentally dependent on the quality of the provided references [ 6 , 2 ] . Current approaches rely either on precise Motion Capture systems (MoCap) or on a posteriori selection of robot trajectories to avoid detrimental demonstrations [ 7 , 8 ] . Recent works extracting human poses from monocular videos enable the use of vast online datasets [ 1 ] . However, the current extraction tools often yield noisy and physically inconsistent trajectories. Furthermore, whether sourced from videos or MoCap, human demonstrations present a morphological mismatch with humanoid robots, leading to both kinematic (size, limb proportions) and dynamic (mass distribution, inertias, actuation limits) discrepancies [ 9 ] . Consequently, retargeting human motion to the robot is necessary to provide IL with relevant references. The current prevailing approach, Geometric Retargeting (GR), aims to find the closest robot posture using Inverse Kinematics (IK) [ 10 , 9 , 2 ] , inherently ignoring dynamic constraints. To address this limitation, Indirect Dynamic Retargeting (IDR) methods tracks the GR reference within a physics simulator [ 11 , 12 , 13 ] . We hypothesize that this intermediate IK step restricts the search space and biases the final dynamically feasible solution. To overcome this, we propose Direct Dynamic Retargeting (DDR), a Model Predictive Control (MPC) framework that directly computes dynamically feasible trajectories from noisy data extracted from videos (See Fig. ). To solve the optimization, we use the Cross-Entropy Method (CEM) [ 14 ] . Unlike gradient-based solvers (e.g., Crocoddyl [ 15 ] ) that require strictly predefined contact sequences, CEM’s sampling-based approach intrinsically handles the ambiguities related to the identification of contacts from noisy video data. This allows us to generate high-quality IL retargeted references, completely bypassing the intermediate geometric bias. We show that DDR outperforms previous methods in terms of physical consistency, demonstration tracking accuracy, and learning efficiency. The main contributions are as follows: • Introduction of Direct Dynamic Retargeting , a novel CEM-based MPC framework that generates dynamically feasible humanoid trajectories directly from demonstrations. • Quantitative evaluation showing that eliminating the intermediate geometric bias significantly improves retargeting accuracy, outperforming GR and IDR baselines on diverse agile motions. • Experimental evidence establishing that utilizing these physically consistent references enhances the learning efficiency and performance of downstream Imitation Learning. • Successful real-world deployment on the Unitree H1-2 humanoid, achieving zero-shot sim-to-real transfer and validating the viability of the framework. The remainder of this paper is organized as follows. Section II reviews the relevant literature and positions our approach within the current state-of-the-art. Section III details the DDR formulation and describes how our CEM-based MPC framework effectively tracks expert demonstrations. Section IV presents our experimental setup and evaluates the performance of DDR against existing baselines, highlighting improvements in both retargeting quality and downstream learning efficiency. Finally, Section V summarizes our findings and outlines directions for future work. II Related Work This section provides a review of the literature on motion retargeting for humanoid robots, highlighting the transition from geometric to dynamic approaches. Then, an overview of imitation learning methods is given to contextualize the need for dynamically retargeted reference motions. II-A Geometric Retargeting Transferring human motion to humanoid robots requires solving the problem of the morphological mismatch between the two embodiments [ 16 ] . Geometric Retargeting typically relies on Inverse Kinematics to map human keypoints or joint angles to the robot’s configuration space [ 17 , 18 ] . While these methods successfully minimize spatial tracking errors and enforce strict joint limits, they inherently ignore the system’s dynamics. Consequently, for highly dynamic systems like humanoid robots, purely geometric references frequently result in physically infeasible trajectories that violate balance constraints, torque limits and contact dynamics [ 19 ] . II-B Indirect Dynamic Retargeting To bridge the gap between kinematic similarity and physical validity, IDR approaches introduce a secondary physics-based refinement phase. Methods in this category formulate the problem as a two-stage pipeline: an initial GR step generates a kinematic reference, which is subsequently tracked by a controller or a RL agent within a physics simulator to yield a dynamically feasible trajectory [ 11 , 13 , 20 , 12 ] . While these methods ensure dynamic feasibility, they inherently rely on the assumption that the intermediate kinematic reference resides near the optimal dynamic motion. A fundamental limitation of these two-stage pipelines is that the initial kinematic projection intrinsically restricts the optimization space, potentially biasing the final motion away from the true dynamic optimum. To the best of our knowledge, no previous work has successfully bypassed this intermediate step to compute direct, dynamically feasible retargeting for humanoid imitation. II-C Imitation Learning from Video Demonstrations Imitation Learning circumvents the need for complex reward shaping by learning humanoid skills directly from expert data [ 8 , 21 ] . Frameworks ranging from direct reference state tracking [ 4 ] to adversarial methods [ 22 , 5 ] have achieved high-fidelity motion reproduction by mimicking expert data. To scale up these approaches, recent works increasingly focus on extracting human demonstrations directly from large-scale monocular video datasets [ 1 ] . Standard imitation pipelines often attempt to train RL agents directly on purely geometric references. However, the learning efficiency and performance of these policies remain sensitive to the quality and dynamical feasiblity of the provided references [ 6 , 2 ] . Kinematic noise or dynamically infeasible reference trajectory lead to suboptimal behaviors, undesirable artifacts, or learning failures. Since generating these optimized references is an

Read the complete text