Dynamic Policy Learning for Legged Robot with Simplified Model Pretraining and Model Homotopy Transfer
Dongyun Kang, Min-Gyu Kim, Tae-Gyu Song, Hajun Kim, Sehoon Ha, Hae-Won Park · 2025
Read Dynamic Policy Learning for Legged Robot with Simplified Model Pretraining and Model Homotopy Transfer by Dongyun Kang, Min-Gyu Kim, Tae-Gyu Song, Hajun Kim, Sehoon Ha, Hae-Won Park (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-24698.txt.
Introduction
Dongyun Kang, Min-Gyu Kim, Tae-Gyu Song, Hajun Kim, and colleagues tackle dynamic motion for legged robots, where reinforcement learning often needs heavy reward tuning or good demonstrations. Their plan starts simple and grows harder. First a policy learns core motion patterns on a single rigid body model, a reduced picture of the robot.
Then a continuation strategy, inspired by model homotopy, gradually transfers that policy to the full-body robot. Step by step the gap between the simple and real models closes. Readers get a way to produce complex, lively gaits while reducing the tuning burden that usually blocks such transfer.
Catalog record
| CREATOR | Dongyun Kang, Min-Gyu Kim, Tae-Gyu Song, Hajun Kim, Sehoon Ha, Hae-Won Park |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-24698
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Dynamic Policy Learning for Legged Robot with Simplified Model Pretraining and Model Homotopy Transfer Dongyun Kang, Min-Gyu Kim, Tae-Gyu Song, Hajun Kim, Sehoon Ha, Hae-Won Park Abstract Generating dynamic motions for legged robots remains a challenging problem. While reinforcement learning has achieved notable success in various legged locomotion tasks, producing highly dynamic behaviors often requires extensive reward tuning or high-quality demonstrations. Leveraging reduced-order models can help mitigate these challenges. However, the model discrepancy poses a significant challenge when transferring policies to full-body dynamics environments. In this work, we introduce a continuation-based learning framework that combines simplified model pretraining and model homotopy transfer to efficiently generate and refine complex dynamic behaviors. First, we pretrain the policy using a single rigid body model to capture core motion patterns in a simplified environment. Next, we employ a continuation strategy to progressively transfer the policy to the full-body environment, minimizing performance loss. To define the continuation path, we introduce a model homotopy from the single rigid body model to the full-body model by gradually redistributing mass and inertia between the trunk and legs. The proposed method not only achieves faster convergence but also demonstrates superior stability during the transfer process compared to baseline methods. Our framework is validated on a range of dynamic tasks, including flips and wall-assisted maneuvers, and is successfully deployed on a real quadrupedal robot. I Introduction Learning complex and dynamic motions for legged robots remains a challenging problem. Successfully generating such behaviors requires navigating the intricate interplay between the robot’s body momentum, leg inertia, and contact forces, making it difficult to develop effective control strategies. Classical motion generation approaches, particularly trajectory optimization (TO), formulate the problem as a constrained optimization [ 32 , 23 , 5 , 15 , 2 , 8 , 35 , 12 , 18 , 14 , 7 , 39 , 33 ] . While TO offers interpretability and principled motion synthesis, solving it reliably is often hindered by nonconvexity and the hybrid nature of contact dynamics. To cope with these challenges, reduced-order models are employed to approximate feasible motions [ 2 , 8 , 35 , 12 , 18 , 14 , 7 , 39 , 33 ] , yet relying on them introduces dynamics mismatches. While hierarchical frameworks [ 14 , 39 , 33 , 24 ] mitigate this by refining trajectories with full-body dynamics, this process is often computationally intensive, making it challenging to achieve real-time reactivity without significant optimization expertise. Reinforcement learning (RL), which has demonstrated impressive results across various legged locomotion tasks [ 21 , 11 , 17 ] , offers an alternative paradigm capable of discovering highly dynamic behaviors. However, generating complex motions without prior information often results in instability and convergence to suboptimal solutions. To address these issues, extensive reward engineering is required, making the process time consuming and difficult to apply to a wide range of dynamic behaviors. While imitation learning [ 4 , 27 , 26 , 28 ] , which utilizes expert demonstrations, is known to be an effective solution for guiding RL, obtaining high-quality demonstrations often requires substantial resources and is sometimes not readily available [ 22 , 10 , 3 , 9 ] . One strategy to alleviate these challenges is to leverage a reduced-order model to train the policy in a simpler environment [ 36 , 30 , 6 , 20 ] . This approach offers several advantages, such as simple reward design and faster simulation speeds, leading to more efficient policy learning [ 36 , 30 ] . However, the discrepancy between simplified models and full-body models, commonly referred to as the model gap, poses a significant hurdle when deploying these policies in real environments. Previous works have addressed this gap by combining simplified model planners with whole-body tracking controllers [ 36 , 6 ] or by employing imitation learning [ 30 ] . Although these methods have shown promise in simple locomotion tasks or 2D scenarios, their applicability to highly dynamic maneuvers remains unproven. Moreover, adhering to the simplified model’s motion often leads to suboptimal solutions for the full-body system. These observations highlight the need for an effective and stable refinement process to bridge the model gap. Fig. 1 : Conceptual illustration of the proposed model homotopy transfer. The framework defines a continuation path from the simplified SRB to full-body dynamics via gradual mass and inertia redistribution. This approach facilitates the stable transfer of core motion patterns to complex, highly dynamic behaviors such as wall-assisted maneuvers. Continuation methods [ 1 ] offer a powerful conceptual framework to tackle such challenges. By gradually transforming a well-understood, simpler system into a more complex target system, continuation methods facilitate smoother transitions and more robust convergence on difficult problems. Building on this conceptual foundation, we propose a continuation-based method that first leverages a simplified model to pretrain the policy in an idealized setting, and then incrementally bridges the gap to the full-body environment. To define the continuation path, we introduce a model homotopy from the Single Rigid Body (SRB) model to the full-body model, constructed by progressively interpolating the mass and inertia between the trunk and legs. Following this controlled progression allows the policy to navigate increasingly complex dynamics and achieve a smooth, reliable transfer. We validate our framework on a series of challenging dynamic tasks, including flips and wall-assisted maneuvers. Our results show that the proposed approach enhances learning stability and achieves faster convergence compared to baseline methods. Finally, we successfully transfer the learned policies to a real quadrupedal robot. In summary, the key contributions of this work are as follows: • We present a continuation-based learning framework that integrates simplified model pretraining and model homotopy transfer to efficiently generate and refine complex dynamic motions. • This framework provides a principled way to smoothly transition from simplified to full-body dynamics, improving both convergence speed and learning stability. • We validate the effectiveness of our method by successfully generating a variety of dynamic behaviors in simulation, and deploy the learned policies on a real robot. II Related Work II-A Trajectory Optimization TO is a systematic process for finding the optimal trajectory that satisfies a set of physical and task constraints. Due to its interpretability and ability to handle diverse constraints, TO has been extensively utilized in legged robot motion generation [ 32 , 23 , 5 , 15 ] . However, the optimization landscape for legged locomotion is fundamentally nonconvex. Furthermore, the hybrid nature of the system, characterized by discontinuous contact modes, makes finding desired solutions extremely challenging, as solvers are prone to getting trapped in poor local optima. To overcome these computational complexities, a common strategy is to decompose the high-dimensional full-body optimization problem into simpler, manageable subproblems. This is typically achieved by pre-specifying contact sequences [ 8 , 12 , 39 ] or employing reduced-order models, such as Linear Inverted Pendulum (LIP) [ 2 ] , SRB [ 8 , 35 , 12 , 18 , 14 ] , and centroidal model [ 7 , 39 , 33 ] , to generate approximate solutions. These simplified solutions are often used as warm starts or references for hierarchical optimization frameworks, which subsequently refine the trajectory using a detailed full-body model to ensure dynamic feasibility