GPO: Growing Policy Optimization for Legged Robot Locomotion and Whole-Body Control

Shuhao Liao, Peizhuo Li, Xinrong Yang, Linnan Chang, Zhaoxin Fan, Qing Wang, Lei Shi, Yuhong Cao, Wenjun Wu, Guillaume Sartoretti · 2026

Read GPO: Growing Policy Optimization for Legged Robot Locomotion and Whole-Body Control by Shuhao Liao, Peizhuo Li, Xinrong Yang, Linnan Chang, Zhaoxin Fan, Qing Wang, Lei Shi, Yuhong Cao, Wenjun Wu, Guillaume Sartoretti (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20668.txt.

Introduction

Shuhao Liao and nine co-authors study how to train legged robots to walk and move their whole bodies. Their 2026 arXiv paper presents a method they call Growing Policy Optimization, or GPO. The work uses reinforcement learning, where a robot learns by trial and error. The authors frame this as a hard training problem.

The paper explains why such training is so hard in practice. The robot must choose smooth actions across many joints at once. Real hardware also limits what the robot can safely try. And the robot gets few chances to test new moves. GPO is the team's answer to these limits. Readers get a clear view of one current path toward robots that walk well and use their full bodies with skill.

Catalog record

CREATORShuhao Liao, Peizhuo Li, Xinrong Yang, Linnan Chang, Zhaoxin Fan, Qing Wang, Lei Shi, Yuhong Cao, Wenjun Wu, Guillaume Sartoretti
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-20668

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GPO: Growing Policy Optimization for Legged Robot Locomotion and Whole-Body Control Shuhao Liao, Peizhuo Li, Xinrong Yang, Linnan Chang, Zhaoxin Fan, Qing Wang, Lei Shi, Yuhong Cao, Wenjun Wu, Guillaume Sartoretti Abstract Training reinforcement learning (RL) policies for legged robots remains challenging due to high-dimensional continuous actions, hardware constraints, and limited exploration. Existing methods for locomotion and whole-body control work well for position-based control with environment-specific heuristics (e.g., reward shaping, curriculum design, and manual initialization), but are less effective for torque-based control, where sufficiently exploring the action space and obtaining informative gradient signals for training is significantly more difficult. We introduce Growing Policy Optimization (GPO), a training framework that applies a time-varying action transformation to restrict the effective action space in the early stage, thereby encouraging more effective data collection and policy learning, and then progressively expands it to enhance exploration and achieve higher expected return. We prove that this transformation preserves the PPO update rule and introduces only bounded, vanishing gradient distortion, thereby ensuring stable training. We evaluate GPO on both quadruped and hexapod robots, including zero-shot deployment of simulation-trained policies on hardware. Policies trained with GPO consistently achieve better performance. These results suggest that GPO provides a general, environment-agnostic optimization framework for learning legged locomotion. I INTRODUCTION Learning-based methods have advanced embodied artificial intelligence, enabling robots to acquire increasingly complex skills through interaction. This progress is particularly evident in legged robot locomotion and whole-body control, where control problems are inherently continuous, involve rich and intermittent contacts with the environment, and demand strong generalization across terrains, disturbances, and operating conditions. Among these approaches, RL has demonstrated strong potential by directly exploiting high-dimensional sensory feedback and complex nonlinear dynamics, resulting in agile and adaptive behaviors beyond the reach of conventional model-based controllers [ 10 , 32 , 23 ] . Despite these advances, learning reliable policies for such systems remains challenging. Current learning-based methods often struggle to simultaneously achieve robust generalization and safe behavior, particularly during training and early deployment. Fig. 1 : Overview of GPO. GPO gradually expands the effective action space during training, enabling stable early optimization and improving asymptotic performance. A central challenge is learning policies over high dimensional, continuous action spaces under strict torque, velocity, contact, and safety constraints. Moreover, real-world rollouts are costly, risky, and difficult to scale, limiting the data available for trial and error learning. To address these challenges, prior work has largely focused on modifying the training pipeline rather than the underlying optimization mechanism. A common strategy is reward shaping, where task priors such as gait regularization, energy penalties, or contact heuristics are manually encoded to stabilize learning and bias policy search toward feasible locomotion patterns [ 38 , 41 , 12 ] . Curriculum learning approaches gradually increase task difficulty by varying terrain complexity, initial states, or disturbance levels, improving performance on specific environments [ 20 , 5 , 24 , 25 ] . Other methods rely on expert demonstrations or supervised pre-training to warm-start RL in favorable regions of the policy manifold [ 30 , 43 , 15 ] . On the action side, practitioners use clipping, rescaling, or saturation of control commands to satisfy hardware limits and reduce destabilizing high variance actuation [ 8 , 18 , 35 ] . These works have enabled impressive results across diverse robotic platforms, suggesting that carefully managing exploration and action magnitudes is crucial for successful embodied RL [ 22 , 27 , 34 ] . Despite these successes, most approaches modify the training pipeline but leave the optimization mechanism unchanged, and their effect on optimization properties is unclear. Reward shaping, curricula, and action clipping change the visited state-action distribution, but their impact on gradient variance, local convergence, and asymptotic performance is not well understood. Importantly, most widely used policy-gradient methods implicitly assume a fixed action space throughout training. This design encourages wide exploration early in training, leading to high gradient variance and a rugged optimization landscape. As a result, the policy may fail to first learn reliable low-amplitude control before converging to an optimal policy. In practice, engineers often restrict the “effective” action space early (e.g., tighter torque limits or aggressive clipping) and relax it later [ 22 , 18 ] . However, there is no principled framework that treats the evolution of the action space as part of the learning algorithm, nor a clear analysis of its effect on optimization and performance. In this work, we take a step toward addressing this gap by proposing GPO, a framework in which the effective action space is no longer fixed but gradually expands during training. GPO introduces a smooth, time-varying transformation that maps a fixed policy output to a growing range of executable actions. This design allows the agent to first learn reliable behaviors within a restricted action space, where the optimization is easier to navigate and gradient estimates are more stable, and then progressively unlock larger action space as training proceeds. Crucially, this transformation is equivalent to PPO updates in structure: the underlying objective and clipping mechanism remain unchanged up to a change of variables, and we prove that the discrepancy between GPO gradients and standard PPO gradients is bounded. Under mild assumptions, GPO achieves faster early-stage convergence and higher asymptotic return than PPO with a fixed action space. We validate these properties in simulation and on real robots under torque-based control, where actions are directly mapped to joint torques, leading to strong coupling and stringent stability requirements. Overall, GPO provides a general, environment-agnostic mechanism for improving training efficiency and final performance in RL-based robotic control. II RELATED WORK Learning-based legged robot locomotion has advanced rapidly in recent years with the adoption of RL, enabling agile and dynamic behaviors [ 29 , 16 , 11 , 19 , 33 , 2 , 14 , 4 ] . Many works focus on proprioceptive locomotion, where policies rely exclusively on onboard sensing data such as IMU and joint encoder measurements and are trained using actor-critic frameworks. To improve state evaluation and policy learning under partial observations, prior approaches incorporate privileged information during training [ 34 ] , historical observations via teacher-student distillation [ 22 , 21 , 32 ] , or direct state estimation from temporal windows [ 28 ] . These techniques have proven effective for reliable locomotion over diverse terrains and have established mature learning pipelines for legged robot control under limited sensing. Beyond state estimation, a wide range of methods aim to improve training efficiency in high-dimensional continuous control. Early approaches combine imitation learning with reinforcement learning to improve sampling efficiency, enabling agile locomotion behaviors that adapt to different terrains and disturbances [ 30 , 31 ] . Subsequent works introduce actuator models and domain randomization, and teacher-student frameworks to enable robust sim-to-real transfer [ 17 ] . Meanwhile, advances in massively parallel simulation have greatly improved sampling efficiency, reducing training duration to mere minutes and supporting more complex control architectures [ 26 , 21 , 34 , 44 , 9 ] . Additional efforts integrate learning-based controllers with traditional control structures, adaptive compensation, or bio-inspired priors to enhance robustness and generalization [ 1 , 40 , 13 , 42 , 3 , 37 ] . While effective in practice, these

Read the complete text