Learning Humanoid Arm Motion via Centroidal Momentum Regularized Multi-Agent Reinforcement Learning

Ho Jae Lee, Se Hwan Jeon, Sangbae Kim · 2025

Read Learning Humanoid Arm Motion via Centroidal Momentum Regularized Multi-Agent Reinforcement Learning by Ho Jae Lee, Se Hwan Jeon, Sangbae Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-04140.txt.

Introduction

Ho Jae Lee, Se Hwan Jeon, and Sangbae Kim teach a humanoid robot to swing its arms while walking, in a 2025 paper. People swing their arms without thinking, and the motion helps balance and cuts wasted turning force. The team copy this idea by training the arms and legs as separate learners that share a view of the whole body's spin.

Each learner has its own goals, and the arm learner is rewarded for damping the body's angular momentum. The split design beat single-learner and other split setups in tests. The trained policy then ran on a real humanoid robot. It walked on flat ground, rough ground, and stairs, and held up well when pushed.

Catalog record

CREATORHo Jae Lee, Se Hwan Jeon, Sangbae Kim
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-04140

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learning Humanoid Arm Motion via Centroidal Momentum Regularized Multi-Agent Reinforcement Learning Ho Jae Lee, Se Hwan Jeon, Sangbae Kim Abstract Humans naturally swing their arms during locomotion to regulate whole-body dynamics, reduce angular momentum, and help maintain balance. Inspired by this principle, we present a limb-level multi-agent reinforcement learning (RL) framework that enables coordinated whole-body control of humanoid robots through emergent arm motion. Our approach employs separate actor-critic structures for the arms and legs, trained with centralized critics but decentralized actors that share only base states and centroidal angular momentum (CAM) observations, allowing each agent to specialize in task-relevant behaviors through modular reward design. The arm agent guided by CAM tracking and damping rewards promotes arm motions that reduce overall angular momentum and vertical ground reaction moments, contributing to improved balance during locomotion or under external perturbations. Comparative studies with single-agent and alternative multi-agent baselines further validate the effectiveness of our approach. Finally, we deploy the learned policy on a humanoid platform, achieving robust performance across diverse locomotion tasks, including flat-ground walking, rough terrain traversal, and stair climbing. I INTRODUCTION Arm swing is a natural and characteristic feature of human locomotion, but its fundamental role remains unclear. Biomechanical studies have offered a variety of explanations and consequences of arm swing during locomotion. Some argue that it reduces the induced angular momentum of the legs, ensuring smoother movement [ 1 ] . Consequently, the required ground reaction moments for locomotion are lower [ 2 ] , improving energy economy of gait patterns [ 3 ] . For humanoid platforms, however, it is less clear how to effectively and intentionally coordinate the arms during locomotion, especially with typical optimization or learning-based control methods. Designing such controllers is difficult due to the complexities of whole-body dynamics, the variability of locomotion tasks, and uncertainty on how arm motion should be guided with respect to the task, stylistically or otherwise. Prior work has typically approached controlling arm and leg motion in a unified manner. Imitation-based approaches try to match distributions of reference data, as in [ 4 , 5 , 6 ] . These controllers are able to exhibit human-like behavior, but it is unclear how (or if) mimicking these references offers advantages for locomotion—the primary reward signal is to imitate trajectories from higher degree-of-freedom systems, which is not necessarily aligned with robustness or performance for the specific platform. With reinforcement learning (RL) methods, potentially conflicting rewards make it challenging to plan arm and leg motions simultaneously, and arm motion is often regularized with heuristics that may have little physical basis other than stylistic tuning [ 7 ] . This balance becomes even more sensitive with loco-manipulation, such as in [ 8 ] , where different sets of limbs can be responsible for entirely different tasks. Model-based controllers have typically leveraged full-body optimization schemes to generate arm motion. Via centroidal momentum tracking [ 9 , 10 , 11 ] , relative angular-acceleration control [ 12 ] , or full-body model predictive control (MPC) [ 13 , 14 ] arm motion can arise naturally with a physically-consistent basis. However, solving these full-body motion plans becomes difficult to scale for online, closed-loop evaluation, often requiring approximations or special treatment for sufficient bandwidth. Additional considerations, such as avoiding self-collisions, state estimation noise, and environmental constraints, can further complicate these model-based controllers. While reasoning about the full dynamics of the legs and arms is ideal, the computational expense can make these control schemes intractable. Fig. 1 : Coordinated arm response under external torque disturbance. Two external forces F are applied from opposite sides to induce a positive yaw torque disturbance \tau_{z} about the vertical axis. The robot reacts with emergent arm motions that counteract the induced momentum, helping to recover and maintain balance. In contrast to these whole-body approaches, there is evidence that the arms and legs are not planned jointly at the biological level. Commands for the arms and legs arise from entirely separate cortical areas, and are only coupled for stabilization elsewhere in the cerebellum [ 15 , 16 ] . It is unclear whether this is simply vestigial. However, this separation in control can present several advantages from an engineering perspective. Optimizations for complex systems can be greatly accelerated by ”sectioning” the physics of the model, such as the front and rear legs of a quadruped, and ensuring consensus downstream [ 17 ] . Similarly, locomotion can be trained separately from manipulation tasks by abstracting arm motion as random inertial wrenches on the body, as in [ 18 ] . This controller design enables the limbs to be reasoned about separately without mutual interference, allowing for coordinated motion without intractable computation or meticulous tuning. Inspired by this architecture, we approach this challenge of coordinating locomotion with arm motion as a multi-agent RL problem —the arms and legs are treated as separate agents in the learning environment, connected only by the centroidal dynamics of the robot. Specifically, we adopt a centralized training and decentralized execution (CTDE) approach [ 19 , 20 ] , wherein decentralized actors for the arms and legs observe only their respective proprioceptive information, while centralized critics have access to global state information from both limb groups. This architecture enables faster convergence during training to a higher asymptotic task reward, and we analyze the effect of having centralized/decentralized observations for both the actor and critic. Additionally, we introduce a centroidal angular momentum reward to guide arm motion, motivated by prior biomechanical studies of locomotion and physically-grounded principles. The resultant controller demonstrates natural arm swing while walking and improved recovery from external disturbances, which we validate on hardware for a humanoid platform. We analyze the ground reaction moments and momentum during execution and find they corroborate the findings from [ 1 , 2 , 3 ] . We summarize our main contributions as follows: 1. We introduce a CAM reward based on biomechanical studies of human walking, and find that it guides the emergence of natural arm swing for stable locomotion and effective push recovery for our policy. 2. We propose a multi-agent RL framework employing separate actor-critic networks for the arms and legs, trained centrally but executed in a decentralized manner. 3. We demonstrate the effectiveness and practicality of our controller by validating its performance on a humanoid platform, both in simulation and hardware experiments 1 1 1 Youtube video : https://youtu.be/uPC4OlujM3k . II Background Fig. 2 : Overview of our limb-level multi-agent reinforcement learning framework with CAM regularization. Separate actor-critic policies are assigned to the arm and leg agents, each operating on distinct observation spaces and reward structures. During training, each critic is provided global information for centralized training, while actors execute using local observations, sharing only base states and CAM. This design enables coordinated whole-body behavior, as the arm and leg agents are implicitly coupled through the shared CAM signal, allowing the arm to respond to momentum generated by leg motion. II-A Centroidal Dynamics and Momentum The centroidal dynamics of a system [ 10 , 11 , 13 ] offers a balanced representation of robot dynamics, bridging the gap between simplified point-mass models and computationally expensive full-body dynamics. It is particularly valuable in humanoid control, as it captures whole-body interactions in a

Read the complete text