Variable Stiffness for Robust Locomotion through Reinforcement Learning* * accepted to the 16th IFAC joint symposia of mechatronics and robotics
Dario Spoljaric, Yan Yashuai, Dongheui Lee · 2025
Read Variable Stiffness for Robust Locomotion through Reinforcement Learning* * accepted to the 16th IFAC joint symposia of mechatronics and robotics by Dario Spoljaric, Yan Yashuai, Dongheui Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-09436.txt.
Introduction
Dario Spoljaric, Yashuai Yan, and Dongheui Lee present a 2025 study on teaching legged robots to walk with adjustable joint stiffness. Reinforcement learning can produce lively gaits, but tuning each joint's stiffness by hand eats time. Their method puts stiffness into the learned controls, grouped three ways: per joint, per leg, or a hybrid of both. The robot then sets stiffness as it walks, alongside joint positions.
Per-leg grouping beat plain position control at tracking speed and at recovering from pushes. The hybrid grouping used the least energy. Although training happened only on flat floors, the robots walked well on varied outdoor ground, a sign of strong transfer from simulation. The approach removes a tedious tuning step while keeping results competitive.
Catalog record
| CREATOR | Dario Spoljaric, Yan Yashuai, Dongheui Lee |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-09436
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Variable Stiffness for Robust Locomotion through Reinforcement Learning* * accepted to the 16th IFAC joint symposia of mechatronics and robotics Dario Spoljaric, Yan Yashuai, Dongheui Lee Abstract Reinforcement-learned locomotion enables legged robots to perform highly dynamic motions but often accompanies time-consuming manual tuning of joint stiffness. This paper introduces a novel control paradigm that integrates variable stiffness into the action space alongside joint positions, enabling grouped stiffness control such as per-joint stiffness (PJS), per-leg stiffness (PLS) and hybrid joint-leg stiffness (HJLS). We show that variable stiffness policies, with grouping in per-leg stiffness (PLS), outperform position-based control in velocity tracking and push recovery. In contrast, HJLS excels in energy efficiency. Despite the fact that our policy is trained on flat floor only, our method showcases robust walking behaviour on diverse outdoor terrains, indicating robust sim-to-real transfer. Our approach simplifies design by eliminating per-joint stiffness tuning while keeping competitive results with various metrics. I Introduction Animal and human-like locomotion has a significant advantage over wheeled mobile robots. They can traverse unstructured, challenging terrains. Therefore, various approaches are developed to solve quadrupedal and bipedal locomotion. Conventionally, this involved model-based controllers ( Donghyun et al., 2019 ; Jared et al., 2018 ) with a complex pipeline that managed gait schedule, state estimation, whole body impulse control and actuator control. Recently, this discipline has made significant progress through model-free reinforcement learning (RL) methods. These approaches enable the design of controllers capable of following high-level commands (walking, running, jumping, etc.) and directly actuating the joint motors, bypassing the need for path planning and other parts within the control pipeline. Usually, these controllers ( Shuxiao et al., 2023 ; Gilbert et al., 2023 ) follow a position- or torque-based paradigm. The high-level controller (RL agent) learns a position policy in the position-based paradigm. Given the state, the RL agent predicts desired joint positions at low frequencies, which are transferred into torques by a high-frequency PD controller. This control paradigm requires manual engineering of motor stiffness and damping for different tasks. In contrast, humans and animals can adjust their stiffness and damping to handle different tasks. For example, we stiffen our foot joints when landing with a foot but relax in the swing phase. The torque-based control circumvents this by directly learning a torque policy, which shows higher compliance ( Donghyeon et al., 2023 ) . Generally, torque-based policies are more challenging to train because they require learning complex dynamics. On the other hand, position control has a good initial pose, easing the learning progress in the beginning. Torque-based RL agents are usually executed at higher speeds, necessitating more powerful hardware or smaller networks. This leaves the researcher with a choice, either additional tuning of joint stiffnesses or a hard-to-train policy with limits in scope. Fig. 1 : A quadrupedal robot traverses across different terrains with variable-stiffness RL policy. Another approach ( Lei et al., 2024 ; Jiaming et al., 2024 ) is variable stiffness control, which is widely used in robotics, particularly in manipulator applications, to improve safety while still being able to track accurately. Preliminary studies from Lei et al., 2024 have shown that applying this technique to locomotion could enhance energy efficiency. This also seems reasonable as higher stiffness is required during contact with the ground, but less is needed during the swing phase. In quadruped locomotion, research from Xinyuan et al., 2022 within model-based frameworks has shown that variable stiffness can reduce contact forces. However, successfully achieving locomotion with variable stiffness remains a significant challenge. This paper studies reinforcement-learned controllers that learn joint stiffness alongside target positions. Consequently, the robot can automatically adjust the motor stiffness according to the task requirements. Our approach shows superior velocity tracking and push recovery performance while maintaining good energy efficiency and robust sim-to-real transfer performance. Results of the walking policy are shown in Fig. 1 and a video is available online 1 1 1 https://drive.google.com/file/d/1SmwQSM5026Ri41Ue6J_IgP0NqYIFxCOt/view?usp=sharing . Fig. 2 : Architecture of the position-based control (blue dashed line) compared to our variable stiffness control. II Related work RL has emerged as a promising approach for solving locomotion tasks, offering two primary paradigms: position-based and torque-based control. Gilbert et al., 2023 and Nahrendra et al., 2023 predict target joint positions, which are actuated using proportional-derivative (PD) controllers. These methods are widely favoured for their ease of training and robustness in high-level command tracking. Within this control, the torque applied to the motors is calculated using Eq. ( 1 ) \tau_{t}=K_{p}(q_{t}^{target}(\theta)-q_{t})+K_{d}(\dot{q}_{des}-\dot{q_{t}}) (1) However, they suffer from limited compliance behaviour due to fixed stiffness and damping values ( K_{p} and K_{d} ), requiring extensive PD gain tuning that is often task- and robot-specific ( Gilbert et al., 2023 ) . In this control, the PD controller acts as a low-level tracking module, where the proportional gain (P-gain) is representative of the stiffness and the derivative gain (d-gain) for the damping. Zhaoming et al., 2021 investigated the impact of the p-gains, suggesting that large proportional gain leads to instabilities in training. In contrast, the low proportional gain has significant tracking errors and behaves like a torque controller. Other research studied the impact on the derivative gain. Laura et al., 2023 showed small derivative gains result in learning instabilities, and large gains prevented tracking the target velocity. To circumvent this PD controller, Shuxiao et al., 2023 and Donghyeon et al., 2023 studied torque control as an alternative and applied it to quadruped and biped locomotion. In this control, the actions are directly applied to the motors. Although this control showed higher achievable rewards in the long term, it must be executed at higher speeds to perform similarly to position control. It is more difficult to train initially. The higher control speeds limit the design freedom of torque-based controllers. Xinyuan et al., 2022 showed for model-based control, that adapting stiffness according to the contact force led to sufficient walking for a quadruped on uneven terrains. Bogdanovic et al., 2020 studied the impact of including joint stiffness alongside joint positions. The torque was then calculated by Eq. ( 2 ). \tau_{t}=K_{t}^{p}(\theta)(q_{t}^{target}(\theta)-q_{t})-K_{t}^{d}(\theta)\dot{q_{t}} (2) Overall, their work did not test their approach to locomotion but showed the superiority of this concept against torque and position control for two scenarios: a single-legged hopper and a manipulator. The single-legged hopper achieved more considerable jumping heights, and the manipulator maintained a continuous contact force during reference motions with the adjustment of the action space. Inspired by these findings, we explore the integration of joint stiffness alongside joint positions in the action space for reinforcement learning-based quadruped locomotion. III Methods We aim to train a policy \pi_{\theta} with parameters \theta that can follow high-level velocity command \mathbf{v}^{cmd}=[\mathbf{v}_{xy}^{cmd},\omega_{yaw}^{cmd}]^{T} . This includes the lateral velocity v_{xy}^{cmd} as well as angular rotation speed \omega_{yaw}^{cmd} . The prediction of joint targets along joint stiffnesses should accomplish this. An overview of our approach is shown in Fig. 2 . III-A Training We train our controllers using Proximal Policy Optimisation (PPO) ( Schulman et al., 2017 ) with 4096 environments in parallel for 2000 epochs. To improve learning efficiency, we apply early termination if the robot’s orientation exceeds 90 degrees from its horizontal position, if joint or torque limits are exceeded or if the robot falls onto its