Reinforcement Learning for Robust Athletic Intelligence: Lessons from the 2nd “AI Olympics with RealAIGym” Competition
Felix Wiebe, Niccolò Turcato, Alberto Dalla Libera, Jean Seong Bjorn Choe, Bumkyu Choi, Tim Lukas Faust, Habib Maraqten, Erfan Aghadavoodi, Marco Cali, Alberto Sinigaglia, Giulio Giacomuzzo, Diego Romeres, Jong-kook Kim, Gian Antonio Susto, Shubham Vyas, Dennis Mronga, Boris Belousov, Jan Peters, Fr · 2025
Read Reinforcement Learning for Robust Athletic Intelligence: Lessons from the 2nd “AI Olympics with RealAIGym” Competition by Felix Wiebe, Niccolò Turcato, Alberto Dalla Libera, Jean Seong Bjorn Choe, Bumkyu Choi, Tim Lukas Faust, Habib Maraqten, Erfan Aghadavoodi, Marco Cali, Alberto Sinigaglia, Giulio Giacomuzzo, Diego Romeres, Jong-kook Kim, Gian Antonio Susto, Shubham Vyas, Dennis Mronga, Boris Belousov, Jan Peters, Fr (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15290.txt.
Introduction
Wiebe and a large team reported lessons in twenty twenty-five from a robotics competition on athletic intelligence. The second AI Olympics with RealAIGym, held at a twenty twenty-four robotics conference, tested controllers on a real double pendulum. This swinging two-arm system is underpowered and chaotic, making it a hard control challenge.
Four teams submitted reinforcement learning methods, where the controller learns by trial and error. The paper describes each method and scores them on the swing-up task using several criteria. It focuses on two practical questions: does skill learned in simulation carry to real hardware, and does control hold up under outside disturbances? Comparing the entries side by side shows the strengths and weaknesses of current learning-based control.
Catalog record
| CREATOR | Felix Wiebe, Niccolò Turcato, Alberto Dalla Libera, Jean Seong Bjorn Choe, Bumkyu Choi, Tim Lukas Faust, Habib Maraqten, Erfan Aghadavoodi, Marco Cali, Alberto Sinigaglia, Giulio Giacomuzzo, Diego Romeres, Jong-kook Kim, Gian Antonio Susto, Shubham Vyas, Dennis Mronga, Boris Belousov, Jan Peters, Fr |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-15290
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reinforcement Learning for Robust Athletic Intelligence: Lessons from the 2nd “AI Olympics with RealAIGym” Competition Thanks: Hessian.AI, Germany Felix Wiebe, Niccolò Turcato, Alberto Dalla Libera, Jean Seong Bjorn Choe, Bumkyu Choi, Tim Lukas Faust, Habib Maraqten, Erfan Aghadavoodi, Marco Cali, Alberto Sinigaglia, Giulio Giacomuzzo, Diego Romeres, Jong-kook Kim, Gian Antonio Susto, Shubham Vyas, Dennis Mronga, Boris Belousov, Jan Peters, Frank Kirchner, Shivesh Kumar Abstract In the field of robotics many different approaches ranging from classical planning over optimal control to reinforcement learning (RL) are developed and borrowed from other fields to achieve reliable control in diverse tasks. In order to get a clear understanding of their individual strengths and weaknesses and their applicability in real world robotic scenarios is it important to benchmark and compare their performances not only in a simulation but also on real hardware. The ’2nd AI Olympics with RealAIGym’ competition was held at the IROS 2024 conference to contribute to this cause and evaluate different controllers according to their ability to solve a dynamic control problem on an underactuated double pendulum system (Fig. 1 ) with chaotic dynamics. This paper describes the four different RL methods submitted by the participating teams, presents their performance in the swing-up task on a real double pendulum, measured against various criteria, and discusses their transferability from simulation to real hardware and their robustness to external disturbances. I Introduction Fig. 1: Onsite setup of the double pendulum hardware (left), the pendulum being disturbed with a stick (top right) and participating teams (bottom right) at IROS 2024. There are many frameworks for comparing control methods in simulation environments (e.g., [ 1 , 2 ] ) but only few provide a standardized real robot environment which is open and easily reproducible such as [ 3 ] with a quadruped robot and [ 4 ] with a three end-effector robot for manipulation tasks. The RealAIGym project focuses on simple, low degree of freedom systems capable of dynamic behaviors, such as a simple pendulum [ 5 ] , AcroMonk [ 6 ] , RicMonk [ 7 ] and the dual purpose double pendulum [ 8 ] used in this competition. With the software and hardware being open source and a user friendly Python API, RealAIGym intends to be an open benchmarking platform for comparing control methods on real hardware. This is the 2nd iteration of this competition after the first one having been held at the IJCAI 2023 conference [ 9 ] . II Competition Rules The competition was conducted with the dual purpose double pendulum system introduced in [ 8 ] which can be operated as an acrobot or pendubot without changing the hardware. For the competition acrobot and pendubot were treated as separate challenges and evaluated in separate tracks. The task on both systems is to swing up the pendulum from the free hanging position to the upright position and stabilize it there. To test the controllers’ robustness, external perturbations are applied on both motors during the execution. The used system had an attached mass of 0.5\,\text{kg} and link lengths of 0.2\text{m} and 0.3\text{m} . Model parameters identified with least squares optimization were provided to the participants, but the teams were free to do their own system identification. The competition was carried out in three phases: (i) A simulation phase, (ii) a remote hardware phase and (iii) an onsite phase. In the simulation phase the participants were asked to develop a controller for the swing-up and balance task on the acrobot and/or pendubot system and integrate it into the double pendulum toolkit on github 1 1 1 https://github.com/dfki-ric-underactuated-lab/double_pendulum . After the simulation phase, the four best performing controllers were advanced to the next phase. In the remote hardware phase, the teams got the opportunity to remotely operate the pendulum hardware and collect data from the real system along with live video feed. The third phase took place on site at the IROS 2024 conference, where the teams were able to fine-tune their controllers before the final evaluation. For the evaluation of the controllers different criteria were considered. Simulation results were evaluated with the performance score , where the controllers show their basic functionality. and with the robustness score , where the controllers show their robustness to external influences. The formula for the performance score is: \displaystyle S_{p}=c_{succ}\left[1-\frac{1}{5}\sum_{i\in\{t,E,\tau_{c},\tau_{s},v\}}\tanh{(w_{i}c_{i})}\right] (1) based on the swing-up success c_{succ}\in\{0,1\} , the swing-up time c_{t} , the used energy c_{E} , the torque cost c_{\tau_{c}} , the torque smoothness c_{\tau_{s}} and the velocity cost c_{v} . These quantities are normalized with the weights in Table I and scaled with the \tanh function to the interval [0,1] . The second score computed from the simulation results is the robustness score: \displaystyle S_{r}=\frac{1}{6}\big(c_{m}+c_{v,\text{noise}}+c_{\tau,\text{noise}}+c_{\tau,\text{resp.}}+c_{d}+c_{p}\big) (2) with the criteria • Model inaccuracies c_{m} : The independent model parameters are varied one at the time in the simulator while using the original model parameters in the controller. • Measurement noise c_{v,\text{noise}} : Artificial noise is added to the velocity measurements. • Torque noise c_{\tau,\text{noise}} : Artificial noise is added to the motor torque. • Torque response c_{\tau,\text{resp.}} : A delayed motor response is modeled by applying the torque \tau=\tau_{t-1}+k_{resp}(\tau_{des}-\tau_{t-1}) instead of the desired torque \tau_{des} , where, \tau_{t-1} is the applied motor torque from the last time step and k_{resp} is the factor which scales the responsiveness. • Time delay c_{d} : A time delay is added to all measurements. • Random perturbations c_{p} : A randomly generated sequence of Gaussian perturbations is added to the applied torque. For each criterion the severity of the external influence is varied in N=21 steps (for the model inaccuracies for each independent model parameter) and the score is the percentage of successful swing-ups under these circumstances. For the perturbation criterion 50 random perturbations profiles are generated and evaluated. For hardware phases, only the performance score, S_{p} ( 1 ), was calculated with slightly adjusted weights (Table I ), but the controllers had to showcase their robustness by reacting to unknown external perturbations during the experiment. The perturbations consist of Gaussian torque profiles on both joints at random times, which were unknown to the controllers. Each trial lasts for 10\,\text{s} and is considered successful if, in the end, the end-effector is at a height above the threshold of 0.45\,\text{cm} above the base. The teams were allowed to use up to 0.5\,\text{Nm} torque on the passive joint for friction compensation. Each controller was tested in 10 trials and the final score is the average of the individual scores. TABLE I: Weights for the performance score ( 1 ). w_{t} w_{E} w_{\tau_{c}} w_{\tau_{s}} w_{v} Simulation \pi/20 \pi/60 \pi/20 10\pi \pi/400 Hardware \pi/20 \pi/60 \pi/100 \pi/4.0 \pi/400 III Algorithms Four teams advanced to the final round of the competition and tested their control methods on the real hardware. This section briefly introduces the four methods and how they approached the task to improve robustness. III-A Model-based RL: MC-PILCO Fig. 2: Schematic representation of the robust MC-PILCO algorithm. Each Episode is composed of (i) model learning, (ii) policy update and (iii) policy execution. MC-PILCO (Monte Carlo - Probabilistic Inference for Learning COntrol) [ 10 ] is a Model-Based policy gradient RL algorithm. MC-PILCO relies on Gaussian Processes (GPs) to learn the system