Reinforcement Learning Enabled Adaptive Multi-Task Control for Bipedal Soccer Robots

Yulai Zhang, Yinrong Zhang, Ting Wu, Linqi Ye · 2026

Read Reinforcement Learning Enabled Adaptive Multi-Task Control for Bipedal Soccer Robots by Yulai Zhang, Yinrong Zhang, Ting Wu, Linqi Ye (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-19104.txt.

Introduction

Zhang and three colleagues propose a reinforcement learning framework for multi-task control of bipedal soccer robots. An open loop oscillator plus a learning residual separates basic gait generation from complex football actions. A posture driven state machine switches between a ball seeking and kicking network and a fall recovery network, preventing state interference. The fall recovery network trains through a progressive force attenuation curriculum. Unity simulations show reliable ball finding and kicking even in restricted corners, plus rapid fall recovery, averaging 0.715 seconds in tests.

Catalog record

CREATORYulai Zhang, Yinrong Zhang, Ting Wu, Linqi Ye
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22911
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.19104

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reinforcement Learning Enabled Adaptive Multi-Task Control for Bipedal Soccer Robots Yulai Zhang, Yinrong Zhang, Ting Wu, Linqi Ye Abstract Developing bipedal football robots in dynamic combat environments presents challenges related to motion stability and deep coupling of multiple tasks, as well as control switching issues between different states such as upright walking and fall recovery. To address these problems, this paper proposes a modular reinforcement learning (RL) framework for achieving adaptive multi-task control. Firstly, this framework combines an open-loop feedforward oscillator with a reinforcement learning-based feedback residual strategy, effectively separating the generation of basic gaits from complex football actions. Secondly, a posture-driven state machine is introduced, clearly switching between the ball seeking and kicking network (BSKN) and the fall recovery network (FRN), fundamentally preventing state interference. The FRN is efficiently trained through a progressive force attenuation curriculum learning strategy. The architecture was verified in Unity simulations of bipedal robots, demonstrating excellent spatial adaptability-reliably finding and kicking the ball even in restricted corner scenarios-and rapid autonomous fall recovery (with an average recovery time of 0.715 seconds). This ensures seamless and stable operation in complex multi-task environments. The source code and demonstration video are available at https://openatom.tech/zhang117182/2025competition_gewu_nanoloong . I INTRODUCTION As artificial intelligence advances toward general intelligence, embodied intelligence has emerged as a pivotal frontier emphasizing physical embodiment and interaction. Unlike passive perception agents, embodied agents actively interact with physical environments to enable adaptive learning via dynamic sensorimotor coupling, bridging theory with real-world deployment [ 1 ] . Robot soccer—requiring the integration of perception, motion control, and decision-making in dynamic adversarial settings—serves as a critical benchmark for embodied intelligence [ 2 ] . Methods combining learning from demonstration and behavior-based control enhance skill learning efficiency [ 3 ] , while posture stabilization strategies address core disturbance rejection in locomotion [ 4 ] , laying a foundation for reliable small-scale robot operations in competition. The integration of reinforcement learning (RL) and physical simulation has significantly accelerated the development of robot soccer control. Deep RL successfully trains basic adversarial behaviors [ 5 ] , while skill primitives validate modular collaboration in open-source platforms [ 6 , 7 ] . For complex task decomposition [ 8 ] , researchers increasingly adopt embodied methods that couple perception with action [ 9 , 10 ] . Additionally, imitation learning [ 11 ] and RL [ 12 ] are crucial for bipedal robot locomotion. System stability and task execution capabilities are further enhanced through multi-posture fall recovery [ 13 ] , multi-objective optimization [ 14 ] , and distributed multi-agent coordination [ 15 , 16 ] . Recent studies highlight spatial perception [ 17 ] , optimized neural networks for ball control [ 18 ] , and curriculum learning [ 19 ] , all of which can be efficiently developed within platforms like Unity [ 20 ] . Despite these advances, developing integrated, multi-skill control policies for bipedal robots remains formidable. First, simultaneously learning foundational locomotion and complex interactions (e.g., ball seeking) intrinsically couples low-level stability with high-level objectives, necessitating intricate reward formulations [ 21 , 22 ] . Second, orchestrating seamless transitions between distinct dynamic states (e.g., upright walking and fall recovery) without feature space interference is highly demanding for state modeling [ 23 , 24 ] . Thus, harmonizing rhythmic gaits with multi-state task execution remains a key pursuit. To address these challenges, this paper proposes a modular RL framework for dynamic adversarial tasks in bipedal robots. To resolve task coupling, an open-loop feedforward oscillator maintains stepping rhythms while an RL policy outputs residual actions for high-level maneuvers. Furthermore, a posture-driven state-machine toggles between ball seeking and kicking network and a fall recovery network, ensuring feature decoupling during state transitions. Validated in Unity, the architecture demonstrates robust fall recovery and exceptional spatial adaptability—reliably kicking balls in constrained corners, providing an efficient paradigm for multi-task embodied control. II RELATED WORK II-A Reinforcement Learning in Robot Soccer Robot soccer (e.g., RoboCup) serves as a standard benchmark for evaluating highly dynamic locomotion and object interaction. Traditionally, bipedal robots relied on hand-crafted kinematic models, such as Zero-Moment Point, and scripted controllers for basic walking and kicking [ 25 ] . Recently, end-to-end Reinforcement Learning (RL) has successfully synthesized agile soccer skills directly from environmental interactions [ 26 ] . However, learning bipedal locomotion alongside complex interaction tasks typically demands extensive reward formulation. To address this, our study integrates an open-loop oscillator to generate basic stepping rhythms, while an RL policy outputs residual actions for specific soccer maneuvers like ball seeking and kicking. II-B Fall Recovery in Bipedal Robots Fall recovery is a critical challenge for bipedal robots, and recent studies have focused on applying RL and multi-task learning to enable robots to recover efficiently from falls. Multi-task learning frameworks aim to optimize the robot’s ability to switch between fall recovery and other tasks such as ball kicking and dribbling. Several approaches, such as [ 27 ] , have investigated fall recovery systems using RL. II-C Curriculum Learning in Robotic Control Curriculum learning is an effective strategy for incrementally training complex tasks by gradually increasing task difficulty. In reinforcement learning, it has been successfully applied to teach robots behaviors, starting with simpler tasks and progressing to more complex ones. In fall recovery, curriculum learning reduces assistance as the robot improves its recovery ability. For instance, higher force assistance was initially provided and gradually reduced as the robot became more proficient [ 28 ] . This progressive approach stabilizes learning, reduces failure risk, and improves task performance. In our work, we use curriculum learning to adjust recovery forces based on the robot’s progress, ensuring smooth and efficient training. III METHODS This section presents a multi-task reinforcement learning control framework for biped robot soccer tasks, aiming to develop an autonomous robot soccer system based on reinforcement learning. III-A Research Goal The soccer task scenario is shown in Fig. 1 , which is a sub-module of Gewu Playground built on Unity (https://github.com/loongOpen/Unity-RL-Playground). The scenario includes a soccer field, a soccer ball, and two Tinker robots. The goal is to implement a fully autonomous 1v1 robot soccer match with no human involvement, including the robot’s ability to stand up after falling. Fig. 1 : Soccer Task Scenario. The robot we used is a small open-source biped robot, NanoLoong-Bipedal (also named Tinker) from OpenLoong (https://github.com/loongOpen/NanoLoong-Bipedal), equipped with 10 rotational joints, with 5 degrees of freedom (DOF) per leg: three at the hip (Y/R/P), one at the knee (P), and one at the ankle (P). Tinker’s zero-position posture is shown in Fig. 2 . This robot is inspired from Disney’s BDX robot and aims to providing an open-source, low-cost research and education platform. Fig. 2 : Tinker zero position. (a) Front view (b) Side view III-B Control Architecture The overall architecture is shown in Fig. 3 . Fig. 3 : Multi-Task RL Control Architecture for Tinker. The framework comprises three core modules: perception layer , multi-task RL layer , and action control layer . The perception layer captures robot posture and soccer environment via state observation for decision-making input. As the core module, the multi-task RL layer dynamically dispatches fall recovery and ball seeking/kicking networks through task decision/switching mechanisms to handle these two core tasks. The action

Read the complete text