Reward Training Wheels: Adaptive Auxiliary Rewards for Robotics Reinforcement Learning
Linji Wang *, Tong Xu *, Yuanjie Lu, Xuesu Xiao · 2025
Read Reward Training Wheels: Adaptive Auxiliary Rewards for Robotics Reinforcement Learning by Linji Wang *, Tong Xu *, Yuanjie Lu, Xuesu Xiao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15724.txt.
Introduction
Linji Wang and three co-authors present a 2025 method that tunes helper rewards for robot learning on the fly. The catalog author field garbles the names with stray marks, so the byline follows the head of the text. Robot training leans on hand-built bonus rewards, which take effort, bake in bias, and cannot shift as the robot improves.
Their teacher-student setup watches the student's growth and reweights each bonus to serve the main goal best. On tight-space navigation and rough-ground driving in simulation, it beat expert-built rewards and trained faster. On real robots it won all five trials against two for the expert set, and cut tilt angles nearly in half. The authors frame the method as training wheels that adjust themselves.
Catalog record
| CREATOR | Linji Wang *, Tong Xu *, Yuanjie Lu, Xuesu Xiao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-15724
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reward Training Wheels: Adaptive Auxiliary Rewards for Robotics Reinforcement Learning Linji Wang *, Tong Xu *, Yuanjie Lu, Xuesu Xiao Abstract Robotics Reinforcement Learning (RL) often relies on carefully engineered auxiliary rewards to supplement sparse primary learning objectives to compensate for the lack of large-scale, real-world, trial-and-error data. While these auxiliary rewards accelerate learning, they require significant engineering effort, may introduce human biases, and cannot adapt to the robot’s evolving capabilities during training. In this paper, we introduce Reward Training Wheels (RTW), a teacher-student framework that automates auxiliary reward adaptation for robotics RL. To be specific, the RTW teacher dynamically adjusts auxiliary reward weights based on the student’s evolving capabilities to determine which auxiliary reward aspects require more or less emphasis to improve the primary objective. We demonstrate RTW on two challenging robot tasks: navigation in highly constrained spaces and off-road vehicle mobility on vertically challenging terrain. In simulation, RTW outperforms expert-designed rewards by 2.35% in navigation success rate and improves off-road mobility performance by 122.62%, while achieving 35% and 3X faster training efficiency, respectively. Physical robot experiments further validate RTW’s effectiveness, achieving a perfect success rate (5/5 trials vs. 2/5 for expert-designed rewards) and improving vehicle stability with up to 47.4% reduction in orientation angles. I Introduction Designing robust and sample-efficient Reinforcement Learning (RL) methods for real-world robotics remains an enduring challenge. Whether the goal is to enable a ground robot to navigate narrow corridors or to drive an off-road vehicle up steep, uneven terrain, reward functions are pivotal to guide an agent’s behavior. In principle, one could rely on sparse, goal-only rewards—for example, awarding a single bonus upon successfully reaching a target. However, such sparse feedback typically proves insufficient for complex robotic tasks, especially considering real-world data scarcity, as the agent has little incentive to avoid obstacles or maintain safe maneuvers, making it difficult to reach the goal and receive the sparse reward through trial and error [ 1 , 2 ] . To address this limitation, roboticists introduce auxiliary rewards—terms that penalize collisions, encourage partial progress, or maintain safety constraints. While these auxiliary signals can dramatically accelerate learning, they also multiply the engineering burden: one must decide which auxiliary terms to include and how much to weigh each one, often through human trial and error and still leading to biased reward design [ 3 , 4 , 5 ] . Worse yet, an auxiliary reward that benefits early exploration (e.g., collision penalties to teach cautious movement) can become counterproductive later, as the robot may overly prioritize always staying unnecessarily far from obstacles rather than completing the task efficiently. Recent works in robotic RL reveal the extent of this auxiliary reward design burden. Tasks like navigation in highly constrained spaces [ 6 , 7 , 8 ] commonly include several hand-tuned rewards (e.g., distance-to-goal bonuses, collision penalties, velocity constraints, or time penalties), carefully balanced by human intuition [ 9 , 10 , 11 ] . Similarly, off-road mobility on vertically challenging terrain often relies on auxiliary terms to mitigate unsafe maneuvers—penalizing rollovers or slipping—while rewarding forward progress [ 12 ] . In each case, manually engineering these auxiliary signals can be time-consuming and prone to human biases and overfitting a particular training phase. These challenges highlight the need for a principled approach that can adapt auxiliary rewards throughout the training process. Imagine training wheels used to learn how to ride a bicycle. An effective human teacher uses training wheels at the beginning to guide learning in a safe manner, encourages the student to depend less and less on them as learning progresses, and eventually completely remove them. Inspired by the diminishing role played by the training wheels, we hypothesize that RL auxiliary rewards also need to adapt to the student capabilities. In this paper, we introduce Reward Training Wheels (RTW), a teacher-student framework that preserves the robot’s ultimate task objective (learning to ride a bicycle) while dynamically adapting auxiliary components (reducing the reliance on the training wheels). The teacher agent monitors key training signals—such as success rate and other performance metrics—to assess the student robot’s current capability and adjust auxiliary reward weights accordingly. As a result, the student leverages RTW to improve learning efficiency and performance through the adaptive auxiliary rewards throughout the learning process. We apply RTW on two difficult robotic tasks: (1) navigation in highly constrained spaces and (2) off-road vehicle mobility on vertically challenging terrain. Compared to expert-designed auxiliary rewards, our approach outperforms by 2.35% in navigation success rate and improves off-road mobility performance by 122.62%, while achieving 35% and 3X faster training efficiency, respectively. Physical robot experiments further validate RTW’s effectiveness, achieving a perfect success rate (5/5 trials vs. 2/5 for expert-designed rewards) and improving vehicle stability with up to 47.4% reduction in orientation angles. II Related Work RTW is related with curriculum learning and reward shaping, for which we review related work in this section. II-A Curriculum Learning Curriculum learning, first formalized by Bengio et al. [ 13 ] , has become increasingly important in RL as a strategy to improve sample efficiency and final performance. Traditional curriculum approaches in robotics focus on progressively increasing task difficulty [ 14 , 15 , 16 ] , often by modifying environmental parameters such as obstacle density [ 17 ] or terrain complexity [ 18 ] . Recent work by Freitag et al. [ 19 ] demonstrated that well-designed curricula can significantly enhance sample efficiency in robotic navigation tasks. Similarly, Mehta et al. [ 20 ] employed a curriculum-based approach for learning complex manipulation skills by progressively introducing more challenging scenarios. While effective, these approaches introduce additional engineering complexity by requiring multiple task variants and can lead to distribution shifts when transferring to the target environment. Unlike these methods that primarily modify task configurations or initial states, our teacher-student framework maintains a consistent environment while dynamically adapting the auxiliary reward structure as the student progresses. II-B Reward Shaping Reward shaping has long been recognized as a critical element in RL [ 1 , 2 ] . Potential-based reward shaping [ 1 ] provides theoretical guarantees for preserving optimal policies, though how to design potential functions to efficiently lead to optimality remains challenging. Devlin and Kudenko [ 21 ] extended this to dynamic potential-based reward shaping, allowing the shaping function to change over time but still requiring domain knowledge for potential function design. Methods for automatic reward design have emerged as alternatives to manual engineering. Singh et al. [ 22 ] explored intrinsically motivated RL, where auxiliary rewards emerge from the agent’s learning progress. More recently, Devidze et al. [ 5 ] proposed metrics to evaluate the informativeness of reward signals during training. These approaches recognize the limitations of fixed auxiliary rewards but typically do not adapt rewards based on the agent’s current capabilities as RTW does. Several recent studies have explored adaptive rewards in robotic RL. Gupta et al. [ 23 ] investigated how different reward formulations affect robotic manipulation learning, showing that reward design choices significantly impact training dynamics. Riedmiller et al. [ 24 ] introduced a learning from play framework that separates reward specification from policy learning, allowing more natural skill acquisition. In the context of mobile robots, Xu et al. [ 10 ]