Active Robot Curriculum Learning from Online Human Demonstrations

Muhan Hou, Koen Hindriks, A.E. Eiben, Kim Baraka · 2025

Read Active Robot Curriculum Learning from Online Human Demonstrations by Muhan Hou, Koen Hindriks, A.E. Eiben, Kim Baraka (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-02277.txt.

Introduction

Muhan Hou and three co-authors present a 2025 arXiv method for robots that learn from human demonstrations without exhausting the teacher. Robots that ask for help usually jump between unrelated situations, confusing demonstrators and inviting mistakes. Their method instead sequences its requests like a curriculum, moving from easy situations to harder ones.

Across four simulated tasks with sparse rewards, the curriculum approach produced higher final success and learned faster than three rival methods. A user study with twenty-six people sharpened the point: guided demonstrators finished faster and failed fewer demonstrations. After the guidance ended, they also taught better in both familiar and brand-new situations. The authors argue that how a robot asks shapes how well humans teach, so active learning should be designed as much around the teacher as around the machine.

Catalog record

CREATORMuhan Hou, Koen Hindriks, A.E. Eiben, Kim Baraka
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-02277

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Active Robot Curriculum Learning from Online Human Demonstrations Muhan Hou, Koen Hindriks, A.E. Eiben, Kim Baraka Abstract Learning from Demonstrations (LfD) allows robots to learn skills from human users, but its effectiveness can suffer due to sub-optimal teaching, especially from untrained demonstrators. Active LfD aims to improve this by letting robots actively request demonstrations to enhance learning. However, this may lead to frequent context switches between various task situations, increasing the human cognitive load and introducing errors to demonstrations. Moreover, few prior studies in active LfD have examined how these active query strategies may impact human teaching in aspects beyond user experience, which can be crucial for developing algorithms that benefit both robot learning and human teaching. To tackle these challenges, we propose an active LfD method that optimizes the query sequence of online human demonstrations via Curriculum Learning (CL), where demonstrators are guided to provide demonstrations in situations of gradually increasing difficulty. We evaluate our method across four simulated robotic tasks with sparse rewards and conduct a user study ( N = 26 N=26 ) to investigate the influence of active LfD methods on human teaching regarding teaching performance, post-guidance teaching adaptivity, and teaching transferability. Our results show that our method significantly improves learning performance compared to three other LfD baselines in terms of the final success rate of the converged policy and sample efficiency. Additionally, results from our user study indicate that our method significantly reduces the time required from human demonstrators and decreases failed demonstration attempts. It also enhances post-guidance human teaching in both seen and unseen scenarios compared to another active LfD baseline, indicating enhanced teaching performance, greater post-guidance teaching adaptivity, and better teaching transferability achieved by our method. I Introduction Learning from Demonstrations (LfD) is a widely used approach that allows robots to acquire task skills by imitating humans perform these tasks. Traditional LfD methods [ 1 , 2 , 3 , 4 ] , however, rely on offline demonstrations collected prior to training. When these demonstrations are selected and provided by everyday human users, the robot learning may suffer from sub-optimal human teaching (e.g., providing demonstrations with a biased distribution) [ 5 ] . To alleviate this issue, recent approaches [ 6 , 7 , 8 ] have enabled robots to actively query for online demonstrations during the learning process, leading to improved policy performance and sample efficiency. Fig. 1 : Overview of our method. We present an active LfD algorithm that optimizes the query sequence of online human demonstrations via Curriculum Learning (CL), where human demonstrators are guided to provide online demonstrations in situations of gradually increasing difficulty. Although active LfD approaches improve learning performance, they come at the cost of greater human effort. Many of these methods require resetting the environment to specific states whenever the learning agent encounters uncertainty [ 7 , 8 , 9 , 10 , 11 ] , prompting the human to assist with the next steps. This process results in frequent and abrupt context switches across different areas of the task space, imposing significant cognitive load on the human demonstrator and potentially introducing errors or noise into the demonstrations [ 6 ] . Moreover, few prior studies in active LfD have examined how these active query strategies may impact human teaching beyond user experience. Important factors, such as shifts in teaching strategies after robot guidance and the ability of human teachers to transfer guidance to unseen scenarios, have been underexamined, yet they are crucial for developing algorithms that benefit both robot learning and human teaching. Aiming to benefit both robot learning and human teaching, we present an active LfD algorithm that leverages online human demonstrations for policy updates. More specifically, our work is inspired by Curriculum Learning (CL) [ 12 ] , the general idea of which is to break the original hard task into a sequence of sub-tasks with gradually increasing difficulties. CL situates the learning agent in the next sub-task only after it masters the simpler ones, proven to be able to improve learning performance [ 13 , 14 , 15 , 16 , 17 , 18 ] . This scaffolding process is intuitive to common human users as it well aligns with how humans learn to perform a new task [ 19 , 20 ] . Furthermore, the gradual progression between related sub-tasks may reduce the human effort required to adapt to context switches, as robot queries are presented in scenarios that are similar to one another but incrementally more challenging. Building on these insights, we design an active LfD algorithm (Fig. 1 ) that optimizes the sequence of demonstration queries with gradually increasing difficulty. These queries not only gather demonstrations for the robot to imitate but also automate the curriculum expansion and iteration process, guiding policy exploration. We evaluate our method on four simulated robotic tasks with sparse rewards and conduct a user study to investigate how our method may benefit human teaching. Compared to three other LfD baselines, our method outperformed them in all four tasks in terms of converging policy performance and sample efficiency. Regarding human teaching, compared with another active LfD baseline, our method takes significantly less human time with much fewer failed demonstration attempts during demonstration collection, and achieves much better post-guidance human teaching in both seen and unseen tasks. These results indicate the improved per-guidance teaching performance, post-guidance teaching adaptivity, and teaching transferability of human teaching achieved by our method. To summarize, the main contributions of our work are as follows: • An active LfD algorithm that leverages Curriculum Learning to optimize the query sequence of online human demonstrations, leading to significant improvements in learning performance on various sparse-reward robotic tasks. • A user study that investigates how human teaching, including per-guidance teaching performance, post-guidance teaching adaptivity, and teaching transferability, can be influenced by active LfD algorithms. II Related Work II-A Curriculum Learning with Demonstrations Curriculum Learning (CL) [ 12 ] is a method that breaks down the original task into a sequence of subtasks of increasing difficulty and presents them to the learning agent incrementally once it has mastered the simpler ones. This scaffolding process has been shown to improve learning performance [ 12 ] , especially for robotic tasks with sparse rewards [ 13 , 14 , 15 , 16 ] . To further automate curriculum design, efforts have been made to utilize offline demonstrations to guide curriculum iteration. For instance, [ 17 ] leverages offline expert demonstrations to divide the task space into discrete stages, iteratively resetting the environment to the area near the goal space and progressively exploring backward to the initial state space. Similarly, [ 21 ] employs a reverse curriculum strategy and enhances the curriculum update rules by incorporating simulation environment parameters to improve adaptability for sim-to-real applications. Additionally, [ 18 ] combines the reverse curriculum with the forward curriculum after the agent masters the task for all the states encountered during the demonstrations, significantly improving sample efficiency with much fewer demonstrations. Beyond reverse curriculum strategies, some approaches [ 22 , 23 ] propose ranking-based methods to iteratively select demonstrations from an offline pool for policy updates. However, most of these approaches rely on offline demonstrations to construct curricula, assuming the distribution

Read the complete text