Robot Policy Transfer with Online Demonstrations: An Active Reinforcement Learning Approach

Muhan Hou, Koen Hindriks, A.E. Eiben, Kim Baraka · 2025

Read Robot Policy Transfer with Online Demonstrations: An Active Reinforcement Learning Approach by Muhan Hou, Koen Hindriks, A.E. Eiben, Kim Baraka (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-12993.txt.

Introduction

A robot that learned one job can borrow the skill for a cousin job, but asking the teacher at the right moments makes the borrowing cheaper. Muhan Hou, Koen Hindriks, A E Eiben, and Kim Baraka add live teaching to policy transfer in a 2025 paper. Most transfer methods use demos gathered before the move starts, which drift out of step with the learner. Their algorithm asks for fresh demos during the move, within a fixed budget.

Eight robot scenarios tested transfers across settings, goals, and body types. The method beat the standard baselines by clear margins, the team reports. Timing and content of each query are chosen by the learner itself. Fixes for the drift problem drive the gains.

Catalog record

CREATORMuhan Hou, Koen Hindriks, A.E. Eiben, Kim Baraka
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-12993

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Robot Policy Transfer with Online Demonstrations: An Active Reinforcement Learning Approach Muhan Hou, Koen Hindriks, A.E. Eiben, Kim Baraka Abstract Transfer Learning (TL) is a powerful tool that enables robots to transfer learned policies across different environments, tasks, or embodiments. To further facilitate this process, efforts have been made to combine it with Learning from Demonstrations (LfD) for more flexible and efficient policy transfer. However, these approaches are almost exclusively limited to offline demonstrations collected before policy transfer starts, which may suffer from the intrinsic issue of covariance shift brought by LfD and harm the performance of policy transfer. Meanwhile, extensive work in the learning-from-scratch setting has shown that online demonstrations can effectively alleviate covariance shift and lead to better policy performance with improved sample efficiency. This work combines these insights to introduce online demonstrations into a policy transfer setting. We present Policy Transfer with Online Demonstrations, an active LfD algorithm for policy transfer that can optimize the timing and content of queries for online episodic expert demonstrations under a limited demonstration budget. We evaluate our method in eight robotic scenarios, involving policy transfer across diverse environment characteristics, task objectives, and robotic embodiments, with the aim to transfer a trained policy from a source task to a related but different target task. The results show that our method significantly outperforms all baselines in terms of average success rate and sample efficiency, compared to two canonical LfD methods with offline demonstrations and one active LfD method with online demonstrations. Additionally, we conduct preliminary sim-to-real tests of the transferred policy on three transfer scenarios in the real-world environment, demonstrating the policy effectiveness on a real robot manipulator. I Introduction Transfer Learning (TL) has recently become increasingly popular in the robotics community. Instead of learning from scratch, TL enables a robot to utilize prior knowledge (e.g., policy) of a source situation to improve learning in a related target situation [ 1 ] . This target situation may differ from the source in various aspects, such as environments [ 2 , 3 ] (e.g., from simulation to real world), tasks [ 4 ] (e.g., grabbing a bottle to grabbing a mug), or even embodiments [ 5 ] (e.g., different robot arms). To facilitate policy transfer, efforts have been made to combine TL with Learning from Demonstrations (LfD). These demonstrations can be utilized to acquire prior skills of the source domain [ 6 , 7 , 8 , 9 , 10 ] and accelerate the policy transfer process to the target domain [ 11 , 4 ] . However, almost all previous work relies on offline demonstrations , which are collected before policy transfer starts. This can lead to the covariance shift [ 12 ] , as the state distribution in the demonstrations differs from that of the ongoing robot policy. This issue can even be further exacerbated by discrepancies in task space between the source and target domains. And to adapt the demonstration distribution to the ongoing policy update, it requires information about the learning process, which is not available a prior and can only be obtained iteratively in an online manner. By contrast, extensive work [ 13 , 14 , 15 ] in the learning-from-scratch setting has shown that online demonstrations , which are collected during the learning process, can effectively alleviate covariance shift and produce better policy performance. Furthermore, by querying for the most suitable demonstrations during the learning process, LfD with online demonstrations can achieve the same level of policy performance with much fewer demonstrations [ 16 ] and improve sample efficiency [ 15 ] . This can be of great importance especially when the number of demonstrations is constrained by time and financial costs (e.g., data collection from human demonstrators). Therefore, we extend this idea to the scenario of policy transfer and ask the question: will online demonstrations benefit policy transfer, and how can we realize that? Fig. 1 : Overview of our method. To this end, we present the first active LfD algorithm for policy transfer that leverages online demonstrations for transfer learning and show that online demonstrations will indeed benefit policy transfer if designed properly. Our method extends the EARLY framework [ 17 ] to the transfer learning scenario and simultaneously optimizes the problems of when to query and what episodic demonstrations of the target task to query during the policy transfer process (see figure 1 ). More specifically, a query strategy module evaluates the trajectory-based uncertainty after each episodic roll-out of the robot policy for the target task. An adaptive threshold of uncertainty is then updated to determine whether a query will be requested. If so, the query strategy module outputs the initial state of one of the recent roll-outs that the robot is most uncertain about and queries an episodic expert demonstration starting from this initial state. We test our approach in 8 robot policy transfer scenarios, involving policy transfer across diverse environment characteristics, task objectives, and robotic embodiments . Compared to EARLY and two other canonical LfD baselines with offline demonstrations, our method achieves significantly higher success rate and sample efficiency in target tasks that the other baseline methods can hardly address. To summarize, our contributions are two-fold: 1. We present an active LfD algorithm for robot policy transfer that effectively transfers policies from source to target tasks across diverse scenarios. 2. We validate the effectiveness of our method through a set of robotic policy transfer scenarios, supported by extensive empirical results from simulations and preliminary sim-to-real experiments in real-world environments. II Related Work Policy Transfer with Demonstrations Policy transfer aims to adapt a policy of a source domain to a target domain that is related but different from the source [ 1 ] . To further facilitate policy transfer, efforts have been made to introduce demonstrations into this process, either from the source domain or the target domain. When these demonstrations are from the source domain, they are commonly used to acquire prior skills of the source domain using different representations [ 6 , 7 , 8 , 9 , 10 ] , whcih can be later fine-tuned for the taregt domain. Additionally, these source domain demonstrations can also be used to infer the dynamics difference between the target and source domain, the knowledge of which can be then utilized to adapt policy to the target domains [ 18 ] . And when these demonstrations are from target domain, they can be utilized to accelerate the policy transfer process [ 11 , 4 ] and better transfer the robot policy from simulation to real-world environments [ 19 ] . However, almost all of these work only use offline demonstrations, which are collected prior to the policy transfer process. By contrast, extensive evidence has shown in the scenario of learning from scratch that online demonstrations can effectively improve sample efficiency and achieve better learning performance by providing the required demonstrations to the learning agent during the training process. Therefore, our work extends this idea to the scenario of policy transfer and presents an active LfD algorithm that queries online demonstrations, achieving better performance in policy transfer with improved sample effiency. Interactive Learning from Demonstrations . To alleviate the issue of covariance shift of conventional Learning

Read the complete text