Using Non-Expert Data to Robustify Imitation Learning via Offline Reinforcement Learning

Kevin Huang, Rosario Scalise, Cleah Winston, Ayush Agrawal, Yunchu Zhang, Rohan Baijal, Markus Grotz, Byron Boots, Benjamin Burchfiel, Masha Itkina, Paarth Shah, Abhishek Gupta · 2025

Read Using Non-Expert Data to Robustify Imitation Learning via Offline Reinforcement Learning by Kevin Huang, Rosario Scalise, Cleah Winston, Ayush Agrawal, Yunchu Zhang, Rohan Baijal, Markus Grotz, Byron Boots, Benjamin Burchfiel, Masha Itkina, Paarth Shah, Abhishek Gupta (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19495.txt.

Introduction

Kevin Huang and eleven co-authors show how messy, non-expert robot data can strengthen imitation learning instead of being thrown away. Imitation learning trains robots from expert demonstrations, but expert data is costly and covers few situations. Play data, partial attempts, and rollouts from weak policies are cheaper and broader, yet standard imitation methods cannot use them well.

The authors argue that offline reinforcement learning, with the right design choices, can unlock that data. Simple changes that widen the range of actions the policy considers let the combined method recover from mistakes and generalize better. In manipulation tasks, adding non-expert data widened the set of starting conditions the robots could handle, using even partial or flawed demonstrations.

Catalog record

CREATORKevin Huang, Rosario Scalise, Cleah Winston, Ayush Agrawal, Yunchu Zhang, Rohan Baijal, Markus Grotz, Byron Boots, Benjamin Burchfiel, Masha Itkina, Paarth Shah, Abhishek Gupta
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-19495

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Using Non-Expert Data to Robustify Imitation Learning via Offline Reinforcement Learning Kevin Huang, Rosario Scalise, Cleah Winston, Ayush Agrawal, Yunchu Zhang, Rohan Baijal, Markus Grotz, Byron Boots, Benjamin Burchfiel, Masha Itkina, Paarth Shah, Abhishek Gupta Abstract Imitation learning has proven effective for training robots to perform complex tasks from expert human demonstrations. However, it remains limited by its reliance on high-quality, task-specific data, restricting adaptability to the diverse range of real-world object configurations and scenarios. In contrast, non-expert data—such as play data, suboptimal demonstrations, partial task completions, or rollouts from suboptimal policies—can offer broader coverage and lower collection costs. However, conventional imitation learning approaches fail to utilize this data effectively. To address these challenges, we posit that with right design decisions, offline reinforcement learning can be used as a tool to harness non-expert data to enhance the performance of imitation learning policies. We show that while standard offline RL approaches can be ineffective at actually leveraging non-expert data under the sparse data coverage settings typically encountered in the real world, simple algorithmic modifications can allow for the utilization of this data, without significant additional assumptions. Our approach shows that broadening the support of the policy distribution can allow imitation algorithms augmented by offline RL to solve tasks robustly, showing considerably enhanced recovery and generalization behavior. In manipulation tasks, these innovations significantly increase the range of initial conditions where learned policies are successful when non-expert data is incorporated. Moreover, we show that these methods are able to leverage all collected data, including partial or suboptimal demonstrations, to bolster task-directed policy performance. This underscores the importance of algorithmic techniques for using non-expert data for robust policy learning in robotics. Paper website: https://uwrobotlearning.github.io/RISE-offline/ I Introduction Imitation learning has enabled remarkable progress in robot learning, training reactive closed-loop policies from high-quality demonstrations. These methods typically perform supervised learning using expressive policy classes such as diffusion models parameterized with large neural networks [ 1 , 2 , 3 ] . Despite impressive performance under conditions similar to the training distribution, these policies can be quite brittle beyond this setting. They show vulnerability to out-of-distribution (OOD) scenarios, where even minor deviations in object configurations or environmental conditions can lead to failure [ 4 ] . A natural way to address policy fragility is to simply collect more expert data, broadening the coverage of expert demonstrations. The challenge is that collecting such data can be expensive and scale poorly, requiring an impractical amount of data collection to cover combinatorial scene conditions. As a result, policies trained with standard imitation learning can struggle to handle real-world variability on deployment. The formulation of imitation learning through supervised learning requires (near) optimal task demonstrations, which have to be carefully curated per task and can be difficult to collect at scale, especially for high precision or high dexterity problems. In most large-scale data collection efforts, not all data satisfies these criteria. This results in a significant fraction of collected data being discarded through the process of data curation/filtering [ 5 , 6 , 7 ] , despite this data containing useful information about the dynamics of the world. This begs the question - can cheaper sources of non-optimal data beyond expert, task-specific demonstrations be used to improve the performance of imitation learning? In this work, we study how cheaper and typically more abundant data sources such as undirected play data, unsuccessful/partial demonstrations (whether from a human demonstrator or policy rollouts), or data from other tasks can be made useful for improving the robustness of imitation learning. Fig. 1 : RISE enables non-expert data to be stitched with alongside expert data to provide robust, high-coverage behavior for robotic manipulation. This allows for all collected data to be used to allow the policy to recover from novel OOD states. We study this problem through the lens of offline reinforcement learning (RL). Typical offline RL algorithms [ 8 , 9 , 10 ] treat all data (whether it be optimal or suboptimal) as arbitrary off-policy data and synthesize optimal policies through reward-based, temporal-difference learning [ 11 ] . Directly applying standard offline RL methods for leveraging suboptimal data can become challenging for real-world problems, since reward can be difficult to specify without considerable domain knowledge or privileged information. In this work, we propose an alternative, instantiating an offline RL algorithm that can learn from simple binary rewards, 1 for optimal data and 0 for suboptimal data, which collected demonstrations often naturally categorize into. This allows the learning algorithm to make use of the suboptimal data to learn how to recover the system back to states in the expert state distribution, while replicating optimal behavior on these expert states. This naturally robustifies the policy to solve tasks from a diversity of states beyond a narrow range of expert states, while requiring minimal infrastructural modifications and assumptions beyond that of typical imitation learning. While offline RL via dynamic programming can in principle “stitch” together useful segments from suboptimal data for data-efficient recovery, we find that practical instantiations of offline RL methods in high-dimensional state-action spaces can fail to demonstrate this stitching capability without an impractically high degree of data coverage. To address the challenges resulting from the lack of data coverage, we introduce a notion of “fuzziness” into the state representation. Specifically, we enforcing a notion of local smoothness on the policy via Lipschitz continuity. For recoverable OOD states, doing so significantly improves the policy’s ability to “stitch” offline data. This enables suboptimal data to easily be used for improving the robustness of imitation learning, even in low data coverage regimes. We make the following contributions - 1) we introduce an offline RL framework for leveraging non-expert data to robustify imitation learning policies, 2) we show the pitfalls of standard offline RL in the low data regime, and introduce the Robust Imitation by Stitching from Experts ( RISE ) algorithm, to improve trajectory stitching 3) We show that RISE is effective across various types of non-optimal data - ranging from undirected play data to suboptimal demonstrations or policy evaluation rollouts, and even multitask datasets, 4) We demonstrate the efficacy of RISE on various tabletop manipulation tasks in simulation and furniture assembly tasks on a real robot. II Related Work Imitation Learning: Imitation learning methods aim to learn closed loop policies from near optimal demonstration data. This is a well studied field, with a plethora of work [ 12 , 13 , 14 , 15 ] on methods and applications. Work in imitation learning has primarily focused on either dealing with compounding error [ 16 , 17 , 18 , 19 , 20 ] , incorporating richer generative distributions [ 1 , 21 , 22 ] or using robust policy backbones [ 23 , 24 , 25 , 26 ] . In this work, we show that in addition to expert demonstrations, non-expert data can be leveraged to robustify imitation learning. Offline Reinforcement Learning: Offline reinforcement learning is a closely related subarea of research, where pre-collected off-policy datasets are used to synthesize task-directed behavior [ 27 ] . These methods do not typically assume that pre-collected data is optimal, instead using rewards to infer which behaviors are optimal from offline datasets. Offline RL methods come in many

Read the complete text