MILE: Model-based Intervention Learning

Yigit Korkmaz, Erdem Bıyık · 2025

Read MILE: Model-based Intervention Learning by Yigit Korkmaz, Erdem Bıyık (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-13519.txt.

Introduction

Yigit Korkmaz and Erdem Bıyık present a learning method that uses expert interventions in this 2025 paper. Robots often learn by copying experts, but copied policies drift into errors, and full expert demos are costly. In interactive setups, an expert watches and takes over when needed, yet most methods learn only from the takeover moments.

The authors model how interventions happen and show that the quiet moments, when the expert lets the robot act, also carry signal about state quality and action choice. Their method learns a policy from just a handful of interventions. Tests covered simulated tasks, a real robot arm task, and a study with human subjects.

Catalog record

CREATORYigit Korkmaz, Erdem Bıyık
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-13519

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MILE: Model-based Intervention Learning Yigit Korkmaz, Erdem Bıyık Abstract Imitation learning techniques have been shown to be highly effective in real-world control scenarios, such as robotics. However, these approaches not only suffer from compounding error issues but also require human experts to provide complete trajectories. Although there exist interactive methods where an expert oversees the robot and intervenes if needed, these extensions usually only utilize the data collected during intervention periods and ignore the feedback signal hidden in non-intervention timesteps. In this work, we create a model to formulate how the interventions occur in such cases, and show that it is possible to learn a policy with just a handful of expert interventions. Our key insight is that it is possible to get crucial information about the quality of the current state and the optimality of the chosen action from expert feedback, regardless of the presence or the absence of intervention. We evaluate our method on various discrete and continuous simulation environments, a real-world robotic manipulation task, as well as a human subject study. Videos and the code can be found at https://liralab.usc.edu/mile . I Introduction Imagine training a household robot to help users place the dishes in the dishwasher. One way to do this is to use reinforcement learning (RL) that has been proven successful in several areas ranging from gaming to dialogue systems and autonomous driving [ 1 , 2 , 3 , 4 ] . However, its need for lots of online interactions with the environment as well as a well-defined reward function make it unsuitable in a real-world situation like this. An alternative is to use imitation learning (IL) where an expert provides demonstrations of how to place the dishes. IL requires fewer interactions in the world than RL and does not require a reward function. A common drawback of this approach is the compounding distributional shift, which results from the accumulation of errors when deploying a learned policy [ 5 ] : the small inaccuracies in the robot’s learned policy will move it to an out-of-distribution state where the policy may fail more significantly, which may result in the robot breaking the dishes. Interactive learning methods try to overcome this compounding errors problem by iteratively querying the expert with system states, and fine-tuning the policy based on the expert actions [ 6 , 7 , 8 , 9 , 10 , 11 ] . Most of these methods do not let the human intervene at will but transfer the control to the human according to some criterion [ 8 , 9 ] . In others, the human can take over the control at any timestep [ 7 , 12 , 13 , 14 ] . Going back to the running example, assume the robot had a mediocre policy in the beginning, thanks to some initial training done in the factory setup. In this interactive scheme, we would control the robot only when we think it is doing or about to do something wrong. This is clearly more convenient than operating the robot for long periods to generate full demonstrations. But what about the intervals where we did not input any actions? Did we not provide any information to the robot? The answer is no: the fact that we chose to not intervene means the robot’s actions were already good enough in those intervals. This is what the existing interactive learning techniques are missing: even though they try to improve the robot’s policy based on the states where the expert intervenes, they do not utilize any structure of how or when those interventions occur, ignoring an important feedback signal that is leaking through the states where the expert does not intervene. To efficiently use the information in both the states with and without human interventions, we argue one must understand and utilize the structure behind how interventions occur. To this end, we make the following contributions: 1. We propose a novel model that is fully differentiable to formulate how and when an expert intervenes. 2. We utilize this intervention model to fine-tune a weak policy and evaluate our method in various simulated and real-world environments to prove its effectiveness. 3. We compare our method against the state-of-the-art baselines which utilize interventions to show its higher sample-efficiency and performance. Fig. 1: Overall MILE System. A human operator oversees the robot during task execution and may decide to take over control at any timestep. The human makes the decision to intervene or not based on their prediction of the robot’s potential failure, without observing the robot’s action in that particular state. During the data collection phase, all interactions with the environment are recorded, both with and without interventions. The policy is then trained on this dataset using an iterative process that incorporates our novel intervention model. II Related Work Interactive Imitation Learning. In imitation learning, expert data is used to train a policy in a supervised way [ 15 , 16 , 17 , 18 , 19 , 20 ] . Interactive imitation learning methods try to overcome its compounding errors problem [ 5 , 6 ] by querying the expert on the learned policy’s rollouts [ 15 , 7 , 8 , 6 , 9 , 21 ] . While the expert relabels either the whole trajectory of the policy with actions, or some parts of it that are automatically selected by estimating various task performance quantities [ 9 , 6 ] , there are also works where the expert has the freedom to intervene at will [ 7 , 12 , 13 , 22 ] . Some of these methods consider the implicit feedback coming from the states where the expert chooses to not intervene. They attempt to incorporate the information leaking from non-intervention intervals either by enforcing those state-action pairs to be constrained in the action-value cost functions [ 14 ] or utilize weighted behavioral cloning (BC) to incorporate the signal from non-interventions, with different heuristics used for assigning weights to on-policy robot samples, and expert human interventions [ 12 , 13 , 22 ] . However, none of these algorithms uses a model to understand and learn from why the expert chooses to not intervene. In our method, we propose a model that attempts to capture how the interventions occur, and how satisfied the user is with the current action in the case of not intervening. Moreover, we investigate how we can efficiently use the information coming from that model for training. To our knowledge, this is the first work that models how human interventions happen in robotics while utilizing it for better policy learning. Reinforcement Learning with Expert Data. Many studies focus on integrating expert demonstrations into RL. This is often achieved by populating the replay buffer to warm-start the initial policy and guide its exploration in a favorable direction [ 23 , 24 , 25 , 26 ] . There are also approaches that incorporate interventions into RL. One approach introduces an extra cost term in the learning objective to minimize interventions [ 27 ] . Another recent method relabels the rewards based on interventions, assigning negative rewards at intervention states, while giving a zero reward to all other states [ 28

Read the complete text