Learning from Demonstrations via Capability-Aware Goal Sampling
Yuanlin Duan, Yuning Wang, Wenjie Qiu, He Zhu · 2026
Read Learning from Demonstrations via Capability-Aware Goal Sampling by Yuanlin Duan, Yuning Wang, Wenjie Qiu, He Zhu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-08731.txt.
Introduction
Yuanlin Duan and three co-authors work on robots that learn by watching a skilled user. Pure copy work breaks down on long tasks. One small slip can grow, and the robot soon drifts far from the shown path. The team wants a way to learn that can bend when the task is long.
Their method is called Cago. It picks goals that fit what the robot can do now, not just goals from the demo. The robot aims for those goals and learns from each try. In this way it is less tied to one fixed path. A reader who trains robots gets a plan that may hold up on long jobs where plain copy work fails.
Catalog record
| CREATOR | Yuanlin Duan, Yuning Wang, Wenjie Qiu, He Zhu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 105 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-08731
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning from Demonstrations via Capability-Aware Goal Sampling Yuanlin Duan, Yuning Wang, Wenjie Qiu, He Zhu Abstract Despite its promise, imitation learning often fails in long-horizon environments where perfect replication of demonstrations is unrealistic and small errors can accumulate catastrophically. We introduce Cago (Capability-Aware Goal Sampling), a novel learning-from-demonstrations method that mitigates the brittle dependence on expert trajectories for direct imitation. Unlike prior methods that rely on demonstrations only for policy initialization or reward shaping, Cago dynamically tracks the agent’s competence along expert trajectories and uses this signal to select intermediate steps—goals that are just beyond the agent’s current reach—to guide learning. This results in an adaptive curriculum that enables steady progress toward solving the full task. Empirical results demonstrate that Cago significantly improves sample efficiency and final performance across a range of sparse-reward, goal-conditioned tasks, consistently outperforming existing learning from-demonstrations baselines. 1 Introduction Imitation Learning (IL) provides a powerful paradigm for training agents using expert demonstrations, effectively alleviating the exploration challenges common in Deep Reinforcement Learning (DRL) ( 2 ) . The simplest form of IL is Behavior Cloning (BC), which directly supervises policy actions based on the states visited by the expert ( 3 ; 50 ) . However, BC often suffers from compounding errors when the learned policy deviates from expert trajectories. To overcome this limitation, modern approaches such as GAIL ( 25 ) , PWIL ( 9 ) , and AdRIL ( 15 ) seek to align the state–action distributions of the agent and the expert through adversarial or distribution-matching objectives. In parallel, Inverse Reinforcement Learning (IRL) methods ( 56 ) aim to infer underlying reward functions from demonstrations, which can then guide reinforcement learning. More recently, advances in offline and offline-to-online RL, such as CQL ( 33 ) and Cal-QL ( 37 ) , integrate demonstrations as anchors to regularize policy learning. These methods penalize value estimates that diverge from demonstrated behavior, mitigating overestimation and instability caused by out-of-distribution actions. However, existing IL methods often struggle with complex, long-horizon tasks because they fail to reason about which parts of the task the agent has already mastered and which remain challenging. In particular, distribution-matching approaches perform flat matching—attempting to align occupancy measures over the entire trajectory distribution without considering the agent’s evolving capabilities. This leads to poor exploration guidance, especially in the early stages of training when the agent seldom reaches meaningful parts of the state space. As a result, the learned reward function tends to assign uniformly low rewards, yielding uninformative gradients and hindering effective policy improvement. Some prior work proposes demonstration-guided curriculum learning that trains agents to solve tasks by starting near the goal or high-reward states and gradually expanding to earlier parts of the trajectories ( 44 ; 45 ; 48 ) . However, these approaches rely on the ability to reset the agent to arbitrary demonstration states—an assumption impractical in real-world settings due to challenges in replicating physical conditions like joint velocities and angular momentum. We propose Cago (Capability-Aware Goal Sampling), a new learning-from-demonstrations framework that explicitly aligns the agent’s learning process with its evolving capabilities. Unlike prior methods that use demonstrations for direct imitation, reward shaping, or offline pretraining, Cago treats demonstrations as structured roadmaps. It continuously monitors which parts of a demonstration the agent can already reach and leverages this signal to sample intermediate goal states in the demonstration, those at the boudary of the agent’s current goal-reaching capabilities. At each episode, a goal-conditioned agent ( 34 ; 40 ) first attempts to reach the sampled goal and then explores forward from it, generating informative, task-relevant data for policy optimization. This iterative process of capability-aware goal selection and curriculum-aligned exploration enables steadily progress toward solving the full task. We evaluate Cago across several sparse-reward environments and demonstrate substantial improvements in both sample efficiency and final task performance over existing imitation-based baselines. Our experiments highlight that capability-aware goal sampling provides a powerful signal for structuring learning, particularly in long-horizon tasks. 2 Background and Problem Setup Reinforcement learning (RL) aims to enable agents to learn optimal behaviors through trial-and-error interactions with an environment. An RL problem is formulated as a Markov Decision Process (MDP), represented as a tuple (\mathcal{S},\mathcal{A},\mathcal{T},\mathcal{G},\eta,R,\rho_{0}) . The agent operates within a state space \mathcal{S} and takes actions from an action space \mathcal{A} , transitioning between states according to the dynamics \mathcal{T}(s^{\prime}|s,a) . R(s,a)\in\mathbb{R} is the reward function and \rho_{0} is the initial state distribution. Given a policy \pi , consider the trajectory \tau=\{s_{0},a_{0},s_{1},a_{1},\ldots\} sampled by \pi , i.e., s_{0}\sim\rho_{0} , a_{t}\sim\pi(\cdot|s_{t}) , and s_{t+1}\sim\mathcal{T}(\cdot|s_{t},a_{t}) . The goal of RL is to learn a return-maximizing policy \pi^{*}=\arg\max_{\pi}\mathbb{E}_{\tau\sim\pi(a_{t}\mid s_{t})}\left[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t})\right] where \gamma\in(0,1] is the discount factor. Learning from Demonstrations. In imitation learning, the agent is provided a dataset of demonstrations \mathcal{D}_{\text{demo}} collected from some expert policy \pi_{\text{expert}} . The objective is to learn a policy \pi that reproduces the expert’s behavior by generalizing from these demonstrations. The simplest approach, behavioral cloning (BC), treats this as a supervised learning problem, minimizing the discrepancy between the agent’s predicted actions and the expert’s. Another line of work, inverse reinforcement learning (IRL) ( 1 ) , aims to infer an underlying reward function that explains the expert’s behavior and then optimizes a policy through RL on this learned reward, thus decoupling reward inference from policy optimization. Building on ideas from IRL, Generative Adversarial Imitation Learning (GAIL) ( 25 ) bypasses explicit reward recovery by training a policy and a discriminator in an adversarial game: the discriminator distinguishes expert from agent trajectories, while its output serves as an implicit, learned reward signal guiding the policy. More broadly, many recent imitation learning algorithms can be interpreted as minimizing a divergence between the expert and agent occupancy measures. State Reset. Several methods attempt to mitigate the exploration challenge in sparse-reward RL environments by resetting the agent to states from expert demonstrations, thereby bypassing the need to discover those states through the agent’s own exploration. These strategies include initializing the agent to states sampled uniformly from demonstration trajectories ( 36 ; 39 ; 26 ) , employing a hand-crafted curriculum ( 55 ) , or using a reverse curriculum that progressively trains the agent from goal or high-reward states backward ( 44 ; 45 ; 48 ) . These approaches assume the ability to reset the agent to arbitrary demonstration states—an assumption that is unrealistic in real-world settings. Goal-Conditioned RL (GCRL) extends the standard RL framework by conditioning policies on specific target goals, guiding agents toward desired goals. The MDPs are augmented with a goal space \mathcal{G} and are associated with states via a mapping \eta:\mathcal{S}\to\mathcal{G} , ensuring that each state corresponds to an achieved goal. In GCRL, the reward signal from the environment is typically sparse and is defined as: R(s,a,s^{\prime},g)=1\{\eta(s^{\prime})=g\} . We assume that each episode has a fixed horizon T and \mathcal{S}=\mathcal{G} . The agent’s objective is to train a goal-conditioned policy \pi^{G}(\cdot|s_{t},g) to achieve a given goal g\in\mathcal{G} through maximizing the expected cumulative reward J(\pi)=\mathbb{E}_{g\sim p_{g},\tau\sim\pi^{G}(a_{t}\mid s_{t},g)}\left[\sum_{t=0}^{T-1}\gamma^{t}\cdot R(s_{t},a_{t},s_{t+1},g)\right] where p_{g} is the goal distribution. This Paper. We introduce Cago, a novel approach that leverages demonstrations as a scaffold for goal-directed reinforcement learning. Rather than direct imitation, Cago uses demonstrations to guide exploration by training a goal-conditioned