Open-World Skill Discovery from Unsegmented Demonstration Videos
Jingwen Deng, Zihao Wang, Shaofei Cai, Anji Liu, Yitao Liang · 2025
Read Open-World Skill Discovery from Unsegmented Demonstration Videos by Jingwen Deng, Zihao Wang, Shaofei Cai, Anji Liu, Yitao Liang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-10684.txt.
Introduction
Jingwen Deng and co-authors teach agents to find skills inside long demo videos in a 2025 paper. Open-world agents must chain basic skills to handle varied tasks. Online demos are long and uncut, with no labels marking where one skill ends and another starts.
Their self-supervised method slices video into segments that stay true to one skill each. A boundary detector borrows from how human minds split events into parts. It watches for spikes in prediction error from a pretrained action model. Sudden error marks a likely skill change, no hand labels needed. Clean segments can then train agents that learn from the open web.
Catalog record
| CREATOR | Jingwen Deng, Zihao Wang, Shaofei Cai, Anji Liu, Yitao Liang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-10684
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Open-World Skill Discovery from Unsegmented Demonstration Videos Jingwen Deng, Zihao Wang, Shaofei Cai, Anji Liu, Yitao Liang Abstract Learning skills in open-world environments is essential for developing agents capable of handling a variety of tasks by combining basic skills. Online demonstration videos are typically long but unsegmented, making them difficult to segment and label with skill identifiers. Unlike existing methods that rely on random splitting or human labeling, we have developed a self-supervised learning-based approach to segment these long videos into a series of semantic-aware and skill-consistent segments. Drawing inspiration from human cognitive event segmentation theory, we introduce Skill Boundary Detection (SBD), an annotation-free temporal video segmentation algorithm. SBD detects skill boundaries in a video by leveraging prediction errors from a pretrained unconditional action-prediction model. This approach is based on the assumption that a significant increase in prediction error indicates a shift in the skill being executed. We evaluated our method in Minecraft, a rich open-world simulator with extensive gameplay videos available online. The SBD-generated segments yielded relative performance improvements of 63.7% and 52.1% for conditioned policies on short-term atomic tasks, and 11.3% and 20.8% for their corresponding hierarchical agents on long-horizon tasks, compared to random segmented baselines. Our method can leverage the diverse YouTube videos to train instruction-following agents. The project page is at https://craftjarvis.github.io/SkillDiscovery/ . † † footnotetext: Preprint version of the paper accepted at ICCV 2025. 1 Introduction Method Illustration Type Random splitting rule-based Reward-driven rule-based Top-down rule-based Bottom-up rule-based Ours (SBD) learning-based Table 1: Comparisons between existing segmentation methods and our method SBD. Existing methods usually rely on human-designed rules, while our method is learning-based. Random splitting can result in a single skill spanning different segments or multiple skills located within one segment. Reward-driven methods require additional reward information, which is challenging for human annotators to label. Top-down methods often result in limited skill diversity and high computation costs. Bottom-up methods struggle in visually partially observable environments. are visual observations and are actions. Most existing LLM-based instruction-following agents adopt a two-tier architecture of a planner and a controller. The planner decomposes high-level instructions into atomic skills, which the controller then maps to actions based on current observations for interaction with the environment [ 40 , 38 , 39 , 26 , 36 , 9 ] . Training such agents typically involves segmenting long trajectories into atomic skill sequences, followed by separate training of the planner and controller. Learning skills from long-sequence videos is critical for building such hierarchical agents. However, real-world video data are often long-horizon, unsegmented, and with no fine-grained skill labels. In addition, the concept of a “skill” is ill-defined and varies widely across domains such as video gaming [ 6 , 36 , 19 , 46 ] , robotic control [ 53 ] , and autonomous driving [ 37 ] . This ambiguity, along with the immense diversity of skills in open worlds, makes skill learning particularly challenging, especially in partially observable settings. To enable open-world skill learning from unsegmented demonstration videos, the first challenge is to segment them into semantically meaningful, self-contained skills. We list the existing segmentation methods in Tab. 1 . The naive random splitting methods [ 6 , 27 ] divide videos into segments of predefined lengths (e.g., fixed length, uniform distribution). However, they do not ensure that each segment contains a distinct skill. Additionally, predefined lengths may not match the actual distribution of skill length in real world (see Sec. 4.4 ). Reward-driven methods [ 34 ] discover skills through the environment’s reward signal. They cannot capture skills with no associated rewards and may split one skill into multiple segments when rewards are repeatedly gained during execution. Top-down methods [ 32 , 4 ] rely on predefined skill sets from human experts. They use manual labeling or supervised learning to segment videos. Although this method can produce reasonable results, it is expensive and limited by the narrow range of predefined skills. Bottom-up methods [ 52 , 29 ] use algorithms such as agglomerative clustering or byte-pair encoding [ 16 ] to split action sequences. However, they struggle in visually partially observable settings where both observations and actions must be considered. All the methods above rely on human-designed rules to segment the videos, which highlights the need for a learning-based method that can adaptively segment skills from unsegmented videos in open world. Inspired by event segmentation theory (EST) [ 44 ] , which proposes that humans naturally split continuous experiences into discrete events when prediction errors in perceptual expectations rise, we introduce Skill Boundary Detection (SBD)—a method for automatically identifying potential skill boundaries in long trajectories. SBD employs a predictive model trained on a dataset of unsegmented videos to predict future actions based on past observations, effectively capturing temporal dependencies [ 3 ] . We then use this pretrained model to make predictions on unsegmented videos and compare them to the ground-truth actions. A significant increase in prediction error indicates a shift in the skill being executed ( Theorem 3.4 ), enabling us to detect boundaries between different skills. SBD relies on self-supervised learning, removing the need for additional human labeling. This allows SBD to utilize a wide range of YouTube videos to train instruction-following agents. We evaluate SBD in Minecraft [ 18 , 15 , 28 ] , a rich open world simulator. First, we apply SBD to create a segmented Minecraft video dataset. We then re-train a video-conditioned policy [ 6 ] and a language-conditioned policy [ 27 ] on this dataset. We evaluate them on a diverse Minecraft skills benchmark [ 28 ] . The improved versions of these policies result in relative performance increases of 63.7% and 52.1% compared to the original ones, respectively. We also test their corresponding hierarchical agents that use behavioral cloning and in-context learning, achieving relative performance increases of 11.3% and 20.8%, respectively. These findings show the effectiveness of our proposed method for skill discovery from unsegmented demonstrations and its potential to advance open-world skill learning. Figure 1 : Pipeline of our method SBD for discovering skills from unsegmented demonstration videos. Stage I: An unconditional Transformer-XL based policy model [ 11 , 3 ] is pretrained on an unsegmented dataset to predict future actions (labeled by an inverse dynamics model) based on past observations using behavioral cloning. Stage II: The pretrained unconditional policy will produce a high predicted action loss when encountering uncertain observations (e.g., deciding whether to kill a new sheep) in open worlds. These timesteps should be marked as skill boundaries, indicating the need for additional instructions to control behaviors. We segment the long unsegmented videos into a series of short atomic skill demonstrations. Stage III: We train a conditional Transformer-XL based policy model on the segmented dataset to master a variety of atomic skills. Stage IV: Finally, we use hierarchical methods (a combination of vision-language models and the conditional policy) to model the long demonstration videos and follow long-horizon instructions. 2 Problem Formulation Skills in Behavioral Cloning Behavioral cloning learns a policy \pi:\mathcal{O}\rightarrow\mathcal{A} from expert demonstrations, mapping observations to actions based on pairs (o_{i},a_{i})_{i=1}^{T} . In partially observable settings, the policy typically takes the observation history o_{1:t} as input. To enable goal-directed behavior, recent works [ 22 ,