Unsupervised Hierarchical Skill Discovery
Damion Harvey, Geraud Nangue Tasse, Branden Ingram, Benjamin Rosman, Steven James · 2026
Read Unsupervised Hierarchical Skill Discovery by Damion Harvey, Geraud Nangue Tasse, Branden Ingram, Benjamin Rosman, Steven James (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-23156.txt.
Introduction
Damion Harvey and four co-authors discover reusable skills in behavior data without any labels. Skill discovery splits long trajectories into segments that can be reused as options. Most current methods need action labels, rewards, or hand-made notes, which limits where they apply.
Their unsupervised method finds both the segments and the hierarchy among them. Skills learned from raw footage could let agents reuse past competence on new tasks. That ability matters for robots trained on large, unlabeled logs of experience. No action labels, rewards, or handcrafted annotations are assumed, which is what makes the discovery method unsupervised.
Catalog record
| CREATOR | Damion Harvey, Geraud Nangue Tasse, Branden Ingram, Benjamin Rosman, Steven James |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-23156
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Unsupervised Hierarchical Skill Discovery Damion Harvey, Geraud Nangue Tasse, Branden Ingram, Benjamin Rosman, Steven James Abstract We consider the problem of unsupervised skill segmentation and hierarchical structure discovery in reinforcement learning. While recent approaches have sought to segment trajectories into reusable skills or options, most rely on action labels, rewards, or handcrafted annotations, limiting their applicability. We propose a method that segments unlabelled trajectories into skills and induces a hierarchical structure over them using a grammar-based approach. The resulting hierarchy captures both low-level behaviours and their composition into higher-level skills. We evaluate our approach in high-dimensional, pixel-based environments, including Craftax and the full, unmodified version of Minecraft. Using metrics for skill segmentation, reuse, and hierarchy quality, we find that our method consistently produces more structured and semantically meaningful hierarchies than existing baselines. Furthermore, as a proof of concept for utility, we demonstrate that these discovered hierarchies accelerate and stabilise learning on downstream reinforcement learning tasks. 1 Introduction Human planning operates hierarchically, reasoning in terms of goals and sub-tasks rather than primitive actions ( 5 ; 12 ) . Similarly, reinforcement learning (RL) agents in high-dimensional environments like Minecraft benefit from hierarchical decompositions to improve learning efficiency and policy reuse ( 30 ; 22 ) . While manually defined hierarchies, such as Hierarchical Task Networks (HTNs) ( 7 ) , are effective, they demand significant human effort and domain expertise to create ( 4 ) . Figure 1 : Example of HiSD applied to a Minecraft trajectory. Step 1 : HiSD segments the observational trajectory into distinct skills, such as get wood , craft tools , and gather stone . Step 2 : Using these segmented trajectories, HiSD applies a grammar-based compression algorithm to induce a hierarchy over the discovered skills, revealing reusable subroutines and their temporal organisation. An alternative is learning structure directly from data. Prior approaches explore this via reward signals or action supervision ( 26 ; 16 ) ; however, these methods rely on strong assumptions, such as access to action labels or known skill orderings, and typically produce flat segmentations rather than deep, compositional hierarchies. We propose Hierarchical Skill Discovery (HiSD), a fully unsupervised framework for extracting reusable, multi-level skill hierarchies purely from observational data. Crucially, HiSD decouples structure discovery from policy execution: unlike baselines that require actions during discovery ( 20 ; 16 ) , HiSD operates on observations only, enabling scalability to abundant unlabelled video data ( 3 ) . Our method combines temporal action segmentation (TAS) to identify latent skills based on visual coherence, and grammar-based sequence compression to induce structured hierarchies over them. An overview of this method is pictured in Figure 1 . We evaluate HiSD on Craftax ( 21 ) and Minecraft ( 3 ) , two domains requiring complex sequential decision-making. Across these domains, HiSD delivers the strongest overall performance among the compared methods, producing accurate skill segmentations and more coherent multi-level hierarchies with fewer assumptions. Our contributions are: (1) a method integrating TAS and grammar induction for unsupervised structure discovery; (2) an observational feature-only pipeline suitable for unlabelled demonstrations; (3) empirical validation of superior segmentation quality and abstraction depth; and (4) a proof of concept demonstrating the utility of these representations in accelerating downstream reinforcement learning. 2 Background 2.1 Skill Segmentation and Identification Temporal action segmentation aims to partition a continuous observation sequence T=(X_{1},\dots,X_{n}) into M temporally contiguous segments, where X\in\mathbb{R}^{d} is a feature vector, an n is the length of the sequence, assigning each a discrete skill label z_{t}\in\{1,\dots,K\} . Here K represents the maximum number of skills in the dataset. In the unsupervised setting, neither the ground-truth labels nor the boundaries are known. Recent state-of-the-art approaches formulate this as an optimal transport problem. Specifically, ASOT ( 32 ) relaxes the segmentation into a soft assignment problem. Let C\in\mathbb{R}^{n\times K} be a cost matrix representing the compatibility between observations and latent skills. ASOT solves for an assignment plan \Gamma\in\mathbb{R}_{+}^{n\times K} by minimising a regularised objective: \displaystyle\min_{\Gamma}\quad \displaystyle\langle C,\Gamma\rangle+\eta\mathcal{R}_{\text{temp}}(\Gamma)+\lambda D_{\text{KL}}(\Gamma^{\top}\mathbf{1}_{n}\,\|\,q), (1) where \mathcal{R}_{\text{temp}} is a temporal regularity term (a Fused Gromov-Wasserstein component) that enforces smoothness, preventing rapid flickering of labels. This formulation utilises unbalanced optimal transport, which allows the algorithm to handle missing and repeated skills. It does not force every latent skill prototype to appear in every episode, nor does it enforce a specific sequential ordering. Furthermore, the KL-divergence term ( D_{\text{KL}} ) acts as a soft constraint on the aggregate skill distribution. This enables the model to handle unbalanced datasets, where certain skills dominate the dataset while critical interaction skills appear only briefly, a characteristic of the datasets used in this work ( 32 ) . 2.2 Grammar Sequence Compression To represent hierarchical structure, we employ the formalism of context-free grammars (CFGs). A CFG is defined as a tuple G=(\mathcal{N},\Sigma,\mathcal{P},S_{0}) , where \Sigma is a set of terminal symbols (atomic units), \mathcal{N} is a set of non-terminal symbols (abstract variables), \mathcal{P} is a set of production rules replacing non-terminals with sequences of symbols, and S_{0} is the distinct start symbol. A prominent method for grammar induction is Sequitur ( 24 ) , a linear-time algorithm capable of inferring hierarchical structure from data. Sequitur operates on a single discrete input string made of terminal and non-terminal symbols in \Sigma , and incrementally builds a hierarchy to compress it. It maintains two invariants: (1) Digram Uniqueness: No pair of adjacent symbols appear more than once in the grammar. If a repetition is found, a new non-terminal rule is created to replace it. (2) Rule Utility: Every rule must be used at least twice. Rules used only once are removed and their contents expanded. Sequitur processes the input string deterministically, producing a grammar where the start symbol S_{0} expands to exactly reproduce the original input string. The resulting derivation tree forms a hierarchy: the root is the full trajectory, internal nodes are discovered non-terminal sub-routines, and leaves are the atomic terminal skills. This is due to the fact that non-terminals can consist of both other non-terminals and terminals, allowing a recursive structure. 3 Related Work Long-horizon decision-making benefits from temporal abstraction and skill reuse, which allow agents to operate at multiple levels of planning ( 5 ; 29 ; 28 ) . We focus on the offline, observation-only setting, where trajectories contain state features but no actions, rewards, or prior segmentation ( 20 ) . 3.1 Skill Segmentation and Discovery Skill segmentation typically assumes access to state-action trajectories or rewards ( 26 ; 16 ) . Approaches such as Compositional Imitation Learning and Execution (CompILE) ( 16 ) and Option-Critic ( 2 ) segment demonstrations into latent skills or options to accelerate RL. However, these methods generally rely on action supervision. CompILE, in particular, suffers from producing location-centric, redundant skills and necessitates prior knowledge of the average length per skill per trajectory (and the number of segments). Similarly, Bayesian methods like Nonparametric Bayesian Reward Segmentation (NPBRS) ( 26 ) are capable of inferring the number of skills from data but typically require heavy supervision, such as explicit rewards, action labels, and environment interaction. Other approaches, including spectral or change-point methods ( 33 ; 17 ) and online unsupervised methods ( 8 ) , typically require active environment interaction. In contrast,